Few-Shot Learning forAbstractive Multi-Document Opinon Summarization

Arthur Brazinskas1 Mirella Lapata1 Ivan Titov1,2

1ILCC, University of Edinburgh2ILLC, University of Amsterdam

abrazinskas@ed.ac.uk, {mlap,ititov}@inf.ed.ac.uk

Abstract

Opinion summarization is an automatic cre-ation of text reflecting subjective informationexpressed in multiple documents, such as userreviews of a product. The task is practicallyimportant and has attracted a lot of attention.However, due to a high cost of summary pro-duction, datasets large enough for training su-pervised models are lacking. Instead, the taskhas been traditionally approached with extrac-tive methods that learn to select text fragmentsin an unsupervised or weakly-supervised way.Recently, it has been shown that abstractivesummaries, potentially more fluent and betterat reflecting conflicting information, can alsobe produced in an unsupervised fashion. How-ever, these models, not being exposed to theactual summaries, fail to capture their essen-tial properties. In this work, we show that evena handful of summaries is sufficient to boot-strap generation of the summary text with allexpected properties, such as writing style, in-formativeness, fluency, and sentiment preser-vation. We start by training a language modelto generate a new product review given avail-able reviews of the product. The model isaware of the properties: it proceeds with firstgenerating property values and then producinga review conditioned on them. We do not useany summaries in this stage and the propertyvalues are derived from reviews with no man-ual effort. In the second stage, we fine-tune themodule predicting the property values on a fewavailable summaries. This lets us switch thegenerator to the summarization mode. Our ap-proach substantially outperforms previous ex-tractive and abstractive methods in automaticand human evaluation.

1 Introduction

Summarization of user opinions expressed in on-line resources, such as blogs, reviews, social media,or internet forums, has drawn much attention due

Gold

These shoes run true to size, do a goodjob supporting the arch of the foot andare well-suited for exercise. They’regood looking, comfortable, and the solefeels soft and cushioned. Overall theyare a nice, light-weight pair of shoes andcome in a variety of stylish colors.

Ours

These running shoes are great! They fittrue to size and are very comfortable torun around in. They are light weight andhave great support. They run a little onthe narrow side, so make sure to order ahalf size larger than normal.

Reviews

perfect fit for me ... supply the supportthat I need ... are flexible and comfort-able ... || ... It is very comfortable ...I enjoy wearing them running ... || ...running shoes ... felt great right out ofthe box ... They run true to size ... ||... my feet and feel like a dream ... To-tally light weight ... || ... shoes run small... fit more true to size ... fit is great!... supports my arch very well ... || ...They are lightweight... usually wear asize women’s 10 ... ordered a 10.5 andthe fit is great!

Table 1: Example summaries produced by our systemand an annotator; colors encode its alignment to theinput reviews. The reviews are truncated, and delimitedwith the symbol ‘||’.

to its potential for various information access appli-cations, such as creating digests, search, and reportgeneration (Hu and Liu, 2004; Medhat et al., 2014;Angelidis and Lapata, 2018).

Although a significant progress has been ob-served in supervised summarization in non-subjective single-document context, such as newsarticles (Rush et al., 2015; Nallapati et al., 2016;Paulus et al., 2017; See et al., 2017; Liu et al.,2018), modern deep learning methods rely on largeamounts of annotated data that are not readily avail-able in the opinion-summarization domain and

arX

iv:2

004.

1488

4v1

[cs

.LG

] 3

0 A

pr 2

020

expensive to produce. The key obstacle makingdata annotation expensive is that the annotatorsneed to consider multiple input texts when writ-ing a summary, which is time-consuming. More-over, annotation would have to be undertaken formultiple domains as online reviews are inherentlymulti-domain (Blitzer et al., 2007) and summariza-tion systems are highly domain-sensitive (Isonumaet al., 2017). This suggests that it is unlikely thathuman-annotated corpora large enough for trainingdeep models will be available.

Recently, a number of unsupervised abstrac-tive multi-document models were introduced (e.g.,Copycat (Brazinskas et al., 2019) and MeanSum(Chu and Liu, 2019)) that are trained on large col-lections of unannotated product reviews1. However,unsurprisingly perhaps, the models, not being ex-posed to the actual summaries, are unable to learntheir key characteristics. For instance, MeanSum(Chu and Liu, 2019) is prone to producing sum-maries that contain a significant amount of infor-mation that is unsupported by reviews; Copycatgenerates summaries that are better aligned withreviews, yet they are limited in details. Moreover,both systems, being trained mostly on subjectivelywritten reviews, tend to generate summaries thathave these characteristics.

The main challenge in the absence of large anno-tated corpora lies in successful utilization of scarceannotated resources. Unlike recent approaches tolanguage model adaptation for abstractive single-document summarization (Hoang et al., 2019; Raf-fel et al., 2019) that utilize hundreds of thousandsof summaries, our two annotated datasets consist ofonly 60 and 100 annotated data-points. As we ob-served, naive retraining of multi-million parametermodels on small corpora leads to rapid over-fittingand poor generalization. In this light, we proposea few-shot learning framework, for which even atiny amount of annotated instances is sufficient tobootstrap generation of the formal summary textthat is both informative and fluent (see Table 1). Tothe best of our knowledge, this work is the first few-shot learning approach applied to summarization.

In our work, we observe that reviews in a largeunannotated collection vary a lot; for example, theydiffer in style, the level of detail, or how much theydiverge from other reviews of the product in termsof content and overall sentiment. We refer to in-

1For simplicity, we use the term ‘product’ to refer to bothAmazon products and Yelp businesses.

dividual review characteristics and their relationsto other reviews as properties. While reviews spana large range of property values, only a subset ofthem is appropriate for summaries. For example,summaries should be close to the product’s reviewsin content, avoid using first-person pronouns andagree with the reviews in sentiment. Our approachstarts with estimating a property-aware model ona large collection of reviews and then adapts themodel using a few annotator-created summaries,effectively switching the generator to the summa-rization regime.

More formally, we estimate a text model on thedataset of reviews; the generator is a Transformerconditional language model (CLM) that is trainedwith the ‘leave-one-out’ objective (Brazinskaset al., 2019) by attending to other reviews of theproduct. We define properties of unannotated datathat are directly related to the end task of summa-rization. Those properties are easy to derive fromreviews, and no extra annotation effort is required.The CLM is conditioned on these properties intraining. The properties encode partial informationabout the target review that is being predicted. Wecapitalize on that by fine-tuning parts of the modeljointly with a tiny plug-in network on a handful ofhuman-written summaries. The plug-in networkis trained to output property values that make thesummaries likely under the trained CLM. The plug-in has less than one percent of the original model’sparameters, and thus is less prone to over-fittingon small datasets. Nevertheless, it can success-fully learn to control dynamics of a large CLM byproviding property values that force generation ofsummaries.

We evaluate our system against both extrac-tive and abstractive methods on Amazon and Yelphuman-created summaries. Summaries generatedby our system are substantially better than thoseproduced by competing methods, as measured byautomatic and human evaluation metrics on bothdatasets. Our contributions can be summarized asfollows:

• we introduce the first few-shot learning frame-work for abstractive summarization;

• we demonstrate that the approach substan-tially outperforms extractive and abstractivemodels, both when measured with automaticmetrics and in human evaluation;

• we provide datasets with abstractive sum-

maries for Amazon products and Yelp busi-nesses.2

2 Unsupervised Training

User reviews about an entity (e.g., a product) arenaturally inter-dependent. For example, knowingthat most reviews are negative about a product’sbattery life, it becomes more likely that the nextreview will also be negative about it. To modelinter-dependencies, yet to avoid intractabilities as-sociated with undirected graphical models (Kollerand Friedman, 2009), we use the ‘leave-one-out’setting (Brazinskas et al., 2019).

Specifically, we assume access to a large cor-pus of user text reviews, which are arranged as Mgroups {r1:N}Mj=1, where r1:N are reviews abouta particular product that are arranged as a tar-get review ri and N − 1 source reviews r−i ={r1, ..., ri−1, ri+1, ..., rN}. Our goal is to estimatethe conditional distribution ri|r−i by optimizingthe parameters θ as shown in Eq. 1.

θ∗ = argmaxθ

1

M N

M∑j=1

N∑i=1

log pθ(rji |r

j−i) (1)

= argmaxθ

1

M N

M∑j=1

N∑i=1

logGθ(rji |Eθ(r

j−i))

(2)

Our model has an encoder-generator architec-ture, where the encoder Eθ produces contextualrepresentations of r−i that are attended by the gen-erator Gθ, which in-turn is a conditional languagemodel (CLM) predicting the target review ri, esti-mated using teacher-forcing (Williams and Zipser,1989). An illustration is presented in Fig. 1.

The objective lets the model exploit common in-formation across reviews, such as rare brand namesor aspect mentions. For example, in Fig. 1, thegenerator can directly attend the word vacuum inthe source reviews to increase its prediction proba-bility.

Additionally, we condition on partial informa-tion about the target review ri using an oracleq(ri, r−i) as shown in Eq. 3.

1

M N

M∑j=1

N∑i=1

logGθ(rji |Eθ(r

j−i), q(r

ji , r

j−i))

(3)2Both the code and datasets will be publicly available.

We refer to this partial information as properties,which correspond to text characteristics of ri or re-lations between ri and r−i. For example, one suchproperty can be a ROUGE score between ri andr−i, which indicates the degree of overlap betweenri and r−i. In Fig. 1, the high value of it can signalto the generator to attend the word vacuum in thesource reviews instead of predicting it based onlanguage statistics. Intuitively, while the model ob-serves a wide distribution of ROUGE scores duringtraining on reviews, during summarization in testtime we can archive a high degree of input-outputtext overlap by setting the property to a high value.We considered three types of properties.

Content Coverage: ROUGE-1, ROUGE-2, andROUGE-L between ri and r−i signals to Gθ howmuch to rely on syntactic information in r−i dur-ing prediction of ri. Writing Style: we proxy theformal and informal writing style by computingpronoun counts, and creating a distribution over 3points of view and an additional class for cases withno pronouns. Rating and Length Deviations: Wecomputed the difference between ri rating and theaverage r−i rating. In the same vein, the differencebetween ri length and the average r−i length.

2.1 Novelty ReductionWhile summary and review generation are techni-cally similar, there is an important difference thatneeds to be addressed. Reviews are often very di-verse, so when a review is predicted, the generatoroften needs to predict content that is not present insource reviews, neither syntactically nor semanti-cally. On the other hand, when a summary is pre-dicted, its content always matches the content ofsource reviews. To address this discrepancy, in ad-dition to using the ROUGE scores as was explainedpreviously, we introduce a novelty reduction tech-nique, which is similar to label smoothing (Pereyraet al., 2017).

Specifically, we add a regularization term L,scaled by λ, that is applied to word distributionsproduced by the generator Gθ as shown in Eq. 4.

1

M N

M∑j=1

N∑i=1

[logGθ(r

ji |Eθ(r

j−i), q(r

ji , r

j−i))

− λL(Gθ(rji |Eθ(rj−i), q(r

ji , r

j−i))

](4)

It penalizes probability mass assignments towords not appearing in r−i as shown in Eq. 5,

Very sturdy vacuum …ri

<latexit sha1_base64="ifRb8gGmfHGhn9c3pLMAUTbXE4g=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0nqQY8FLx4rmrbQhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zYHbX0w8Hhvhpl5YSqFQdf9dkobm1vbO+Xdyt7+weFR9fikbZJMM+6zRCa6G1LDpVDcR4GSd1PNaRxK3gknt3O/88S1EYl6xGnKg5iOlIgEo2ilBz0Qg2rNrbsLkHXiFaQGBVqD6ld/mLAs5gqZpMb0PDfFIKcaBZN8VulnhqeUTeiI9yxVNOYmyBenzsiFVYYkSrQthWSh/p7IaWzMNA5tZ0xxbFa9ufif18swuglyodIMuWLLRVEmCSZk/jcZCs0ZyqkllGlhbyVsTDVlaNOp2BC81ZfXSbtR967qjXuv1nSLOMpwBudwCR5cQxPuoAU+MBjBM7zCmyOdF+fd+Vi2lpxi5hT+wPn8AVJgjcE=</latexit>

ri<latexit sha1_base64="ifRb8gGmfHGhn9c3pLMAUTbXE4g=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0nqQY8FLx4rmrbQhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zYHbX0w8Hhvhpl5YSqFQdf9dkobm1vbO+Xdyt7+weFR9fikbZJMM+6zRCa6G1LDpVDcR4GSd1PNaRxK3gknt3O/88S1EYl6xGnKg5iOlIgEo2ilBz0Qg2rNrbsLkHXiFaQGBVqD6ld/mLAs5gqZpMb0PDfFIKcaBZN8VulnhqeUTeiI9yxVNOYmyBenzsiFVYYkSrQthWSh/p7IaWzMNA5tZ0xxbFa9ufif18swuglyodIMuWLLRVEmCSZk/jcZCs0ZyqkllGlhbyVsTDVlaNOp2BC81ZfXSbtR967qjXuv1nSLOMpwBudwCR5cQxPuoAU+MBjBM7zCmyOdF+fd+Vi2lpxi5hT+wPn8AVJgjcE=</latexit>

r1<latexit sha1_base64="ITAWcSYLnAbJ1YRZ+d/uPbEJTXU=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0nqQY8FLx4rmrbQhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zYHbX0w8Hhvhpl5YSqFQdf9dkobm1vbO+Xdyt7+weFR9fikbZJMM+6zRCa6G1LDpVDcR4GSd1PNaRxK3gknt3O/88S1EYl6xGnKg5iOlIgEo2ilBz3wBtWaW3cXIOvEK0gNCrQG1a/+MGFZzBUySY3peW6KQU41Cib5rNLPDE8pm9AR71mqaMxNkC9OnZELqwxJlGhbCslC/T2R09iYaRzazpji2Kx6c/E/r5dhdBPkQqUZcsWWi6JMEkzI/G8yFJozlFNLKNPC3krYmGrK0KZTsSF4qy+vk3aj7l3VG/derekWcZThDM7hEjy4hibcQQt8YDCCZ3iFN0c6L86787FsLTnFzCn8gfP5A/1xjYk=</latexit>

r1<latexit sha1_base64="ITAWcSYLnAbJ1YRZ+d/uPbEJTXU=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0nqQY8FLx4rmrbQhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zYHbX0w8Hhvhpl5YSqFQdf9dkobm1vbO+Xdyt7+weFR9fikbZJMM+6zRCa6G1LDpVDcR4GSd1PNaRxK3gknt3O/88S1EYl6xGnKg5iOlIgEo2ilBz3wBtWaW3cXIOvEK0gNCrQG1a/+MGFZzBUySY3peW6KQU41Cib5rNLPDE8pm9AR71mqaMxNkC9OnZELqwxJlGhbCslC/T2R09iYaRzazpji2Kx6c/E/r5dhdBPkQqUZcsWWi6JMEkzI/G8yFJozlFNLKNPC3krYmGrK0KZTsSF4qy+vk3aj7l3VG/derekWcZThDM7hEjy4hibcQQt8YDCCZ3iFN0c6L86787FsLTnFzCn8gfP5A/1xjYk=</latexit>

…… … …

Great vacuum …rN

<latexit sha1_base64="JKEkOV4gu1F6zr8kz7MYyBlbM0M=">AAAB6nicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iEcuAjZVENB+QHGFvM5cs2ds7dveEcOQn2FgoYusvsvPfuEmu0MQHA4/3ZpiZFySCa+O6305hY3Nre6e4W9rbPzg8Kh+ftHWcKoYtFotYdQOqUXCJLcONwG6ikEaBwE4wuZn7nSdUmsfy0UwT9CM6kjzkjBorPajB3aBccavuAmSdeDmpQI7moPzVH8YsjVAaJqjWPc9NjJ9RZTgTOCv1U40JZRM6wp6lkkao/Wxx6oxcWGVIwljZkoYs1N8TGY20nkaB7YyoGetVby7+5/VSE177GZdJalCy5aIwFcTEZP43GXKFzIipJZQpbm8lbEwVZcamU7IheKsvr5N2repdVWv3XqXh5nEU4QzO4RI8qEMDbqEJLWAwgmd4hTdHOC/Ou/OxbC04+cwp/IHz+QMpdI2m</latexit>

rN<latexit sha1_base64="JKEkOV4gu1F6zr8kz7MYyBlbM0M=">AAAB6nicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iEcuAjZVENB+QHGFvM5cs2ds7dveEcOQn2FgoYusvsvPfuEmu0MQHA4/3ZpiZFySCa+O6305hY3Nre6e4W9rbPzg8Kh+ftHWcKoYtFotYdQOqUXCJLcONwG6ikEaBwE4wuZn7nSdUmsfy0UwT9CM6kjzkjBorPajB3aBccavuAmSdeDmpQI7moPzVH8YsjVAaJqjWPc9NjJ9RZTgTOCv1U40JZRM6wp6lkkao/Wxx6oxcWGVIwljZkoYs1N8TGY20nkaB7YyoGetVby7+5/VSE177GZdJalCy5aIwFcTEZP43GXKFzIipJZQpbm8lbEwVZcamU7IheKsvr5N2repdVWv3XqXh5nEU4QzO4RI8qEMDbqEJLWAwgmd4hTdHOC/Ou/OxbC04+cwp/IHz+QMpdI2m</latexit>

… … ……

Enco

der S

tate

s

This …

vacuumhooverproduct

Generator States

……

……

q(ri, r�i)<latexit sha1_base64="HlUyG/jukbL1w4JcMizTkACLrXc=">AAAB9XicbVBNS8NAEJ3Ur1q/qh69LBahgpakHvRY8OKxgv2ANobNdtMu3Wzi7kYpIf/DiwdFvPpfvPlv3LY5aOuDgcd7M8zM82POlLbtb6uwsrq2vlHcLG1t7+zulfcP2ipKJKEtEvFIdn2sKGeCtjTTnHZjSXHoc9rxx9dTv/NIpWKRuNOTmLohHgoWMIK1ke4fqtJjZ0h66TnLTr1yxa7ZM6Bl4uSkAjmaXvmrP4hIElKhCcdK9Rw71m6KpWaE06zUTxSNMRnjIe0ZKnBIlZvOrs7QiVEGKIikKaHRTP09keJQqUnom84Q65Fa9Kbif14v0cGVmzIRJ5oKMl8UJBzpCE0jQAMmKdF8YggmkplbERlhiYk2QZVMCM7iy8ukXa85F7X6rVNp2HkcRTiCY6iCA5fQgBtoQgsISHiGV3iznqwX6936mLcWrHzmEP7A+vwBN5ORnA==</latexit>

q(ri, r�i)<latexit sha1_base64="HlUyG/jukbL1w4JcMizTkACLrXc=">AAAB9XicbVBNS8NAEJ3Ur1q/qh69LBahgpakHvRY8OKxgv2ANobNdtMu3Wzi7kYpIf/DiwdFvPpfvPlv3LY5aOuDgcd7M8zM82POlLbtb6uwsrq2vlHcLG1t7+zulfcP2ipKJKEtEvFIdn2sKGeCtjTTnHZjSXHoc9rxx9dTv/NIpWKRuNOTmLohHgoWMIK1ke4fqtJjZ0h66TnLTr1yxa7ZM6Bl4uSkAjmaXvmrP4hIElKhCcdK9Rw71m6KpWaE06zUTxSNMRnjIe0ZKnBIlZvOrs7QiVEGKIikKaHRTP09keJQqUnom84Q65Fa9Kbif14v0cGVmzIRJ5oKMl8UJBzpCE0jQAMmKdF8YggmkplbERlhiYk2QZVMCM7iy8ukXa85F7X6rVNp2HkcRTiCY6iCA5fQgBtoQgsISHiGV3iznqwX6936mLcWrHzmEP7A+vwBN5ORnA==</latexit>

Attention

Figure 1: Illustration of the model with the leave-one-out objective. Here predictions of the target review ri isperformed by conditioning on the encoded source reviews r−i. The generator attends the last encoder layer’soutput to extract common information. Additionally, the generator has partial information about ri passed by theoracle q(ri, r−i).

and thus steers towards generation of text that ismore grounded in content of r−i.

L(Gθ(ri|r−i, q(ri, r−i))) =T∑t=1

∑w 6∈V (r−i)

Gθ(Wt = w|r1:t−1i , r−i, q(ri, r−i))

(5)Here T is the length of ri, and the inner sum

is over all words that do not appear in the wordvocabulary of r−i. Intuitively, in Fig. 1, the penaltycould reduce the probability of the word hooverto be predicted as it does not appear in the sourcereviews.

3 Summary Adaptation

Once the unsupervised model is trained on reviews,our task is to adapt it to generation of summaries.Here we assume access to a small quantity ofannotator-written summaries (sk, rk1:N )

Kk=1 where

s is a summary for r1:N input reviews. As we willshow in Sec. 6.1, naive retraining of the unsuper-vised model on a handful of annotated data-pointsleads to a poor generalization. Instead, we capital-ize on the fact that the generator Gθ has observeda wide range of property values associated withri during the unsupervised training phase. Intu-itively, some combinations of property values driveit into generation of text that has qualities of a sum-

mary while others of a review. However, we mightnot know values in advance that are necessary forgeneration of summaries. Furthermore, q(ri, r−i)cannot be applied in test time as it requires accessto target texts. In the following section we describea solution that switches the generator to the sum-marization mode relying only on input reviews.

3.1 Plug-in Network InitializationWe start by introducing a parametrized plug-in net-work pφ(r−i) that yields the same types of prop-erties as q(ri, r−i). From a practical perspective,the plug-in should be input-permutation invariantand allow for an arbitrary number of input reviews(Zaheer et al., 2017). Importantly, the trainableplug-in can have a marginal fraction of the mainmodel’s parameters, which makes it less prone toover-fitting when trained on small datasets. Weinitialize the parameters of pφ(r−i) by matchingits output to q(ri, r−i) on the unannotated reviews.Specifically, we used a weighted combination ofdistances as shown for one group of reviews inEq. 6.

N∑i=1

L∑l=1

αlDl(pφ(r−i)l, q(ri, r−i)

l) (6)

Here, Dl(pφ(r−i)l, q(ri, r−i)

l) is a distance forthe property l, and αl is an associated weight.

Specifically, we used L1 norm for Content Cover-age, Rating and Length Deviations, and Kullback-Leibler divergence for Writing Style.

3.2 Fine-Tuning

Unsurprisingly, perhaps, the network pφ being ini-tialized on unannotated reviews inherits a strongbias towards outputting property values resultingin generation of reviews, which should not be ap-propriate for generating summaries. Fortunately,due to the simplicity of the chosen properties, it ispossible to fine-tune pφ to match the output of q onthe annotated data (sk, rk1:N )

Kk=1 using Eq. 6. An

alternative is to optimize the plug-in to directly in-crease the likelihood of summaries under Gθ whilekeeping all other parameters fixed 3.

Second, we observe that the generator Gθ mighthave rarely been exposed to the summary relatedproperty values during the unsupervised trainingphase (described in Sec. 2). Additionally, the gener-ator might have not encountered a sufficient amountof reviews that are written as summaries and re-quire high reliance on input reviews. We addressthat by unfreezing the attention module of Gθ overinput reviews and the plug-in pφ, and by maximiz-ing the likelihood of summaries as shown in Eq. 7:

1

K

K∑k=1

[logGθ(s

k|Eθ(rk1:N ), pφ(rk1:N ))]

(7)

This allows the system to learn an interaction be-tween Gθ and pφ that should result in summariesbeing likely under Gθ.

4 Experimental Setup

4.1 Dataset

For training we used customer reviews from Ama-zon (He and McAuley, 2016) and Yelp.4 From theAmazon reviews we selected 4 categories: Elec-tronics; Clothing, Shoes and Jewelry; Home andKitchen; Health and Personal Care. We used asimilar pre-processing schema as in (Brazinskaset al., 2019), details are presented in Appendix 8.1.For training, we partitioned each business/productreviews to groups of 9 reviews by sampling with-out replacement. The data-statistics are shown inTable 2.

3We explored that option, and observed that it works simi-larly, yet leads to a slightly worse result.

4https://www.yelp.com/dataset/challenge

Dataset Training ValidationYelp 38,913/1,016,347 4,324/113,886

Amazon 182,932/3,889,782 9,629/205,992

Table 2: Data statistics after pre-processing. Theformat in the cells is Businesses/Reviews and Prod-ucts/Reviews for Yelp and Amazon, respectively.

For evaluation, we re-annotated the reviews of 60products in Brazinskas et al. (2019) using AmazonMechanical Turk (AMT) as the original summariesshare the informal writing style with customer re-views. We asked workers to write summaries for-mally, and each product received three summariesproduced by different workers. We used 28, 12,20 products for training, validation, and testing,respectively. In the same vein, we annotated 100Yelp businesses and used 30, 30, 40 for training,validation, and testing, respectively.

4.2 Experimental Details

We used the Transformer architecture (Vaswaniet al., 2017) with trainable length embeddings andshared encoder-generator parameters (Raffel et al.,2019). Subwords were obtained with BPE (Sen-nrich et al., 2015) using 32000 merges. Subwordembeddings were shared across the model as aform of regularization (Press and Wolf, 2016). Fora fair comparison, we approximately matched thenumber of parameters to the abstractive baselinemodels MeanSum (Chu and Liu, 2019) and Copy-cat (Brazinskas et al., 2019). We randomly initial-ized all parameters with Glorot (Glorot and Bengio,2010). For the plug-in network we employed aTransformer stack with an attention sub-layer oversource reviews at each layer. After the last layer, weperformed a linear projection to compute propertyvalues as mentioned in Sec. 2. Further, parameteroptimization was performed using Adam (Kingmaand Ba, 2014), and beam search with n-gram block-ing was applied to our model and Copycat for sum-mary generation. Details of the training procedureare presented in Appendix. 8.2.

Hyperparameters Our parameter-sharedencoder-generator model uses a 8-head and 6-layerTransformer stack. Dropout in sub-layers andsubword embeddings dropout was both set to0.1, and we used 1000 dimensional position-wisefeed-forward neural networks. We set subword andlength embeddings to 390 and 10 respectively, andboth were concatenated to be used as input. For

the plug-in network we set the model dimensionto 30, position-wise feed-forward networks to 20dimensions. Further, we used a 3-layers stack with3 heads. We applied 0.4 sub-layer dropout, andreduced the memory attention dropout to 0.15.Property values produced by the plug-in wereconcatenated with subword and length embeddingsand linearly projected before being passed to thegenerator. In total, our model had approximately25M parameters, while the plug-in network only100K (i.e., less than 0.5 % of the main model’sparameters).

4.3 Baselines

LexRank (Erkan and Radev, 2004) is an unsuper-vised extractive graph based algorithm selectingsentences based on graph centrality. Sentences rep-resent nodes in a graph whose edges have weightsdenoting similarity computed with tf-idf.

Copycat is the state-of-the-art unsupervised ab-stractive summarizer (Brazinskas et al., 2019) thatuses continues latent representations to model re-view groups and individual review semantics. Ithas an implicit mechanism for novelty reductionand a copy mechanism applied over source reviews.

MeanSum is an unsupervised abstractive sum-marization model (Chu and Liu, 2019) that treatsa summary as a discrete latent state of an autoen-coder. The model is trained in a multi-task fashionwith two objectives, one for prediction of reviewsand the other one for summary-reviews alignmentin the semantic space using the cosine similarity.

As is common in the summarization literature,we also employed a number of simple summariza-tion baselines. First, the clustroid review was com-puted for each group of reviews as follows. We tookeach review from a group and computed ROUGE-L with respect to all other reviews. The reviewwith the highest ROUGE score was selected as theclustroid review. Second, we sampled a randomreview from each group to be used the summary.Third, we constructed the summary by selectingthe leading sentences from each review of a group.

5 Evaluation Results

5.1 Automatic Evaluation

We report ROUGE F1 score based evaluation re-sults on the Amazon and Yelp test sets in Tables 3and 4, respectively. The results indicate that ourmodel outperforms abstractive and extractive meth-ods on both datasets. Additionally, summaries pro-

R1 R2 RLOurs 0.3356 0.0716 0.2149Copycat 0.2785 0.0477 0.1886MeanSum 0.2663 0.0489 0.1711LexRank 0.2772 0.0506 0.1704Clustroid 0.2716 0.0361 0.1677Lead 0.2700 0.0492 0.1495Random 0.2500 0.0382 0.1572

Table 3: ROUGE scores on the Amazon test set.

R1 R2 RLOurs 0.3729 0.0992 0.2276Copycat 0.2812 0.0589 0.1832MeanSum 0.2750 0.0354 0.1609LexRank 0.2696 0.0493 0.1613Clustroid 0.2890 0.0490 0.1800Lead 0.2620 0.0457 0.1432Random 0.2148 0.0259 0.1387

Table 4: ROUGE scores on the Yelp test set.

duced by our model are qualitative better, see ex-amples in Appendix.

5.2 Human Evaluation

Best-Worst Scaling We performed human eval-uation based on the Amazon and Yelp test setsusing the AMT platform. We assigned 3 workersto each tuple containing summaries from Copy-cat, our model, LexRank, and human annotators.We presented the associated reviews in a randomorder and asked the workers to judge summariesusing the Best-Worst scaling (BWS) (Louviere andWoodworth, 1991; Louviere et al., 2015) that isknown to produce more reliable results than rank-ing scales (Kiritchenko and Mohammad, 2016).The judgment criteria are presented below, wherenon-redundancy and coherence were taken fromDang (2005).

Fluency: the summary sentences should be gram-matically correct, easy to read and understand; Co-herence: the summary should be well structuredand well organized; Non-redundancy: there shouldbe no unnecessary repetition in the summary; In-formativeness: how much useful information aboutthe product does the summary provide?; Sentiment:how well the sentiment of the summary agrees withthe overall sentiment of the original reviews?

For every criterion, a system’s score is computedas the percentage of times it was selected as best

Fluency Coherence Non-Red. Informativeness SentimentOurs 0.1000 0.1429 0.1250 0.2000 0.3061Copycat -0.1765 -0.5333 -0.2727 -0.7455 -0.7143LexRank -0.4848 -0.5161 -0.5862 -0.3488 -0.0909Gold 0.5667 0.6364 0.6066 0.6944 0.4138

Table 5: Human evaluation results in terms of the Best-Worst scaling on the Amazon test set.

Fluency Coherence Non-Red. Informativeness SentimentOurs 0.1636 0.1429 0.0000 0.3793 0.3725Copycat -0.2000 -0.0769 0.1053 -0.4386 -0.2857LexRank -0.5588 -0.5312 -0.6393 -0.6552 -0.4769Gold 0.5278 0.3784 0.4795 0.6119 0.4118

Table 6: Human evaluation results in terms of the Best-Worst scaling on the Yelp test set.

minus the percentage of times it was selected asworst (Orme, 2009). The scores range from -1(unanimously worst) to +1 (unanimously best).

The results are presented in Tables 5 and 6 forAmazon and Yelp, respectively. On the Amazondata, they indicate that our model is preferredacross the board over the baselines. Copycat ispreferred over LexRank in terms of fluency andnon-redundancy, yet it shows worse results in termsof informativeness and overall sentiment preser-vation. In the same vein, on Yelp in Table 6 ourmodel outperforms other models. We also observedthat Copycat’s summaries are scored higher thanLexrank’s in terms of informativeness and over-all sentiment preservation. We hypothesize thatgeneric summaries, such as the ones produced byCopycat, are preferred Yelp as they abstract awayfrom details, which may be not so important in therestaurant domain. For example, the word ‘food’may work better than enumerating multiple menuitems. On the other hand, abstracting away fromdetails is less desirable for Amazon product sum-maries.

All pairwise differences between our modeland other models are statistically significant atp < 0.05, using post-hoc HD Tukey tests. The onlyexception is non-redundency on Yelp when com-paring our model and Copycat (where our modelshows a slightly lower score).

Content Support As was observed byBrazinskas et al. (2019); Falke et al. (2019),the ROUGE metric relies on unweighted n-gramoverlap and can be insensitive to hallucinatingfacts and entities. We also investigated how welltext generated by our system is supported by

Full Partial NoOurs 0.4309 0.3414 0.2276Copycat 0.4615 0.2718 0.2667

Table 7: Content support on the Amazon test set, per-centages.

input reviews. We split summaries generated byour model and Copycat into sentences. Thenfor each summary sentence, we hired 3 AMTworkers to judge how well content of the sentenceis supported by the reviews. Three followingoptions were available. Full support: all thecontent is reflected in the reviews; Partial support:only some content is reflected in the reviews; Nosupport: content is not reflected in the reviews.

The results in Table 7 indicate that our modelgenerates content that is well supported by inputreviews.

6 Analysis

6.1 Alternative Adaptation StrategiesWe further explored alternative utilization ap-proaches of annotated data-points, based on thesame split of the Amazon summaries as explainedin Sec. 4.1. First, we trained a model in the unsu-pervised learning setting (USL) on the Amazonreviews with the leave-one-out objective in Eq.2. In this setting, the model has no exposure tosummaries, neither summary properties as the or-acle q(ri, r−i) is not used. Further, we consideredtwo alternative settings how the pre-trained unsu-pervised model can be adapted on the gold sum-maries. In the first setting, the pre-trained unsu-pervised model is further re-trained by predicting

summaries conditioned on input reviews (USL+R).In the second one, similar to Hoang et al. (2019),we performed adaptation in a multi-tasking learn-ing (MTL) fashion. Here USL is further trained ona mixture of unannotated corpus review and goldsummary batches with domain learned embeddingadded at each position during prediction. The re-sults are presented in Table 8.

First, we observed that USL generates sum-maries that get the worst ROUGE scores. Addi-tionally, the generated text tends to be informaland substantially shorter than an average summary,we shall discuss that in Sec. 6.2. Second, whenthe model is re-trained on the gold summaries(USL+R), it noticeably improves the results, yetthey are substantially lower than of our proposedfew-shot approach. It can be explained by a strongbias of the unannotated data stored in millions of pa-rameters that requires more annotated data-pointsto overwrite. Finally, we observed that MTL failsto decouple the domains indicated only by a slightimprovement over USL.

R1 R2 RLOurs 0.3356 0.0716 0.2149MTL 0.2403 0.0435 0.1627USL+R 0.2823 0.0624 0.1964USL 0.2145 0.0315 0.1523Random 0.2500 0.0382 0.1572

Table 8: ROUGE scores on the Amazon test set foralternative summary adaptation strategies.

6.2 Bias of Unannotated Data

We further analyzed how a plain re-training on sum-maries differs from our approach in terms of sum-mary characteristics adaptation. For comparison,we used USL and USL+R, which are presented inSec. 6.1. Additionally, we analyzed unannotatedreviews from the Amazon training set. Specifically,we focused on text formality and the average wordcount difference (Len) from gold summaries in theAmazon gold test set. As a proxy for the former,we computed marginal distributions over points ofview (POV) based on pronoun counts, additionalclass (NoPr) was allocated to cases of no pronouns,same as described in Sec. 2. The results are pre-sented in Table 9.

First, we observed that the training reviews arelargely informal, indicated by 56.3% of probabil-ity mass allocated to the 1st and 2nd POV pro-

1st 2nd 3rd NoPr LenGold 0.00 1.67 60.00 38.33 0.0Ours 0.50 0.13 83.17 15.00 3.4USL+R 29.67 0.00 45.33 25.00 -28.6USL 56.68 0.00 43.32 0.00 -32.7Reviews 49.00 7.31 35.56 8.13 -17.6

Table 9: Text characteristics of generated summariesby different models on the Amazon test set. The firstcolumn block corresponds to the POV marginal distri-bution; the second to the average length deviation (inwords) from the human-created summaries, where anegative value corresponds to on average shorter gen-erated text than summaries.

Gold

These shoes run true to size, do a goodjob supporting the arch of the foot andare well-suited for exercise. They’regood looking, comfortable, and the solefeels soft and cushioned. Overall theyare a nice, light-weight pair of shoes andcome in a variety of stylish colors.

Ours

These running shoes are great! They fittrue to size and are very comfortable torun around in. They are light weight andhave great support. They run a little onthe narrow side, so make sure to order ahalf size larger than normal.

USL+R

This is my second pair of Reebok run-ning shoes and they are the best run-ning shoes I have ever owned. They arelightweight, comfortable, and providegreat support for my feet.

USL

This is my second pair of Reebok run-ning shoes and I love them. They arethe most comfortable shoes I have everworn.

Table 10: Example summaries produced by modelswith different adaptation approaches.

nouns. Unsurprisingly, the model trained only onthe reviews (USL) transfers a similar trait to thesummaries that it generates.5 On the contrary, thegold summaries are largely formal - indicated byan almost complete absence of the aforementionedpronouns. Also, an average review is substantiallyshorter than an average gold summary, and con-sequently, the generated summaries by USL arealso shorter. Example summaries are presented inTable 10.

5As beam search, attempting to find the most likely can-didate sequence, was utilized, opposed to a random sequencesampling, we observed that generated sequences had no casesof the 2nd POV pronouns or complete absence of pronouns(NoPr).

Further, we investigated how well USL+R,adapts to the summary characteristics by being re-trained on summaries. Indeed, we observed thatUSL+R starts to shift in the direction of the sum-maries by reducing the pronouns of the 1st POVand increasing the average summary length. Never-theless, the gap is still wide, which would probablyrequire more data to be bridged. Finally, we ob-served that our approach adapts much better to thedesired characteristics by producing well-formedsummary text that is also very close in length tothe gold summaries. This is also evident from theexample summaries in Table 10.

7 Conclusion

In this work, we introduced the first to our knowl-edge few-shot framework for abstractive summa-rization. We show that it can efficiently utilizeeven a handful of annotated reviews-summary pairsto train models that generate fluent, informative,and overall sentiment reflecting summaries. Wepropose to exploit summary related properties inunannotated reviews that are used for unsupervisedtraining of a generator. Then post-hoc train a tinyplug-in network that learns to switch the generatorto the summarization regime. We demonstrate thatour approach substantially outperforms competi-tive ones, both abstractive and extractive, in humanand automatic evaluation.

Acknowledgments

We would like to thank members of EdinburghNLP group for discussion. We gratefully acknowl-edge the support of the European Research Council(Titov: ERC StG BroadSem 678254; Lapata: ERCCoG TransModal 681760) and the Dutch NationalScience Foundation (NWO VIDI 639.022.518).

References

Stefanos Angelidis and Mirella Lapata. 2018. Sum-marizing opinions: Aspect extraction meets senti-ment prediction and they are both weakly supervised.arXiv preprint arXiv:1808.08858.

John Blitzer, Mark Dredze, and Fernando Pereira. 2007.Biographies, bollywood, boom-boxes and blenders:Domain adaptation for sentiment classification. InProceedings of the 45th annual meeting of the asso-ciation of computational linguistics, pages 440–447.

Arthur Brazinskas, Mirella Lapata, and Ivan Titov.2019. Unsupervised multi-document opinion sum-marization as copycat-review generation. arXivpreprint arXiv:1911.02247.

Eric Chu and Peter Liu. 2019. Meansum: a neu-ral model for unsupervised multi-document abstrac-tive summarization. In Proceedings of InternationalConference on Machine Learning (ICML), pages1223–1232.

Hoa Trang Dang. 2005. Overview of duc 2005. In Pro-ceedings of the document understanding conference,volume 2005, pages 1–12.

Gunes Erkan and Dragomir R Radev. 2004. Lexrank:Graph-based lexical centrality as salience in textsummarization. Journal of artificial intelligence re-search, 22:457–479.

Tobias Falke, Leonardo FR Ribeiro, Prasetya AjieUtama, Ido Dagan, and Iryna Gurevych. 2019.Ranking generated summaries by correctness: An in-teresting but challenging application for natural lan-guage inference. In Proceedings of the 57th AnnualMeeting of the Association for Computational Lin-guistics, pages 2214–2220.

Xavier Glorot and Yoshua Bengio. 2010. Understand-ing the difficulty of training deep feedforward neuralnetworks. In Proceedings of the thirteenth interna-tional conference on artificial intelligence and statis-tics, pages 249–256.

Ruining He and Julian McAuley. 2016. Ups and downs:Modeling the visual evolution of fashion trends withone-class collaborative filtering. In proceedings ofthe 25th international conference on world wideweb, pages 507–517.

Andrew Hoang, Antoine Bosselut, Asli Celikyilmaz,and Yejin Choi. 2019. Efficient adaptation of pre-trained transformers for abstractive summarization.arXiv preprint arXiv:1906.00138.

Minqing Hu and Bing Liu. 2004. Mining and summa-rizing customer reviews. In Proceedings of the tenthACM SIGKDD international conference on Knowl-edge discovery and data mining, pages 168–177.ACM.

Masaru Isonuma, Toru Fujino, Junichiro Mori, YutakaMatsuo, and Ichiro Sakata. 2017. Extractive sum-marization using multi-task learning with documentclassification. In Proceedings of the 2017 Confer-ence on Empirical Methods in Natural LanguageProcessing, pages 2101–2110.

Diederik P Kingma and Jimmy Ba. 2014. Adam: Amethod for stochastic optimization. arXiv preprintarXiv:1412.6980.

Svetlana Kiritchenko and Saif M Mohammad. 2016.Capturing reliable fine-grained sentiment associa-tions by crowdsourcing and best–worst scaling. InProceedings of the 2016 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies,pages 811–817.

Daphne Koller and Nir Friedman. 2009. Probabilisticgraphical models: principles and techniques. MITpress.

Peter J Liu, Mohammad Saleh, Etienne Pot, BenGoodrich, Ryan Sepassi, Lukasz Kaiser, and NoamShazeer. 2018. Generating wikipedia by summariz-ing long sequences. In Proceedings of InternationalConference on Learning Representations (ICLR).

Jordan J Louviere, Terry N Flynn, and Anthony Al-fred John Marley. 2015. Best-worst scaling: The-ory, methods and applications. Cambridge Univer-sity Press.

Jordan J Louviere and George G Woodworth. 1991.Best-worst scaling: A model for the largest differ-ence judgments. University of Alberta: Working Pa-per.

Walaa Medhat, Ahmed Hassan, and Hoda Korashy.2014. Sentiment analysis algorithms and applica-tions: A survey. Ain Shams engineering journal,5(4):1093–1113.

Ramesh Nallapati, Bowen Zhou, Cicero dos Santos,Caglar Gulcehre, and Bing Xiang. 2016. Abstrac-tive text summarization using sequence-to-sequencernns and beyond. In Proceedings of The 20thSIGNLL Conference on Computational Natural Lan-guage Learning, pages 280–290.

Bryan Orme. 2009. Maxdiff analysis: Simple counting,individual-level logit, and hb. Sequim, WA: Saw-tooth Software.

Romain Paulus, Caiming Xiong, and Richard Socher.2017. A deep reinforced model for abstractive sum-marization. arXiv preprint arXiv:1705.04304.

Gabriel Pereyra, George Tucker, Jan Chorowski,Łukasz Kaiser, and Geoffrey Hinton. 2017. Regular-izing neural networks by penalizing confident outputdistributions. arXiv preprint arXiv:1701.06548.

Ofir Press and Lior Wolf. 2016. Using the outputembedding to improve language models. arXivpreprint arXiv:1608.05859.

Colin Raffel, Noam Shazeer, Adam Roberts, KatherineLee, Sharan Narang, Michael Matena, Yanqi Zhou,Wei Li, and Peter J Liu. 2019. Exploring the limitsof transfer learning with a unified text-to-text trans-former. arXiv preprint arXiv:1910.10683.

Alexander M Rush, Sumit Chopra, and Jason Weston.2015. A neural attention model for abstractive sen-tence summarization. In Proceedings of the 2015Conference on Empirical Methods in Natural Lan-guage Processing, pages 379–389.

Abigail See, Peter J Liu, and Christopher D Manning.2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of Associationfor Computational Linguistics (ACL).

Rico Sennrich, Barry Haddow, and Alexandra Birch.2015. Neural machine translation of rare words withsubword units. arXiv preprint arXiv:1508.07909.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, ŁukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In Advances in neural information pro-cessing systems, pages 5998–6008.

Ronald J Williams and David Zipser. 1989. A learn-ing algorithm for continually running fully recurrentneural networks. Neural computation, 1(2):270–280.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh,Barnabas Poczos, Russ R Salakhutdinov, andAlexander J Smola. 2017. Deep sets. In Advances inneural information processing systems, pages 3391–3401.

8 Appendices

8.1 Dataset Pre-Processing

We selected only Amazon products and Yelp busi-nesses with minimum of 10 reviews, and the min-imum and maximum lengths of 20 and 70 words,respectively. Also, popular products/businessesabove the 90th percentile were removed. Fromeach business/product we sampled 9 reviews with-out replacement to form groups of reviews.

8.2 Training Procedure

First, to speed-up the training phase, we trained anunconditional language model for 13 epoch on theAmazon reviews with the learning rate (LR) set to5 ∗ 10−4. On Yelp we trained it for 27 epochs withLR set to 7 ∗ 10−4. The language model was usedto initialize both the encoder and generator of themain model.

Subsequently, we trained the model using Eq. 3for 9 epochs on the Amazon reviews with 6 ∗ 10−5LR, and for 57 epochs with LR set to 5 ∗ 10−5.Additionally, we reduced novelty using Eq. 5 bytraining the model further for 1 epoch with 10−5

LR and λ = 2 on Amazon; On Yelp we trained for4 epochs, with 3 ∗ 10−5 LR, and lambda = 2.5.

For the plugin network’s initialization, as ex-plained in Sec. 3.1, we performed optimization byoutput matching with the oracle for 11 epochs onthe unannotated Amazon reviews with 1∗10−5 LR.On Yelp we trained for 87 epochs with 1 ∗ 10−5Lastly, we fine-tuned the plugin network on thehuman-written summaries by output matching withthe oracle6. On the Amazon data for 98 epochs with7 ∗ 10−4, and for 62 epochs with 7 ∗ 10−5 on Yelp.We set weights to 0.1, 1., 0.08, 0.5 for length devi-ation, rating deviation, POV, and ROUGE scores,respectively. Then fine-tuned the attention part ofthe model and the plug-in network jointly for 33epochs with 1 ∗ 10−4 on the Amazon data. And 23epochs with 1 ∗ 10−4 LR on Yelp.

8.3 Human Evaluation Setup

To perform the human evaluation experiments de-scribed in Sec 5.2, we hired workers with 98%approval rate, 1000+ HITS, Location: USA, UK,Canada, and the maximum score on a qualificationtest that we had designed. The test was asking ifthe workers were native English speakers, and was

6We set rating deviation to 0 as summaries do not haveassociated human-annotated ratings.

verifying that they correctly understood the instruc-tions of both the best-worst scaling and contentsupport tasks.

8.4 Summary AnnotationFor summary annotation we reused 60 Amazonproducts from Brazinskas et al. (2019) and sampled100 businesses from Yelp. We assigned 3 workersto each product/business, and instructed them toread the reviews and produce a summary text. Weused the following instructions:

• The summary should reflect user commonopinions expressed in the reviews. Try to pre-serve the common sentiment of the opinionsand their details (e.g. what exactly the userslike or dislike). For example, if most reviewsare negative about the sound quality, then alsowrite negatively about it.

• Please make the summary coherent and fluentin terms of sentence and information struc-ture. Iterate over the written summary mul-tiple times to improve it, and re-read the re-views whenever necessary.

• The summary should not look like a review,please write formally.

• Keep the length of the summary reasonablyclose to the average length of the reviews.

• Please try to write the summary using yourown words instead of copying text directlyfrom the reviews. Using the exact words fromthe reviews is allowed but do not copy morethan 5 consecutive words from a review.

GoldBennett Medical has poor customer service. Phone calls can take a long time to get answeredand leaving voice mails tend to be fruitless. The products are overpriced and take a long time tobe refilled. Using this medical supply company can be a hassle.

OursThis medical supply is the worst medical supply company in the valley. The customer service ishorrible, the staff is rude, the wait times are long, and the service reps are not helpful at all. Donot recommend this company to anyone.

CopycatIf I could give them 0 stars I would. The customer service is terrible, the staff is extremely rudeand helpful. If you’re looking for a new provider, this is the place to be. I will never use themagain.

MeanSum

Service is horrible, especially the manager. I have a lot of kids but not this place. Two monthslater I was able to go in and get a new one to go in the next day. They would tell me that theywould have to do a lot of our water to be there to get a new one. But this is the first time I havedealt with him and we will never go back again. Thanks for your hard work, and I will never goanywhere else.

LexrankBennett Medical for Cpap supplies are horrible. Never enough staff to answer phone, so you’llneed to leave messages. DON’T use this medical supply. If I could give Bennett Medical zerostars I would! Will be moving to another medical supply as soon as I can.

Review 1Bennett Medical for Cpap supplies are horrible. We have waited for three weeks to refill suppliesand we are still waiting. This company does not have good customer service, you can only leavemessages, and they never call back. If I could give Bennett Medical zero stars I would!

Review 2

Teachers Health Trust, please look into the practice of the billing and filling of durable services.The mask cushions go for 45 to 50 days because of the lack of communication. The people incharge of billing are very argumentative and lack customer service. I will drop them after annual,because of my insurance obligations.

Review 3Fantastic service from Jocelyn at the front desk, we had a really hard time getting the rightpaperwork together from Drs but she stuck with us and helped us every step of the way, evencalling to keep us updated and to update info we might have for her. Thanks Jocelyn.

Review 4

I hardly ever write reviews, but I’d like to spare someone else from what I experienced. So awarning to the wise... If you like rude incompetent employees, almost an hour long wait for justpicking up a phone order, and basically being treated like a second class citizen then look nofurther than Bennett Medical.

Review 5

DON’T use this medical supply. Never enough staff to answer phone, so you’ll need to leavemessages. No return phone calls. I am unable to get my CPAP supplies every quarter withouthours of calling / waiting / calling. Poor customer service. Will be moving to another medicalsupply as soon as I can.

Review 6

Terrible experience. They have ridiculous price also bad customer services. You can get nebulizermachine around $50 at amazon, bennet medical charge you almost twice more expensive price.And breathing kit price was unbelievable too. Because of deduction, I had to pay them all out ofmy pocket whatever they charged. I don’t recommand this medical company to anyone.

Review 7

Good luck getting a phone call back or someone to answer the phone without hanging upimmediately. I have called over 20 times left 5 voicemails over the last 30 days, just to refill amask perscription. This is an ongoing issue that is beyond frustrating. Not trying to destroy thisbusinesses name just want the owners to implement some basic customer service skills.

Review 8Always receive friendly customer service whenever we call or go into the location. My questionsare always answered and I am very happy with the supplies we get from them. Great peopleproviding a great service! Thank you for all you do!

Table 11: Example summaries produced by different systems on Yelp data.

GoldIt is very clean and nice inside. Everyone is so kind and friendly. They do an amazing job onboth nails and pedis. They get it done with speed and precision with a price that is very muchaffordable. They have the best customer service.

OursThis nail salon is very clean and the staff is very friendly. They have a wide variety of gel colorsto choose from. The prices are very reasonable and they do a great job. The nail techs are verynice and do great work.

Copycat This is the best nail salon I have ever been to. Everyone is very friendly and professional. Mynails look great and I’m glad I did! I will definitely be coming back to this place.

MeanSum

The owner is so nice and accommodating. I went to get my nails done by a friend, and I wasextremely happy with this salon. Everyone was very friendly and I was able to use them for nails.They did a great job on my nails and the best part about it was that it was a busy day but it was atreat! Highly recommend them.

LexrankI really enjoy coming here to get my nails done. B did an amazing job on my nails. Amazingservice and nails. However B did an AMAZING job on my coffin chrome nails and Nancy wasextremely helpful figuring out how I wanted my nails done too. Everyone is so friendly there too.

Review 1Tim and Tami always always always have the best customer service and do the best nails. I willNEVER go anywhere else. Even after weeks my nails look and feel as good as they did when Ifirst got them done! I’m so dedicated I recommend and bring in all my friends!

Review 2

Definitely my new nail salon! Everyone is so friendly and kind, I felt so welcomed! B did anamazing job on my nails. He made sure everything was perfect and happily changed somethingto make me happy. I would highly recommend this place to anyone who wants A + work at atotally affordable price. Love it!!:)

Review 3Amazing service and nails. This is the second time I have been here, they did a perfect job again.They get it done fast yet with precision. Everyone is so friendly there too. Best nail salon I haveever been too. I’m glad I found it.

Review 4I really enjoy coming here to get my nails done. They do a wonderful job on both pedis and nails.It is nice and clean inside. They are very friendly and welcoming. It is worth it to stop in and tryit out.

Review 5

My first set of acrylics ever... I decided 27 years was a lot enough time to wait, and I’m SOhappy with them. I’m not a huge nail person, and was glad to stumble upon this salon. My nailtech was quiet, clean, and very detail-oriented. Very pleased with my experience here and Irecommend this place.

Review 6I called to make an appointment for later today for 3 adults and 2 kids and the man who answeredthe phone said ’we only have 2 techs today’ we can’t do that. Poor customer service and I nevereven went in.

Review 7Golden Nails has been my nail place for almost a year so it was surprising to see new management.However B did an AMAZING job on my coffin chrome nails and Nancy was extremely helpfulfiguring out how I wanted my nails done too. Definitely excited to keep coming back!

Review 8Seriously the best service I have ever gotten at a Tempe nail salon!! I walked in and they helpedme right away. Nancy helped me pick the perfect color and was very honest and up front abouteverything! I wanted something very natural and using the dip method, I love my nails!!

Table 12: Example summaries produced by different systems on Yelp data.

GoldThese are a very comfortable and cute sandal. This thong sandal goes with a variety of outfitsand the cushy sole allows for all day comfort. However, they do run a little small, sizing upprovides a better fit. Overall, a reasonably priced shoe that will last for years to come.

OursThese sandals are very cute and comfortable. They fit true to size and are very comfortable towear. They look great with a variety of outfits and can be dressed up or down depending on theoccasion.

Copycat I love these sandals.They are very comfortable and I can wear them all day without any discomfort.I wear them to work and they are comfortable to wear.

MeanSumI love these shoes. They are so comfortable and I love the style. They are very comfortable andthe perfect price! I would definitely recommend this product to anyone. They are comfortableand stylish.

LexrankI have been wearing White Mountain beaded sandals for a couple of years now and they arewonderful. I will never buy from white mountain again. I love White Mountain sandels. Lots ofcompliments every time I wear them.

Review 1I get constant compliments on these sandals. I order them every summer in a variety of colors.I had heel spurs and back problems so the cushy softness of these is the only thing I can wearcomfortably and the small wedge heel is perfect for my back.

Review 2

These thongs are fun, festive, flexible and surprisingly comfortable. I have very sensitive feetand I can wear these cuties all day. The arch support is great and there is a nice give in the sole. Ilove these so much I want to put a few pairs away in case they discontinue them. They go witheverything.

Review 3I have been wearing White Mountain beaded sandals for a couple of years now and they arewonderful. They are lightweight and cushion the feet when worn for long hours. They are alsobeautiful and usually hold up for two or more seasons.

Review 4This was great price for this cute sandal. Unfortunately, the toe piece was very hard and theywere a little narrow... unusual since I normally wear a B width. For the right person, they wouldprobably be fine. They just didn’t work for me.

Review 5I love White Mountain sandels. this is my 2nd pair of these shoes. I wore out the last pair after2yrs. They are very very blingy and I like that. Would I order another pair? you bet I would /will.

Review 6Item was too small, purchased for a friend their size 9 is smaller than the size 8 in the store. Sentit to the wrong address, and I can not seem to find anyone that will tell me where my bill is. Iwill never buy from white mountain again.

Review 7I lived in sandals that looked exactly like this but I thought they were by Bjorn. I couldn’t findthem anywhere, but found these (go figure). While they aren’t quite as comfy as my other ones, Ithink with a bit of breaking in they’ll be just fine. Lots of compliments every time I wear them.

Review 8Not only are these SUPER comfortable (yes, even between your toes), they look great with justabout anything I wear! I have been complimented on these daily!! I typically wear a 6 1 / 2, Iordered a 7 and they fit perfect!! I need more of these!! HIGHLY RECOMMENDED!!!

Table 13: Example summaries produced by different systems on Amazon data.

GoldThis is a perfect compact table that fits well in many places. The chairs are surprisingly verycomfortable as well. It is cute and perfect for smaller living quarters and the best part is assemblyis simple and straightforward.

OursThis is a very nice table set for the price. It was easy to assemble and looks great in the kitchen.The only problem is that it is not sturdy enough to hold a lot of weight. It would be nice if it hada little more weight to it so that it would not tip over.

Copycat This is a great table set for the price. It was easy to put together and looks great. The only thingis that the chairs are a little flimsy, but they are easy to assemble.

MeanSumThe table was very easy to assemble and was easy to assemble. The only thing I would say isthat the box is very small and not very sturdy. The table is very sturdy. I would recommend it toanyone looking for a sturdy table and to put on the wall.

LexrankThe table and chairs are very nice but not quite the color I expected (but I am getting used to it).The table and chairs are solid and sturdy! I received this table and chairs completely damaged.Table and chairs delivered by the carrier right on time and with no damage.

Review 1It was easy to put together and looks great. However, when the item was shipped to me, one ofthe backs of the chairs was broken. I just fixed it myself with wood glue. Its not even visiblenow. The rest of it was in perfect condition.

Review 2The table and chairs are very nice but not quite the color I expected (but I am getting used toit). Table and chairs delivered by the carrier right on time and with no damage. Very easy toassemble, but very difficult to get out of the box it was so well protected.

Review 3This table was super easy to put together. The table and chairs are solid and sturdy! The seats arevery comfortable. The table is the perfect size for our not so big kitchen. We are very pleasedwith this purchase.

Review 4Moved to smaller living quarters and this just fits the bill. Color is perfect and it was easy toassemble. One fault to find is that the top scratches easily. It even came with a scratch. Otherthan that it is fine.

Review 5

I love my new dining room set. The set is very sturdy, the walnut finish is a nice color.This setis great for a small area, kitchen nook.Would not recommend for a large eating area.Table issmall and so are the chairs.Yet strong enough to hold big boys and girls, thumps up, great price,packed well, arrived in a timely matter.

Review 6It fits perfectly in the kitchen at the office. My staff assembled it without any delay. Everyoneloves the dining set and they can’t believe I ordered it on-line. I made the measurements andmade sure of the dimensions of the room and the dining set and it’s a perfect fit.

Review 7I received this table and chairs completely damaged. The customer service experience with thiscompany was terrible. In my opinion, this set is cheap and overpriced. It’s not durable and notworth the money. Don’t waste your time.

Review 8

The box looked like it had been opened, and then re-taped for resell. One of the chairs wasbroken, and the broken piece was nowhere close to the originating piece. Possibly other piecesdamaged too, though didn’t bother looking, instead just re-taped it back up to be sent back. Ihope they don’t just resell it to someone else.

Table 14: Example summaries produced by different systems on Amazon data.