data science weekend 2017. cleverdata. text mining of beauty blogs: о чем говорят...

TextminingofBeautyBlogs:

TextminingofBeautyBlogs:Очемговорятженщины?

АртемПросветовDataScientist,CleverDATA

cleverdata.ru |[email protected]

RawblogdataRawdata:98,496 pagesinformatof~1,000,000files.Readyforanalysis:58,719 Englishpages(59.6%)

40.4%data:emptypagesandpageswitherrors,notEnglishpages(23,461),photo/videopageswithouttext(2,315),articlesfromtechcrunch.com(3,402)


From60k ofpages→~2000 authors.

Pages→Authors


Meanblogpostsize(inwords)

One can distinguish 2 populationsof bloggers:

• twitter style' authors with shortposts (~20%)

• full-length bloggers with 200-500 mean words per post(~80%)


Used APIs and services:

- Sentity (https://sentity.io/)

- Twinword (https://www.twinword.com/)

- Textualinsights (http://www.textualinsights.com/)

- VivekN (https://github.com/vivekn/sentiment-web)

Sentimentanalysis


Sentimentanalysis

• - the resulting sentiment rate is basedon 4 independent rate systems.

• - the majority of the blogs have positiveemotion rate.

• - the mean sentiment rate is «positivewarm» 0.72.

• - all this results are intuitively consistentand are in a good agreement withmanual tests


We used afew traffic rank systems:

Estimation of blog efficiency

• Alexa Rank,that basically audits and makes public the frequency ofvisits on various Web sites.

• Yandex Thematic Citation Index (TIC),that determines the“credibility”of Internet resources based on aqualitative assessmentof links to other sites.

• Google Page Rank,that works by counting the number and qualityof links to blog to determine arough estimate of how important thewebsite is.


Content relevance rate is based on fuzzy string matching:

- Every company product name was string matched with all amount of blogs.- String matching is based on Levinstein's metric.- Pages with 90%matching rate were marked up.- Tests with direct brand name matching showed that we get about 90-100%accuracy on each product name deppends on words in title.- The result relevance rate for each author is summed from all marks ofhis/hers pages.

RelevanceRate


Levenshtein distance is astring metric for measuring the difference betweentwo sequences.

Informally,the Levenshtein distance between two words is the minimumnumber of single-character edits (i.e.insertions,deletions or substitutions)required to change one word into the other.

Levinshtein distance between 'beer'and 'bread'is 44/100

Levenshteindistance


Themostactiveauthors

writewithsentiment

rateinshortrange:

0.74+/- 0.03

Sentiment rate

Blog

siz

e (p

ages

)

SentimentsvsBlogsize


Themostdiscussed

blogshavemiddle-

sizeauthors.

Log(Blog size)

Mea

n di

scus

sion

DiscussionvsBlogsize


Again,2kindsofbloggers:

- 'twitterstyle'authorswithshortposts

- full-lengthbloggers

Log(mean words per page)

Log(

Blog

siz

e)

WordsvsPages


fyouwanttomakeabigdiscussion,youshouldpraisesomething.

Allhighlydiscussedauthorsaresentimentpositive(>=0.4)

Sentiment rate

Mea

n di

scus

sion

DiscussionvsSentiments


We use Klout service to rank authorsaccording to online social influence.Klout measures the size of auser'ssocial media network and correlates thecontent created to measure how otherusers interact with that content.

- the median Klout score is 40.1

UsingofKloutscoreforbloggers


One can distinguish apopulationof beginner bloggers with lowKlout score,that have tendencyto amplification of sentiments.

Sentiment rate

Klou

tsco

re

SentimentsvsKloutscore


• Amountofblogpages

• Meandiscussionsize

• AlexaRank +YandexTIC +GooglePageRank

• Relevancerate

• Sentimentrate

• Klout score

FinalAuthorRatingisbasedon


4independentsentimentratingsystemsarecombined

AlexaRank

YandexThematicCitationIndex

GooglePageRank

listofmostPReffectiveauthors

Pragmaticstatisticalinformation

keyrecommendationsforblogger

resultingsentimentrateisfullyconsistentwithtests

Blogefficiencyrating

Blogrelevancerating

Sentimentanalysis

Makeyourdataclever

Basedonfuzzystringmatching

Blogratinginaccordancetomentionsofcompanyproductsintext


Name Url Sentiment Pages MeanComments

HayleyCarr http://www.londonbeautyqueen.com 0.71 229 10.9

Luzanne http://pinkpeonies.co.za 0.77 66 68.3

Allison http://www.neversaydiebeauty.com 0.70 182 42.9

MicaKelly,Beth,JessicaDiner http://blog.birchbox.co.uk 0.74 196 0.26

Poonam http://beautyandmakeupmatters.com 0.78 142 4.3

Silvie http://mysillylittlegang.com 0.74 571 0.64

TOPRatedAuthors


Testingtheresult

HayleyCarr (TopRatedAuthor):“BlaBlaBla isdefinitelyabrandtobereckonedwith...AlloftheBlaBlaBla productshavemultiplepurposes,aswellassmellingandfeelingfabulous;thepackagingiscleanandfreshwhilststilllookinggreatinyourbathroom,aswellashavinguniqueapplicationmethodsthatonlyaidtheproductperformance...It'sdefinitelyworthcheckingoutthisgrowingbrand,beforeitstartstakingovertheworld.“


Authors←→Products


Inordertoassociateabloggerwithaproductwemust:

• Findproductsforpromotion

• Findmaintopicsofeachblogger

• Matchtopicsofeachbloggerwith productnames

• Findbestcombinationsofbloggerandproduct


Findingthemostperspectiveforpromotionproducts


Let'sbuilddocument-termmatrix,whereeachrowisadocument,eachtermisacolumnandacolorintensityindicatesthatatermappearsinadocumentatleastonce.

WecanuseTF-IDFmethodtogetdocument-termmatrix.

Findingtopics:thedocument-termmatrix


Findingtopics:TF- IDF

• TermfrequencyTF(t,d) isthenumberoftimesthattermtoccursindocumentd.

• Theinversedocumentfrequency(IDF)isameasureofhowmuchinformationthewordprovides,thatis,whetherthetermiscommonorrareacrossalldocuments.

• Termfrequency–inversedocumentfrequency,isanumericalstatisticthatisintendedtoreflecthowimportantawordistoadocumentinacollectionorcorpus.


• NMFisavariantofMatrixFactorizationwherewestartwithamatrixD withdocument-termmatrix,andconstraintheelementsofW andT tobenon-negative.

• LetsusinterpreteachrowoftheT matrixasatopic.

Topicextraction:NMF


• Foreachauthorwebuilddocument-termmatrix.

• Foreachdocument-termmatrixweperformmatrixfactorizationandfindmaintopics

• Foreachproductwematchproductnamewithmaintopicsofauthorandfindtherateofintensity.

• Ifauthorhaveexactproductnameinoneofhis/herstitles,wesettherateofintensityto0 (theauthorhasalreadymadereviewofthetheproduct).

Topicextraction


Thusforeachpairofauthor-productwefindrateofintensityandwecanvisualizeitinformofheatmapwhereproductsaresortedbymeanrateof

intensityandauthorsaresortedbyauthorrating:

Note:themostratedauthorsarehighlyintensiveonmatrix

Theintensitymatrix


Nextweextractthemostresonancepeaksfromproduct-authormatrixofintensity.Aftereachpeakextractionthecolumnwithapeakisdropped,soforeachauthorwegetonlyoneproduct.

Weneedtobuildrecommendationsonlyfor4productsandwecanselect40bestratedauthorsforthistask.

Theintensitymatrix







• Profit!


BlaBlaBlaBodyOil Allison http://www.neversaydiebeauty.com

BlaBlaBlaWrinkleRepair CindyBatchelor http://mystylespot.net

BlaBlaBlaFaceSerum MariePapachatzis http://iamthemakeupjunkie.blogspot.ru

BlaBlaBlaFaceOil Emily- StyleLobster http://stylelobster.com

Theresultingassociations

data science weekend 2017. cleverdata. text mining of beauty blogs: о чем говорят...

Data & Analytics