the artful business of data mining: computational statistics with open source tools

The Artful Businessof Data Mining

Computational Statistics with Open Source Tool

Wednesday 20 March 13

David Coallier@davidcoallier


Data ScientistAt Engine Yard (.com)


Find Data


Clean Data


Analyse Data?


Analyse Data


Question Data


Report Findings


Data Scientist


Data Janitor


Actual Tasks


“If your modelis elegant, it’s probably wrong”


“The Times they area-Changing”

— Bob Dylan


Python & R


SciPyhttp://www.scipy.org


http://www.scipy.org

http://www.scipy.org

scipy.stats


scipy.statsDescriptive Statistics


from scipy.stats import describe

s = [1,2,1,3,4,5]

print describe(s)


scipy.statsProbability Distributions


ExamplePoisson Distribution


f (k;λ) = λ ke−k

k!for k >= 0


import scipy.stats.poissonp = poisson.pmf([1,2,3,4,1,2,3], 2)


print p.mean()print p.sum()...


NumPyhttp://www.numpy.org/


http://www.numpy.org/

http://www.numpy.org/

NumPyLinear Algebra


1 00 1

⎛⎝⎜

⎞⎠⎟


import numpy as npx = np.array([ [1, 0], [0, 1] ])vec, val = np.linalg.eig(x)np.linalg.eigvals(x)


>>> np.linalg.eig(x) ( array([ 1., 1.]), array([ [ 1., 0.], [ 0., 1.] ]) )


MatplotlibPython Plotting


statsmodelsAdvanced Statistics Modeling


NLTKNatural Language Tool Kit


scikit-learnMachine Learning


from sklearn import treeX = [[0, 0], [1, 1]]Y = [0, 1]clf = tree.DecisionTreeClassifier()clf = clf.fit(X, Y)

clf.predict([[2., 2.]])>>> array([1])


PyBrain... Machine Learning


PyMCBayesian Inference


PatternWeb Mining for Python


NetworkXStudy Networks


MILKMOAR machine LEARNING!


Pandaseasy-to-use

data structures


from pandas import *x = DataFrame([ {"age": 26}, {"age": 19}, {"age": 21}, {"age": 18}])

print x[x['age'] > 20].count()print x[x['age'] > 20].mean()


RWednesday 20 March 13

RStudioThe IDE


lubridateand zoo

Dealing with Dates...


yy/mm/dd mm/dd/yyYYYY-mm-dd HH:MM:ss TZyy-mm-dd 1363784094.513425yy/mm different timezone


reshape2Reshape your Data


ggplot2Visualise your Data


RCurl, RJSONIOFind more Data


HMiscMiscellaneous useful functions


forecastCan you guess?


garchAnd ruGarch


quantmodStatistical Financial Trading


xtsExtensible Time Series


igraphStudy Networks


maptoolsRead & View Maps


map('state', region = c(row.names(USArrests)), col=cm.colors(16, 1)[floor(USArrests$Rape/max(USArrests$Rape)*28)], fill=T)


StorageWednesday 20 March 13

Oppose“big” Data


“Learn how

to sample”


ExperimentsWednesday 20 March 13

What DoYou Want to Answer?


UnderstandYour Audience


ScientificReporting


Busy-nessTime is money


PublicVisualisation


Best Visualisation,Bad Data


Best Forecastingmodels...Bad Visualisation


SeanchaíWednesday 20 March 13

FeelitWednesday 20 March 13

“Don’t be scared of bar charts.”


Mathematical StatisticsEngineering BusinessEconomicsCuriosity


davidcoallier.github.com@davidcoallier on Twitter


http://davidcoallier.github.com

http://davidcoallier.github.com

the artful business of data mining: computational statistics with open source tools

Technology