Upload File

power-law distributions in binned empirica_data

33
  a   r    X    i   v   :    1    2    0    8  .    3    5    2    4   v    1    [   p    h   y   s    i   c   s  .    d   a    t   a   -   a   n    ]    1    7    A   u   g    2    0    1    2 Submitted to the Annals of Applied Statistics POWER-LA W DISTRIBUTIONS IN BINNED EMPI RICAL DATA By Yogesh Virkar † and Aaron Clauset †,‡,∗ University of Colorado at Boulder † Santa Fe Institute ‡ Many man-made and natural phenomena, including the intensity of earthquakes, population of cities, and size of international wars, are believed to follow power-la w distributions. The accurate identification of power-law patterns has significant consequences for developing an understanding of complex systems. However, statistical evidence for or against the power-law hypothesis is complicated by large fluctua- tions in the empirical distribution’s tail, and these are worsened when information is lost from binning the data. We adapt the statistically principled framework for testing the power-law hypothesis, developed by Clauset, Shalizi and Newman, to the case of binned data. This ap- proach includes maximum-likelihood fitting, a hypothesis test based on the Kolmogorov-Smirnov goodness-of-fit statistic and likelihood ratio tests for comparing against alternative explanations. We evalu- ate the effectiveness of these methods on synthetic binned data with known structure and apply them to twelve real-world binned data sets with heavy-tailed patterns. 1. Int roducti on.  Po wer-la w distributions have attracted broad scien- tific interest  [36] both for their mathematical properties, which sometimes lead to surprising consequences, and for their appearance in a wide range of natural and man-made phenomena, spanning physics, chemistry, biology, computer science, economics and the social sciences [ 21,  23 ,  33,  13 ]. Qua nt iti es that fol lo w a power- law dis tri but ion are someti mes said to exhibit “scal e inv arianc e”, indica ting that common, small even ts are not qualitatively distinct from rare, large events. Identifying this pattern in em- pirical data can indicate the presence of unusual underlying or endogenous processes, e.g., feedback loops, network effects, self-organization or optimiza- tion, although not always [ 29]. Knowing that a quantity does or does not follow a power law provides important theoretical clues about the underlying generative mechanisms we should consider. It can also facilitate statistical extrapolations about the likelihood of very large events [ 7]. ∗ To whom correspondence should be addressed. AMS 2000 subject classifications:  Primary 62G32, 62-07; secondary 65C05 Keywords and phrases:  power-law distribution, heavy-tailed distributions, model selec- tion, binned data 1

Upload: jcaceresal

Post on 03-Jun-2018

221 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 1/33

http://www.imstat.org/aoas/
http://arxiv.org/abs/1208.3524v1
Page 2: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 2/33

Page 3: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 3/33

Page 4: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 4/33

Page 5: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 5/33

Page 6: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 6/33

Page 7: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 7/33

Page 8: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 8/33

Page 9: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 9/33

Page 10: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 10/33

Page 11: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 11/33

Page 12: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 12/33

Page 13: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 13/33

Page 14: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 14/33

Page 15: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 15/33

Page 16: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 16/33

Page 17: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 17/33

Page 18: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 18/33

Page 19: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 19/33

Page 20: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 20/33

Page 21: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 21/33

Page 22: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 22/33

Page 23: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 23/33

Page 24: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 24/33

Page 25: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 25/33

Page 26: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 26/33

Page 27: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 27/33

Page 28: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 28/33

Page 29: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 29/33

Page 30: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 30/33

Page 31: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 31/33

Page 32: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 32/33

http://1.usa.gov/cCcwTg
http://bit.ly/wG8Psl
Page 33: Power-law Distributions in Binned Empirica_data

8/13/2019 Power-law Distributions in Binned Empirica_data

http://slidepdf.com/reader/full/power-law-distributions-in-binned-empiricadata 33/33

mailto:[email protected]
mailto:[email protected]
http://bit.ly/MhLdt6
http://1.usa.gov/Lj7cC9
http://bit.ly/MezSZ6

LONG TAIL ECONOMICS IMPLICATIONS OF POWER LAW DISTRIBUTIONS

Robust estimation of inequality from binned incomes

Power laws, Pareto distributions and Zipf’s law

BENFORD’S LAW, FAMILIES OF DISTRIBUTIONS AND A …johnmorrow.info/pdf/benford.pdf · BENFORD’S LAW, FAMILIES OF DISTRIBUTIONS AND A TEST BASIS John Morrow CEP, London School of

SPS 3D Binned Attributes

Classification Using Binned NMR Spectral Data

Power Law Size Distributions - Principles of Complex ... · Power Law Size Distributions Definition Examples Wildvs.Mild CCDFs Zipf’s law Zipf⇔CCDF References 20 of 36 Size distributions

Power Law Size Distributions - Principles of Complex

gauss's law applied to cylindrical and planar charge distributions

Multifractal Power Law Distributions - Yale University

Empirical Methods for Dynamic Power Law Distributions in ... · Empirical Methods for Dynamic Power Law Distributions Rank-based, nonparametric methods characterize general power

Power laws, Pareto distributions and Zipf’s lawtuvalu.santafe.edu/~aaronc/courses/7000/readings/... · Power laws, Pareto distributions and Zipf’s law ... When the probability

Deep Action Unit Classification using a Binned Intensity

Sitation' distributions and Bradford's law in a closed ...hera.ugr.es/doi/14977710.pdf · ª Sitationº distributions and Bradford’s law in a closed ... The sitation distributions

Clauset, A Et Al_2009 - Power-Law Distributions in Empirical Dada

binscatter: Binned Scatterplots in Stata - Michael Stepner ·  · 2018-03-08binscatter: Binned Scatterplots in Stata Michael Stepner MIT August 1, 2014 Michael Stepner binscatter

Using 3X3 Binned Mode for MOS Cameras for Fast(er) Imaging

Stat 1510: Sampling Distributions. Agenda 2 Parameters and Statistics Statistical Estimation and the Law of Large Numbers Sampling Distributions

Power laws, Pareto distributions and Zipf’s lawtuvalu.santafe.edu/~aaronc/courses/5352/readings/... · Power laws, Pareto distributions and Zipf’s law M.E.J. NEWMAN* Department

Binned C ZZ and systematics for ZZ->4l

Power laws, Pareto distributions and Zipf’s la · PDF filePower laws, Pareto distributions and Zipf’s law ... laws in a variety of systems and describe some of the ... from a power-law

Power-Law Size Distributions - University of Vermont€¦ · Your turn—ideal: 0 20 40 60 80 100 Power-Law Size Distributions Our Intuition Definition Examples Wild vs. Mild CCDFs

Black-body radiation: Planck distribution (Rayleigh-Jeans, Wien distributions) Wien’s Law

Power-Law Size Distributions Power-Law Size Distributions€¦ · Power-Law Size Distributions Our Intuition Definition Examples Wild vs. Mild CCDFs Zipf’s law Zipf , CCDF References

2D binned likelihood source detection

Fractality and Power Law Distributions: Shifting

Producing Power-Law Distributions and Damping Word Frequencies

Power-Law Distributions in Empirical Datacslui/CMSC5734/Clauset_Shalizi...Power-law distributions come in two basic flavors: continuous distributions gov erning continuous real numbers

Power-law distributions in binned empirical data · POWER-LAW DISTRIBUTIONS IN BINNED EMPIRICAL DATA 3 thus, such quantities are not well characterized by quoting a typical or av-erage

BENFORD’S LAW, FAMILIES OF DISTRIBUTIONS AND A TEST BASISjmorrow.net/projects/benford/benfordMain.pdf · BENFORD’S LAW, FAMILIES OF DISTRIBUTIONS AND A TEST BASIS JOHNMORROWy

Power-Law Size Distributions - University of Vermont...Power-Law Size Distributions Our Intuition Definition Examples Wild vs. Mild CCDFs Zipf’s law Zipf ,CCDF References 7 of 43

Nadiah Alanazi1 24.2 Gauss’s Law 24.3 Application of Gauss’s Law to Various Charge Distributions

Producing Power-Law Distributions and Damping Word ...€¦ · tics that produce power-law distributions. Section 5 shows how some other Bayesian language models can be viewed as

Power-Law Distributions in Empirical Data - Tuvalutuvalu.santafe.edu/~aaronc/courses/5352/readings/Clauset... · 2013-09-19 · POWER-LAW DISTRIBUTIONS IN EMPIRICAL DATA 663 Box 1:

Beehive: Exploiting Power Law Query Distributions …Beehive: Exploiting Power Law Query Distributions for O(1) Lookup Performance in Peer to Peer Overlays Venugopalan Ramasubramanian