![Page 1: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/1.jpg)
Scott ShawcroftJuly 22, 2009
http://oswatershed.org
![Page 2: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/2.jpg)
Scott Shawcroft●Class of 2009 ~ University of Washington
●Computer Engineer
●Creative Commons, Google and more Google
●OS Projects: touchd, Menzies, Annoamp, denu
●Linux since spring '04.
●LFS Gentoo Ubuntu Gentoo→ → →
![Page 3: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/3.jpg)
Watershed
![Page 4: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/4.jpg)
distrologyThe formal study of open source software distributions.
dis·trol·o·gy d -str l- -jĭ ŏ ə ē
Study of
![Page 5: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/5.jpg)
Gather release information from upstreamand downstream.
●Name
●Version
●Date
●Revision
Data Gathered
![Page 6: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/6.jpg)
●Name
●Codename
●Component
●Architecture
Data SourcesUpstream
Distributions/Repositories
●Directory Listings
●Sourceforge
●Experimental
●Future
●Current
●LTS
●Past
Branches
![Page 7: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/7.jpg)
Upstream/Downstream relationship metrics:
●% Obsolete
●# Obsolete
●Lag
Results
oswatershed.org●Per Package Data (badges)
●Per Distro Data
●Different Group Analysis
●Data Quality Tools
![Page 8: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/8.jpg)
Scott's Chosen 20alsa-utilscupsemacsfirefoxgccghostscript-gplgimpglibcgnome-desktopgnupg
httpd (apache)kdebaselinuxNetworkManageropensshpidginpostgresqlpythonrubyxorg-server
![Page 9: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/9.jpg)
Ubuntu/Gentoo (% obsolete)
![Page 10: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/10.jpg)
Ubuntu/Gentoo (# obsolete)
![Page 11: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/11.jpg)
Ubuntu/Gentoo (lag)
![Page 12: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/12.jpg)
LAMPPP (lag)
![Page 13: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/13.jpg)
LAMPPP (lag)
![Page 14: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/14.jpg)
Challenges●Lots of data.
●Comparing it all.● Normalizing names.● Determining obsoletion.
(aka understanding versions)
![Page 15: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/15.jpg)
Lots of Data
●9 Distributions● Each has its own custom crawl script.
●78,476 Total Packages● Mostly inflated by custom distro names. 10K – 15k
estimated distinct.
●735,859 Releases● Distinct package name and version combinations.
Skewed by different naming.
●2,463 Upstream Packages● 78 Sourceforge Sources● 106 Directory Sources● 3 Custom Scripts
![Page 16: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/16.jpg)
Distros must also deal with package branches. Gentoo uses 'slotting', most use new package names.
Upstream
Php
db
Ubuntu/Debian
Php3php4php5db4.2db4.3db4.4db4.5db4.6db4.7
Normalizing Names
![Page 17: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/17.jpg)
Ordering Versions
2008-10-02 20:24:00 2.62008-11-07 04:30:00 3.0rc22008-11-21 02:50:00 3.0rc32008-12-03 20:37:00 3.02008-12-05 05:57:00 2.6.12008-12-13 14:43:00 2.4.6c12008-12-13 16:47:00 2.5.3c12008-12-19 16:14:00 2.4.62008-12-19 16:15:00 2.5.32008-12-23 14:28:00 2.5.42009-02-14 01:10:00 3.0.1
Should we obsolete 2.6.1 with 2.4.6?
●All new releases obsolete old ones.●Any new downstream release that doesn't match an upstream release is completely fresh.
Original ordering based only on release date.
Problems:
![Page 18: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/18.jpg)
Ordering Versions1) Split the version.2) Build a tree with children sorted by release date.
NewerOlder
For Python version 2.6.1:
![Page 19: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/19.jpg)
ConclusionsRelease cycle does not effect overall freshness.
Package management includes many hacks.●Many downstream to one upstream.
● Libpng libpng, libpng-dev→
●Mangling package names. (Slotting)● Php php3, php4, php5→
●Mangling version numbers.● Mysql 5.1.30really5.0.83
![Page 20: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/20.jpg)
OSW Future
Need volunteers and supporters!
●User centric features brainstormed.● Add sources.● Verify sources.● Link packages.● Custom package groups.
●More eyes on all of the data and code.
●Publicity! Online articles, blogs and badges.
![Page 21: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/21.jpg)
●scott.shawcroft @ gmail.com
●oswatershed.org
●github.com/tannewt/open-source-watershed
Links
![Page 22: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/22.jpg)
Appendix
●Arch●Debian●Fedora●Gentoo●OpenSUSE●Sabayon●Slackware●Ubuntu
![Page 23: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/23.jpg)
Arch
![Page 24: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/24.jpg)
Debian
![Page 25: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/25.jpg)
Fedora
![Page 26: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/26.jpg)
Gentoo
![Page 27: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/27.jpg)
OpenSuse
![Page 28: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/28.jpg)
Sabayon
![Page 29: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/29.jpg)
Slackware
![Page 30: Scott Shawcroft July 22, 2009tannewt.org/slides/shawcroft-oscon_presentation.pdf · Package management includes many hacks. Many downstream to one upstream. Libpng libpng, libpng-dev→](https://reader034.vdocument.in/reader034/viewer/2022042412/5f2c8ce370210f203d3b27ce/html5/thumbnails/30.jpg)
Ubuntu