the infrastructure of the libreoffice project · mediawiki, etherpad, planet, plone internally:...
TRANSCRIPT
1
The infrastructure ofThe infrastructure ofthe LibreOffice projectthe LibreOffice project
Florian EffenbergerAlexander WernerLibreOffice ConferenceMilan, September 26, 2013
2
Why do we need infrastructure?
3
We are a “virtual” project
the community (“the project”)millions of users worldwidehundreds of contributors around the globeactive 24 hours a day, 7 days a weekacross all languages and time zones→ “the project never sleeps”
the foundationthe board is located worldwidewe have no physical office
4
Infrastructure...
...is best when you don't recognize it
...is required 24x7
...has to be dynamic scaleable and stable
...enables the community to work
...is the framework for community work
Infrastructure provides the playgroundInfrastructure provides the playgroundwhere all the magic happenswhere all the magic happens
5
Where we came from
before LibreOffice: proprietary frameworkmainly tied to one IT service providermany integrated services plus some externalinfrastructure defined community structure
with LibreOffice: open and free systemcommunity defines infrastructurewe had the “luxury” of starting from zero
6
Our credo
Our infrastructure Our infrastructure development reflects thedevelopment reflects the freedom, openness and freedom, openness and transparencytransparency that TDF that TDF
stands forstands for
7
What we are cooking with
8
Three-tier approach
dedicated servers rented by TDFmailing lists, websites, wiki, online help, MirrorBrain, rsync, CMS, WebDAV, OTRS, blog planet, Torrent tracker, AskBot, forum, Gerrit, extensions & templates – most of things seen by the public6 physical servers and up to 29 virtual machinesIntel Xeon Quad-Core, 32 GB RAM, SATA disks, Gbit
dedicated servers hosted by third partiesbuildbots and tinderboxes for compiling, but also e.g. Pootle
external services managed by othersBugZilla, developer mailing lists, blog, conference callslibreoffice.org accounts for mail, Jabber/XMPP and SIP
9
Some numbers
team of 11 administrators from 4 countries
6 TDF servers, up to 29 VMs, plus external stuff
monthly ~8 TB trafficw/o download & development
112 mirrors worldwide for downloads
127 mailing lists in 35 languages
18,000+ subscribers on non-developer lists
in total ~170.000 e-mails in non-developer lists
10
Open, free and transparent
11
Including the community
all knowledge is within the communityregular admin meetings in real life and via phoneprofessional documentation using Sphinx
everything is based on free softwarewe contribute back to other projects
PyMIME mailing list filter (stripping HTML)translations and bug reports for mlmmjSaltStack modulesMirrorBrain IPv6 repository also used by others
12
Building on reliable free software
Ubuntu 12.04 LTS as base system, w/ LVM+RAIDPostfix, Dovecot, amavisd, SpamAssassin, ClamAVMailman and mlmmj for mailing listsMySQL and PostgreSQL as databasesApache/Nginx with PHP, plus SilverstripeMirrorBrain and Mirmon for mirror managementMediaWiki, Etherpad, Planet, Ploneinternally: Nagios, OTRS, BackupPC, ownCloudall services are IPv6-enabled
13
MirrorBrain - the mirror mastermind
C++ Apache module, Perl and Python scriptsdatabase of mirrors w/ respective (priority)selection of best download server based on GeoIPfiles are automatically scanned, indexed and hashedGPG signatures are verified automaticallylog files for download statistics are generatedthree-tier mirroring system
12 fastest hosts are push mirrors with “bit flip”47 staging mirrors with “bit flip”53 regular mirrors
14
Glimpse into the future
15
Migration plans
evaluating Debian 7.0 as second base OSmigrating MySQL to PostgreSQLfor replication and performance reasonsconverting from LXC to KVMfor better guest isolationand virtualization of Windows hostsmigrating from Apache to Nginx for performanceevaluating Amazon EC2 cloud for certain hosts→ plan to migrate to fully virtualized services later
HA with failover and loadbalancingDRBD, Gluster
16
Central deployment with SaltStack
infra needs to keep pace with community→ organic growth to provide services in timeright now, handcrafted scripts for initial setupin the future: SaltStack as deployment solution
one “Salt master” server with configuration repositoryservers and virtual machines are “Salt minions”deployment and regular configuration updatecentrally managed user database (without LDAP)base system already done, setup in ~5 minutes→ optionally including automatic creation of cloud instances
17
Join us, it's fun!
18
Communication channels
#tdf-infra at [email protected] listsubscribe with [email protected]
own wiki category at https://wiki.documentfoundation.org/Inframail the admin team at [email protected]
All text and image content in this document, unless otherwise specified, is licensed under the Creative Commons Attribution-Share Alike 3.0 License
This does not include the LibreOffice name, logo, or icon
19
Thank you for your attention!
Any questions?E-Mail: [email protected]
[email protected]: @floeff / @alxwerner