scaling compiled applications - highload 2013

scaling compiled applications

Joe Damato@joedamato

about me•Joe Damato

•San Francisco, CA

•i like:

•systems programming

•debugging

•performance analysis

@joedamato

timetobleed.com

sassanosystems.com

http://jenkins-ci.org/

Repeatable builds•the most important thing for compiled software

•set up a build server

•with jenkins (or another type of software)

•backup your built objects

•copying them to s3 is not a horrible idea

•regardless of source control system or branching strategy, ensure you can always rebuild any version of your software

Jenkins problems

•git plugin didn’t work on windows (maybe fixed?)

•branch building is painful, jenkins API can help

•getting windows build agent working is painful

why?

почему?

tools for...•RPMs

•DEBs

•Everything else

•windows installers (MSIs)

•other linux/unix like OS’s

•etc

chroot ??

chroot: an operation that changes the apparent root directory for the current running process [...].

A program that is run in such a modified environment cannot name (and therefore normally not access) files outside the designated directory tree.

(from wikipedia)

RPM

https://fedorahosted.org/mock/

mock

DEB

https://wiki.ubuntu.com/PbuilderHowto

pbuilder

Everything else

•KVM

•Amazon EC2

•other virtualization

KVM EC2

KVM

•Create a base image on disk

•Clone base image

•Boot the cloned image

•Do the build and copy built object out.

•Delete cloned image when done

•Base image is still pristine and can be reused.

Create builds in cleanroom

•Avoid contaminating builds with artifacts from previous builds.

•chroots help

•use mock or pbuilder for RPMs and DEBs

•KVM, EC2, or equivalent for everything else

•Always create pristine base images and do builds in a copy.

•Use SSDs

Tool problems

•git-buildpackage can’t set changelog distribution field

•signing key setup is really painful (especially for RPMs)

•deb naming scheme for packages is quite painful

•all tools are undocumented and very hard to actually use

•recent versions of KVM provided by ubuntu fail to boot VMs sometimes

two types of linking....

• dynamic linking

• static linking

static linking

•calls to functions in library are resolved at compile time.

•code from the library is copied into the resulting binary.

dynamic linking

•calls to functions are resolved at runtime.

•code for a library lives in it’s own object:

•libblah.so.4

•libblah.dll

http://www.akkadia.org/drepper/no_static_linking.html

why?

почему?

static linking•figure out which libraries your app needs

•pick a supported major release, if possible

•build and package this library

•link it statically against your binary during build

•you now have fewer stones to turn over when debugging

static linking•you will need to track your upstream deps

•you will probably want to package your upstream deps

•you can then point your chroots and build envs at private deb, rpm, etc repos in your internal infrastructure

•merging upstream changes in becomes its own project

why?

почему?

Use filessrc/

redhat5/

some_internal_thing.c

ubuntu1204/

some_internal_thing.c

debian6/

some_internal_thing.c

Use the build systemDetermine which

file to build at compile time.

Use modules

•break up ifdef soup into separate files

•use the build system to compile the right file at build time

•this seems obvious but many C libraries and programs are full of crazy ifdef soup.

Use modules

•very easy to fall down a rabbit hole breaking things apart

•can make the build process more complicated and harder to debug

why?

почему?

Capture debug symbols

•DEB and RPM can both output debug packages that contain debug symbols.

•output these packages.

•store these and make backups.

•(or just don’t strip your binary)

Use google-coredumper•you can use google-coredumper to catch

segfaults, bus errors, and other bad things.

•you can output a coredump when this happens.

•you can use this coredump and your debug symbols to figure out what is going on.

Plan for failure•Have a backup plan

•Capture debug symbols during your automated build process.

•Store them somewhere safe (and make backups).

•Capture coredumps (if possible).

•Use coredumps and debug symbols to figure out what happened.

Plan for failure

•can significantly increase complexity

•google coredumper can’t help if your kernel is buggy

•some linux distributions don’t allow ptrace

•google coredumper only supports linux

http://jenkins-ci.org/

Check things like...

•Is the binary actually statically linked?

•Does it get copied to the right path?

•Are the right config files autogenerated?

•Does the version string the program outputs match the package version?

•....

You also need...correctness testing.

RSpec is useful for this.

Automated Testing•It will be impossible to build and test every

change on every supported platform.

•Use your build server to do this for you.

•Test things like:

•installing/uninstalling the object

•object is actually statically linked

•correctness testing (RSpec can be useful)

Automated testing

•Fine line between not enough and too much

•Windows can be painful, but cygwin can help with scripting

•Easy to forget about testing the actual package install/removal

•Can be difficult to get working with branch builds

To summarize...

sassanosystems.com

@joedamato

спасибо

?