stupid buildout tricks

21
Stupid Buildout Tricks Ricardo Newbery Wednesday, November 9, 2011

Upload: ricardo-newbery

Post on 17-Dec-2014

2.740 views

Category:

Technology


0 download

DESCRIPTION

 

TRANSCRIPT

Page 1: Stupid Buildout Tricks

Stupid Buildout Tricks

Ricardo Newbery

Wednesday, November 9, 2011

Page 2: Stupid Buildout Tricks

First, a basic buildout[buildout]parts = pyramid myapp

[pyramid]recipe = zc.recipe.eggdependent-scripts = trueeggs = pyramidinterpreter = py

[myapp]recipe = collective.recipe.templateinput = templates/myapp.ini.inoutput = ${buildout:directory}/etc/myapp.inikey_a = value_akey_b = value_b

Wednesday, November 9, 2011

Page 3: Stupid Buildout Tricks

The problemMultiple environments (dev, qa, staging, production) requiring different buildout configurations.

$ bin/buildout -c dev.cfg

$ bin/buildout -c qa1.cfg

$ bin/buildout -c qa2.cfg

ad nauseum ...

The non-dev builds are performed by release engineers who have better things to do than to read complicated release notes to figure out the right buildout incantation for the current environment.

Wednesday, November 9, 2011

Page 4: Stupid Buildout Tricks

The solution

Add environment variables into each non-dev environmentand have buildout figure out which config is the right one.

Wednesday, November 9, 2011

Page 5: Stupid Buildout Tricks

First try...

Used a buildout extension to munge the buildout config dynamically. Similar to the way collective.recipe.python works.

Two issues:

1) The variant configs are hard to read.

2) Doesn't play well with mr.developer since "bin/develop" turns off extensions when reading the config.

This was painful so I redid this w/o the buildout extensions...

Wednesday, November 9, 2011

Page 6: Stupid Buildout Tricks

Nothing fancy...$ bin/buildout \ -c config/variant-staging.cfg \ buildout:directory=<<PATH TO BUILDOUT ROOT>>

The "buildout:directory" value fixes the problem where buildout assumes the build context is the directory where the main config resides.

Now, I just need a script to generate this call to buildout...

$ bin/environ

Let's make this script...

Wednesday, November 9, 2011

Page 7: Stupid Buildout Tricks

import osimport sysimport loggingimport subprocess

logger = logging.getLogger('buildout.variants')logger.setLevel(logging.INFO)handler = logging.StreamHandler()handler.setFormatter(logging.Formatter('%(message)s'))logger.propagate = Falselogger.addHandler(handler)log = logger.info

# Is config file specified on command line?config_file = Falseargs_in = sys.argv[1:]while args_in: op = args_in.pop(0) if op[:2] == '-c': config_file = True log("buildout.variants: Disabled.")

# If config file not specified, check for one matching the environmentargs = ['bin/buildout']if not config_file: environ = os.environ.get('BUILDOUT_ENVIRON') if environ: log("buildout.variants: Detected environment '%s'" %environ) config_file = 'config/variant-%s.cfg' %environ if not os.path.exists(config_file): log("buildout.variants: Not found '%s'" %config_file) config_file = False if config_file: args.append('-c%s' %config_file) args.append('buildout:directory=%s' %os.getcwd()) log("buildout.variants: Applying variant '%s'" %config_file) else: log("buildout.variants: No variants found.")

# Run buildoutargs = args + sys.argv[1:]subprocess.call(args)

Wednesday, November 9, 2011

Page 8: Stupid Buildout Tricks

# Is config file specified on command line?

config_file = Falseargs_in = sys.argv[1:]while args_in: op = args_in.pop(0) if op[:2] == '-c': config_file = True log("buildout.variants: Disabled.")

Wednesday, November 9, 2011

Page 9: Stupid Buildout Tricks

# Check the environment

args = ['bin/buildout']

if not config_file: environ = os.environ.get('BUILDOUT_ENVIRON') if environ: log("buildout.variants: Detected '%s'" %environ) config_file = 'config/variant-%s.cfg' %environ if not os.path.exists(config_file): log("buildout.variants: Not found '%s'" %config_file) config_file = False if config_file: args.append('-c%s' %config_file) args.append('buildout:directory=%s' %os.getcwd()) log("buildout.variants: Applying variant '%s'" %config_file) else: log("buildout.variants: No variants found.")

Wednesday, November 9, 2011

Page 10: Stupid Buildout Tricks

# Run buildout script

args = args + sys.argv[1:]subprocess.call(args)

Wednesday, November 9, 2011

Page 11: Stupid Buildout Tricks

release engineers = :-)

Only need to know one command to update a buildout

$ bin/environ

Environment buildouts can always be overridden.

$ bin/environ -c local.cfg

or

$ BUILDOUT_ENVIRON=staging bin/environ

Wednesday, November 9, 2011

Page 12: Stupid Buildout Tricks

Extra credit

All the documentation says to run buildout with...

$ bin/buildout

Let's not confuse the release engineers with yet another way to run a buildout so I just created a custom buildout script (generated by the bootstrap.py) that folds this environment hook into bin/buildout itself.

[customizing buildout script left as an exercise for the reader]

So no more 'bin/environ'. Just 'bin/buildout' again.

Release engineers even happier :-)

Wednesday, November 9, 2011

Page 13: Stupid Buildout Tricks

An alternative

Another way to get environment-aware buildout configurations is with mr.scripty,

http://pypi.python.org/pypi/mr.scripty

Allows defining values in a buildout part with regular python expressions. This is possibly a more general solution although the configs may be a little harder to read. You also can't adjust a buildout config "extends" value with this method.

Wednesday, November 9, 2011

Page 14: Stupid Buildout Tricks

Another problemStaging environment sucks.

Occasionally, stuff in python site packages in staging borks the buildout. I could work on fixing the staging environment but I've got better things to do.

So let's isolate the buildout from site packages.

Some possible solutions...

1) virtualenv

2) zc.buildout, version 1.5.x

Let's show the virtualenv method...

Wednesday, November 9, 2011

Page 15: Stupid Buildout Tricks

Release notes 2.0

Using virtualenv...

$ hg clone <path-to-repo> MyBuildout

$ virtualenv --no-site-packages MyBuildout

$ cd MyBuildout

$ bin/python bootstrap.py

$ bin/buildout

Wednesday, November 9, 2011

Page 16: Stupid Buildout Tricks

A virtualenv bootstrapVirtualenv must be installed, or we need to create a bootstrap.

$ virtualenv.create_bootstrap_script(extra_text)

The extra_text is just some python script to hook in extra behavior during a bootstrap.

extend_parser(optparse_parser): Add or remove options from the parser here.

adjust_options(options, args): Change options or args here.

after_install(options, home_dir): After everything is installed, this function is called.

See the virtualenv docs for details.

Wednesday, November 9, 2011

Page 17: Stupid Buildout Tricks

(cont.)>>> extra_text = """>>> def adjust_options(options, args):>>> >>> # Create virtualenv in current dir>>> args = ['.']>>> >>> # Isolate from system python site-packages>>> options.no_site_packages = True>>>>>> # Use distribute instead of setuptools>>> options.use_distribute = True>>> """>>>>>> import virtualenv>>> out = virtualenv.create_bootstrap_script(extra_text)>>> open('virtualenv.py', 'w').write(out)

Wednesday, November 9, 2011

Page 18: Stupid Buildout Tricks

Release notes 3.0

Virtualenv not installed in system python...

$ hg clone <path-to-repo> MyBuildout

$ cd MyBuildout

$ python2.6 virtualenv.py

$ bin/python bootstrap.py

$ bin/buildout

Wednesday, November 9, 2011

Page 19: Stupid Buildout Tricks

But wait, there's more

Add the following to the virtualenv bootstrap script,

def after_install(options, home_dir): if os.path.exists('bootstrap.py'): logger.notify('Running bootstrap.py') subprocess.call([os.path.join(home_dir, 'bin', 'python'), \ 'bootstrap.py'])

Now bootstrapping virtualenv also bootstraps buildout.

Wednesday, November 9, 2011

Page 20: Stupid Buildout Tricks

Release notes 4.0

$ hg clone <path-to-repo> MyBuildout

$ cd MyBuildout

$ python2.6 virtualenv.py

$ bin/buildout

Wednesday, November 9, 2011

Page 21: Stupid Buildout Tricks

Conclusion

* Much simpler release process... happy release engineers.

* Much simpler release notes... happy developers

* Joy all around.

Wednesday, November 9, 2011