jenkins entwined with deployment and pragmatism

Novadex GmbH 2013. All rights reserved, also regarding any disposal, exploita>on, reproduc>on, edi>ng, distribu>on, as well as in the event of applica>ons for industrial property rights.

Jenkins entwined with deployment and pragma>sm

Eric Ritchie DevOps Engineer Novadex +49 (0) 7142 / 788905-‐0 [email protected]


Eric Ritchie bids you welcome!

20/11/13 2

§  currently working as DevOps Engineer and System Administrator at Novadex GmbH

§  a Zend Framework and PHP ZCE & MySQL DBA

§  a Linux System Administrator with 18 years experience

§  a PHP developer since version 3.0.18 (or so)

§  known to speak at a conference or two

Never heard of him? Aside from looking like this, he is:


LeaerMaschine: Custom-‐tailored Communica>on

Personalized data stream Cross-‐Channel Output

Collabora>on

Portal Op>on Campaign management

CRM integra>on

Content management E-‐commerce integra>on

Data protec>on Security

20/11/13 3


What I read in a book once…

20/11/13 4

„DevOPs should aim for 10 deployments per day“


Novadex’s key tool: Jenkins

20/11/13 5

§  Great for Con?nuous Integra?on

§  Can hide complex processes behind a nice buDon

§  Deployment remains one of those complex processes §  Requires a lot of scrip>ng/manual work §  This risks our deployment goal


An example of that scrip>ng

20/11/13 6

<target name="package"> <echo message="...using UPSTREAM_BUILD_NUMBER received from triggering job for packagenaming. UPSTREAM_BUILD_NUMBER is [${UPSTREAM_BUILD_NUMBER}]"/> <!-‐-‐ leaermaschine-‐complete packaging -‐-‐> <!-‐-‐ wipe the dir from possible previous copy opera>on -‐-‐> <delete includeemptydirs="true"> <fileset dir="${package.contents.basedir}" includes="**/*"/> <fileset dir="${package.basedir}/DEBIAN" includes="**/*"/> </delete> <!-‐-‐ prepare debian control file -‐-‐> <copy todir="${package.basedir}/DEBIAN" overwrite="true"> <fileset file="resource/package/test/control"/> <filterset begintoken="[[" endtoken="]]"> <filter token="version" value="${UPSTREAM_BUILD_NUMBER}-‐1"/> <filter token="name" value="leaermaschine-‐complete"/> <filter token="depends" value=""/> <filter token="descrip>on" value="Complete Leaermaschine applica>on"/> </filterset> </copy> <!-‐-‐ copy pos>nstalla>on script -‐-‐> <copy file="${basedir}/resource/package/test/pos>nst_lm" tofile="${package.basedir}/DEBIAN/pos>nst"/> <chmod file="${package.basedir}/DEBIAN/pos>nst" perm="775"/> <copy todir="${package.contents.basedir}"> <fileset dir="${export.basedir}"> <snip> </fileset> </copy> <exec executable="sudo" dir="${working.basedir}" failonerror="true"> <arg line="sudo dpkg -‐b ${package.basedir} ${dist.basedir}/leaermaschine-‐complete_${UPSTREAM_BUILD_NUMBER}-‐1_all.deb"/> </exec> </target>


Deployment v1.x?

20/11/13 7

§  Applica?on bundled into a .deb package §  As shown on the previous slide (yeah, that’s what it did) §  Shell scrip>ng available for in deployment manipula>ons

§  SCP’d to target machine, installed via SSH exec §  Need to set up SSL cer>ficates to allow password free access


Problems?

20/11/13 8

§  Who is comfortable with bash shell scrip?ng?

§  What happens when something goes wrong? §  Ever tried to roll back a copied .deb package in a hurry?

§  Not really all that cluster friendly §  Ok, we have Chef/Puppet if we like overkill

§  Disk cluDer §  Each version of the .deb package is cached

§  Need to grant unlimited access to live servers, argh! §  Remember that SSL cer>ficate we installed?

§  What if we need to support Red Hat also? §  Hypothe>cal in the case of Novadex, but s>ll a valid problem.


Enter Zend Server deployment (and me)

20/11/13 9

Cluster/Cloud ready, plaxorm agnos>c, PHP scrip>ng and version handling with integrated roll back.

Yes, I used to work for Zend ;)


Tale of the liale known tool: zs-‐manage

20/11/13 10

§  We all have heard about the Web API, right? §  You may be forgiven for thinking it is a new thing

§  We have had command line deployment tools since day one §  But nobody knew about them, which is a pity

§  zs-‐manage has many advantages §  Can leverage all deployment features §  Does not rely on SSL cer>ficates for security §  Same commands for local/cluster/cloud deployment

§  But beware, there are some caveats §  Pragma>sm to follow...


Let’s empower Jenkins…

20/11/13 11

<!-‐-‐ retrieve the app.id for the applica>on, specified by app.name. may be empty if app hasn't already deployed -‐-‐> <exec executable="bash" dir="${basedir}" outputproperty="app.id" errorproperty="err" failonerror="true"> <arg value="-‐c" /> <arg value="/usr/local/zend/bin/zs-‐manage app-‐get-‐status -‐N ${zs.api.name} -‐U ${zs.url} -‐K ${zs.api.key} 2>&1 | grep "APPINFO\\s.*${app.name}.*" | awk '{print $2}'" /> </exec> <if> <equals arg1="${app.id}" arg2="" /> <then> <echo message="Couldn't determine app.id for app.name '${app.name}' -‐ preparing ini>al deployment of applica>on ..." /> <!-‐-‐ deploy applica>on via zend server command -‐-‐> <exec executable="/usr/local/zend/bin/zs-‐manage" dir="/tmp" failonerror="true"> <arg line="app-‐deploy -‐u "NDX_ENV=${zs.ndx.env}" -‐c -‐b hap://${app.name}/ -‐a ${app.name} -‐p ${zs.package.des>na>ondir}/${zs.package.name} -‐N ${zs.api.name} -‐K ${zs.api.key} -‐U ${zs.url}" /> </exec> </then> <else> <echo message="app.name '${app.name}', app.id is ${app.id} -‐ preparing the update of applica>on ..." /> <!-‐-‐ update applica>on via zend server command -‐-‐> <exec executable="/usr/local/zend/bin/zs-‐manage" dir="/tmp" failonerror="true"> <arg line="app-‐update -‐u "NDX_ENV=${zs.ndx.env}" -‐i ${app.id} -‐p ${zs.package.des>na>ondir}/${zs.package.name} -‐N ${zs.api.name} -‐K ${zs.api.key} -‐U ${zs.url}" /> </exec> </else> </if>


Crea>ng the package to be deployed

20/11/13 12

§  We have a choice §  Zend Studio -‐ a manual process §  zdpack -‐ Zend standard §  zip -‐ needs no introduc>on

§  Novadex have used zip at ?mes §  But currently use the following code

<exec executable="/usr/local/zend/bin/zdpack" dir="${working.basedir}" failonerror="true"> <!-‐-‐ NOTE: name of the created package is the name specified in deployment.xml name-‐tag -‐-‐> <arg line="pack -‐-‐src-‐dir=${package.contents.basedir} -‐-‐scripts-‐dir=${basedir}/resource/zs_deploy_new/scripts -‐-‐package-‐descriptor=${working.basedir}/deployment.xml -‐-‐output-‐dir=${working.basedir}" /> </exec> <!-‐-‐ rename created package to reflect version of package contents -‐-‐> <move file="${working.basedir}/${app.name}.zpk" tofile="${working.basedir}/${zs.package.name}" overwrite="true"/> <!-‐-‐ copy the new package from temp loca>on to package directory -‐-‐> <copy file="${working.basedir}/${zs.package.name}" todir="${zs.package.des>na>ondir}" overwrite="true"/>


Sadly, size does maaer

20/11/13 13

§  You don’t leave your doors unlocked when you leave home §  Most admins are equally careful with their servers

§  Web API could be an aDack vector §  Which is why Zend take steps to secure it §  Web API calls must be signed with, amongst other informa>on, a >mestamp

§  To help prevent replay aDacks, calls must complete within 300 seconds §  This security feature can shoot us in the foot §  In short, the network speed determines our maximum packet size


Working around the size issue

20/11/13 14

§  No perfect solu?ons, yet §  I am working on Zend ;)

§  But we can apply some pragma?sm §  Package can be broken into sub-‐packages §  Zend Deployment has a new Library feature, use it §  Pull instead of push §  Upgrade the pipe §  Time travel §  Or, emulate the GUI


Our solu>on

20/11/13 15

<echo message="copying ${zs.package.name} to ${remote.hostname}" /> <scp file="${zs.package.des>na>ondir}/${zs.package.name}" todir="${remote.user}@${remote.hostname}:${remote.dir}" keyfile="${remote.keyfile}" /> <echo message="app.name '${app.name}', app.id is '${app.id}' -‐ preparing the update of applica>on ..." /> <!-‐-‐ execute zs-‐manage locally on host where package should be deployed! -‐-‐> <sshexec host="${remote.hostname}" username="${remote.user}" keyfile="${remote.keyfile}" command="/usr/local/zend/bin/zs-‐manage app-‐update -‐u "NDX_ENV=${zs.ndx.env}" -‐i ${app.id} -‐p ${remote.dir}/${zs.package.name} -‐N ${zs.api.name} -‐K ${zs.api.key} -‐U ${remote.webapi.url}" />

GUI emula?on: It works for us

Just remember: SCP/SSH may not be allowed by your opera>ons team. You may need another solu>on ! !


Memory and lots of it

20/11/13 16

§  Ques?on: Where does all this deployment data go? §  Hint: The WebAPI is part of the GUI

§  The GUI allocates memory equivalent to three ?mes the packet size §  Since the data is loaded, copied and finally unpacked (don’t ask)

§  Where is the GUI’s memory limit configured? §  Not in the GUI, that would spoil the fun ;) §  Can you guess?

Answer: The memory_limit variable in: /usr/local/zend/gui/lighapd/etc/php-‐fcgi.ini


Aaack of the cluster

20/11/13 17

§  Zend deployment can be hard to synchronise §  In other words, you cannot guarantee when all nodes will run the

same version of the applica>on

§  An extreme case is the “run once” node §  Database migra>ons are usually performed here, so deployment is slower

§  This caused problems with our “In maintenance” page §  Time for some extreme pragma>sm…


I am a Job Queue fan… so I wrote this

20/11/13 18

// Sanity check if (!class_exists('ZendJobQueue')){ logger("Job Queue seems to be disabled!"); exit(1); } if (getenv("ZS_RUN_ONCE_NODE") == 1) { $name = DEPLOYMENT_JOB_BASENAME.$appCurrentVersion; $queue = new ZendJobQueue(); // Before scheduling our job, just make sure that it has not been scheduled before $total = null; $jobArray = $queue-‐>getJobsList(array('name' => $name), $total); if ($total != 0) { foreach($jobArray AS $job) { $queue-‐>removeJob($job['id']); } } // Schedule the job script to run a long >me in the future (since we don't really need it to run) $runAt = date('Y-‐m-‐d H:i:s', strto>me('+7 days')); $semaphoreJobId = $queue-‐>createHapJob('hap://www.google.de/search?q=dummy', array(), array('name' => $name, 'schedule_?me' => $runAt)); if (!$semaphoreJobId) { logger("Semphore job was not accepted in the queue!"); exit(1); } }


…and this at the end of the script

20/11/13 19

if (getenv("ZS_RUN_ONCE_NODE") == 1) { // We are the RUN ONCE script so clear the semaphore $queue-‐>removeJob($semaphoreJobId); } else { // We are not the RUN ONCE script so wait for it to clear the semaphore $queue = new ZendJobQueue(); $total = 1; $status = ZendJobQueue::JOB_STATUS_PENDING | ZendJobQueue::JOB_STATUS_SCHEDULED; while ($total != 0) { $jobArray = $queue-‐>getJobsList(array('name' => $name, 'status' => $status), $total); sleep(1); } }

We now plan to use an Nginx, that spontaneously appeared in our architecture, to set our maintenance page. Nevertheless, it was s>ll a cool idea ;)


Finding out what is happening

20/11/13 20

§  So far so good, but we s?ll don’t have true integra?on §  Jenkins has no liale* visibility over the deployment process

§  We can log the output of our deployment scripts §  But these are not available to Jenkins §  We can make them available to the GUI, at least, by adding the paths to the

GUI_AVAILABLE_LOGS table in the Zend Server database §  More convenient, but we s>ll need to go check the GUI a�er each deployment

* Last minute slide hack


WebAPI to the rescue… again

20/11/13 21

§  Features added to the WebAPI allow registered logs to be remotely read §  logsReadLines §  logsGetLogfile

§  We can finally integrate log output into our Jenkins reports

§  Just a few problems §  Finding the “run once” node §  These methods are not implemented in zs-‐manage (yet) §  Wri>ng the code (I haven’t had the >me yet)


www.novadex.com

www.leaermaschine.com

Please scan this …and tell me where it goes

Honestly, marke>ng wouldn’t tell me!

Thank you very much for your aaen>on

20/11/13 22

Eric Ritchie DevOps Engineer hDps://joind.in/9926

+49 (0) 7142 / 788905-‐0 [email protected]

Novadex GmbH Stuagarter Straße 59 74321 Bie>gheim-‐Bissingen Germany

jenkins entwined with deployment and pragmatism

Technology

novadexgmbh2013 distribuon

explo distribuon

alsoregardi distribuon

iusedtoworkforzendn

alsorega distribuon

leset novadexgmbh2013

therearesome pragmasmtofollow

jenkins greatforcon