osdc 2014 test driven infrastructure

Test Driven Infrastructurewww.immobilienscout24.de

Berlin | 09.04.2014 | Schlomo SchapiroSystemarchitekt, Open Source Evangelist

License: http://creativecommons.org/licenses/by-nc-nd/3.0/

DevOps Risk Mitigation

@[email protected]

http://creativecommons.org/licenses/by-nc-nd/3.0/

| Test Driven Infrastructure | Schlomo Schapiro

www.ImmobilienScout24.de

>2 billion PI per month

2 data center with ~1600 VM

~2.5 million outgoing email/day

total of ~600 employees

~30 crossfunctional IT teams

~160 in IT

15 years in business

part of Deutsche Telekom


ExpensiveFix

CheapFix

Costs Of Finding Bugs In Production


PlanDesignBudget

Develop Test

RU

NDEV

PlanDesignBudget

Develop Test RUN OPSTIME


PlanDesignBudget

Develop Test

PlanDesignBudget

Develop Test RUN

Proxyconfig

Databaseborked

„Buy Now“button broken

MTA dropall mail

Load BalancerConfiguration

Everything costsonly 0 €

Login possiblewithout password

NFS notavailable

DB Replicationstopped

No adsshown

BrokenCSS / JS

Tomcat won'tstart/stop

Service usernot defined

sudoersinvalid


DevOps: Respect & Learning

DEV learn from OPS to think about:

● Resources (CPU, RAM, Disk)● Services (Start, Stop, Status)● Dependencies (Start DB before App)● Logfiles (Rotate, Remove)● Disk Space● Monitoring and Alarming● ...

OPS learn from DEV to think about:

● Incremental Improvement● Infrastructure as Code● Version Control System● Coding (OO, Functions, Libraries …)● Code Quality● Unit & Integration Tests● Test Automation● ...


Untested=

Broken


Unit TestsTest the smallest possible components in an artifical environment.

System TestsTest the entire application in a real(istic) environment together with other applications.


Part of build process

Syntax checks

Scripts

Config Files

Data Files

Unit tests for functions/libs

Run program with test data

Check result

Check program behaviour with wrong/broken test data

Also run on Developer desktop

Quick feedback (~ seconds)

Install on test server

Run tests from outside

HTTP calls

Send emails

Try to login

Run tests from inside

Remote Exec (rsh, ssh …)http://go.schapiro.org/rshpitfall

Service Start, Stop & Status

Modify server to create good & bad test scenarios

Reboot

Unit Tests System Tests

http://go.schapiro.org/rshpitfall


UnitTests


%prep%setup -q

%installinstall … %{buildroot}/…install … %{buildroot}/…

%files%defattr(-,root,root,-)/...


BuildRequires: sudo%buildset -evisudo -c -f sudoers %installinstall -m 0440 sudoers -D \ %{buildroot}/etc/suoders.d/%{name} %files%defattr(-,root,root,-)/etc/suoders.d/%{name}


BuildRequires: PyYAML, pylint%buildset -e# syntax checksbash -n my_script.sh# Should be valid python codepylint -E yum-repo-propagate# should be valid YAML filepython -c "↩import yaml ↩yaml.safe_load(open('config.yaml'))↩"...


BuildRequires: python-unittest2, python-teamcity-messages, ...

%buildset +x +e # for teamcity

exec 1>&2 # join stdout to stderr to synchronize between them for teamcity output

FAILED=0TEAMCITY_PROJECT_NAME=1 python unit_tests.py || let FAILED++bash -n oldhomeinfo.sh || let FAILED++

(( FAILED == 0 )) || exit 1


More Examples for Unit Tests

Patching nsswitch.conf and PAM files

Syntax checking HTTPD, DNS, DHCP ... configuration files

Checking SSH Server & Client configurationshttp://go.schapiro.org/sshconfigtest

...

http://go.schapiro.org/sshconfigtest


SystemTests

http://impreza-gt-club.ch/V2.0/Tests/WRX08/Koch2.jpg


BuildAuto-

mationSourceCode(SVN)

Monitor

Changes

Build Server

Run BuildJob

● Check out source● Run Unit Tests● Create RPM●

Sourc

eCod

eDEVYUMRepo

UploadPROYUMRepo

PropagateRPM

Test Server

Deploy

and

Run

Test

Job

yum

Prod Server

Deploy

to PROD

yum


BuildAuto-

mationSourceCode(SVN)

Monitor

Changes

Build Server

Run BuildJob

● Check out source● Run Unit Tests● Create RPM●

Sou

rce

Cod

eDEVYUMRepo

UploadPROYUMRepo

PropagateRPM

Test Server

Deploy

and

Run

Test Job

yum

Prod Server

Deploy

to PROD

yum


SAN mount service

Test via rsh

Mock SAN deviceswith losetup

service start, stop mounts/umounts

Error handling


HTTP Proxy Configuration

X-Forwarded-For header spoofs source

Check result for ERR_ACCESS_DENIED

Run tests for all function groups!

Test Server

GET http://external.com/X-Forwarded-For: 10.11.12.01

502 Bad Gateway✔GET http://external.com/X-Forwarded-For: 10.34.56.01

403 Forbidden

ERR_ACCESS_DENIED by proxy server

✘✘


Subversion Server Configuration

Tests Config RPM Makerhttps://github.com/yadt/yadt-config-rpm-maker

Tests 2 servers: Master & Slave

Replication

Failure and Recovery

Backup and Restore

MasterSVN svnsync

Backup

SlaveSVN

Backup

https://github.com/yadt/yadt-config-rpm-maker


PAM & NSS Configuration

PAM & nsswitch.conf patching

Mock setup via rsh

Mock AD groups and users with nss_db

Check service status

Test login via ssh

Thorben Wengert / pixelio.de


VM Provisioning & Kickstart Installation

Test via HTTP API

Create broken VMs and check error reports

Create valid VM and install Linux OS

Scrape VM screen via OCR

http://github.com/Immobilienscout24/lab-manager-light

http://github.com/Immobilienscout24/lab-manager-light


34 35 36

2 3

53

87 88 89 90 91 92 93 94 95 96 97

TIME

Continous Live Deployment

Deploy every application when it is ready.Automate the delivery chain from source till production.

| Test Driven Infrastructure | Schlomo SchapiroLow Risk – Lots of Fun

http://go.schapiro.org/slides

http://go.schapiro.org/slides


Kontakt:Immobilien Scout GmbHAndreasstraße 1010243 Berlin

Fon: +49 30 243 01-1229 Email: [email protected]: www.immobilienscout24.de

Thank you very much!Please contact me for further questions and discussions.

osdc 2014 test driven infrastructure

Internet

mation source code

test driven infrastructure

schlomo schapiro buildrequires

schapiro immobilienscout24

schapiro

http

immobilienscout24

svn