Load Testing with Gatling

Alan Baird QA Lead at LivingSocial

@bluegrasscoder abaird@bairdsnet.net

!https://goo.gl/Ka0PPr

How do you QA for:

Black Friday

Mother’s Day deal

New Feature rollout

Load testing!

JMeter

Gatling

Tsung

The Grinder

LoadUI

LoadRunner

Neoload

Which should you use?

¯\_(ツ)_/¯

My Reasons

• Gatling lets you write code

• It’s not XML

• Open Source

• I don’t work with Java developers, and there are Scala developers where I work

…other reasons• Nice charts!

• no heavy IDE (works with Vim!)

!

!

!

!

• yeah...there’s a recorder too

Gatling is…

…a load test DSL written in Scala !

it leverages Scala’s concurrency so it’s fast, efficient, and takes advantage of other functional things I don’t understand...

…but this is easy enough to understand

https://blog.flood.io/stress-testing-jmeter-and-gatling/

A Gatling Simulation is made up of Virtual Users

who carry out repeated Scenarios over time

Simulation

./bin/gatling.sh -s LoadTest

1 class LoadTest extends Simulation {! 2 ...! 3 }!

Scenario 1 val edit = exec(http("Form")! 2 .get("/computers/new"))! 3 .pause(1)! 4 .exec(http("Post")! 5 .post("/computers")! 6 .headers(headers_10)! 7 .formParam("name", "Computer")! 8 .formParam("company", "37"))! 9 }!

Virtual Users 1 setUp( ! 2 scenario1.inject(atOnceusers(1)),! 3 scenario2.inject(rampUsers(10) over (2 ! 4 seconds)),! 5 scenario3.inject(constantUsersPerSec(2) ! 6 over (10 seconds))! 7 )!

And now for an extended example...

The Pizza API

Where is your favorite pizza? A project to track the the best pizza places in the world!

(unless you like deep dish*) !

https://github.com/stevekinney/pizza

🍕🍕🍕🍕🍕🍕🍕🍕🍕🍕🍕* this repository’s hating on deep dish pizza does not represent the views of the author of this presentation

INDEX: returns all locations api/v1/pizzerias !SHOW: returns specific location by id api/v1/pizzerias/:some_pizzeria_id !SEARCH: returns all properties for valid params api/v1/properties/search?property=value

live at: http://pizza-dev.elasticbeanstalk.com*

*Please don’t break my server until after the presentation

1 class LoadTest extends Simulation {! 2 val httpConf = http! 3 .baseURL("http://pizza-dev.elasticbeanstalk.com")! 4 ! 5 val scn = scenario("Pizzerias")! 6 .exec(http("get all pizzerias")! 7 .get("/api/v1/pizzerias"))! 8 ! 9 setUp( ! 10 scn.inject(atOnceUsers(100)) ! 11 ).protocols(httpConf) ! 12 } !

A simple index test

Great...so we got the list page test coded,

what about individual pizzerias?

1 class Pizzerias2 extends Simulation {! 2 ! 3 val httpProtocol = http! 4 .baseURL("http://pizza-dev.elasticbeanstalk.com")! 5 ! 6 val scnA = scenario("Pizza Place 1")! 7 .exec(http("get Pizza Place 1")! 8 .get("/api/v1/pizzerias/1"))! 9 ! 10 val scnB = scenario("Pizza Place 2")! 11 .exec(http("get Pizza Place 2")! 12 .get("/api/v1/pizzerias/2"))! 13 ! 14 setUp(! 15 scnA.inject(constantUsersPerSec(10) during(30 seconds)),! 16 scnB.inject(constantUsersPerSec(10) during(30 seconds))! 17 ).protocols(httpProtocol)! 18 }! 19 ! 20 !

• some applications cache the answers to certain requests

• if the same request is made over and over again, the cache is used to reply quickly

• if you want to characterize the actual service, caching will skew your results

• if you are going for real world load, this might not be bad

Aside: caching

• We need a feeder

• Feeders are gatling’s way of setting up an iterator that will make data available to a Scenario in a map

• You can make an iterator yourself

• You can make an iterator from a CSV

• You can make an iterator from a JSON source

• and others...

How do we fix this?

1 Iterator.continually(Map("id" -> (Random.nextInt(133))))

1 class PizzeriasRandom extends Simulation {! 2 ! 3 val httpProtocol = http! 4 .baseURL("http://pizza-dev.elasticbeanstalk.com")! 5 ! 6 val feeder = Iterator.continually(Map("id" ->! 7 (Random.nextInt(133)))) ! 8 ! 9 val scnR = scenario("Pizza Place Random")! 10 .feed(feeder)! 11 .exec(http("get Pizza Place 1")! 12 .get("/api/v1/pizzerias/${id}"))! 13 ! 14 setUp(scnR! 15 .inject(constantUsersPerSec(2) during(60 seconds)))! 16 .protocols(httpProtocol)! 17 }

remember what the API looks like: !

SEARCH: returns all properties for valid params !api/v1/properties/search?some_param=some query

And now for search

1 class PizzeriasQuery extends Simulation {! 2 ! 3 val httpProtocol = http! 4 .baseURL("http://pizza-dev.elasticbeanstalk.com")! 5 ! 6 val scn = scenario("Pizza Place Searches")! 7 .exec(http("search for pizza place")! 8 .get("/api/v1/properties/search")! 9 .queryParam("city", "Oklahoma City"))! 10 ! 11 setUp(scn.inject(constantUsersPerSec(2) ! 12 during(10 seconds)))! 13 .protocols(httpProtocol)! 14 }! 15 !

1 property,value! 2 city,Oklahoma City! 3 city,Boston! 4 city,Dallas! 5 city,Atlantis!

1 class PizzeriasQueryWithCSV extends Simulation {! 2 ! 3 val httpProtocol = http! 4 .baseURL("http://pizza-dev.elasticbeanstalk.com")! 5 ! 6 val searches = csv("pizza_search.csv").circular! 7 ! 8 val scn = scenario("Pizza Place Searches")! 9 .feed(searches)! 10 .exec(http("search for pizza place")! 11 .get("/api/v1/properties/search")! 12 .queryParam("${property}","${value}"))! 13 ! 14 setUp(scn.inject(constantUsersPerSec(2) ! 15 during(10 seconds)))! 16 .protocols(httpProtocol)! 17 } !

Debugging 1 <?xml version="1.0" encoding="UTF-8"?>! 2 <configuration>! 3 ! 4 <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">! 5 <encoder>! 6 <pattern>%d{HH:mm:ss.SSS} [%-5level] %logger{15} - %msg%n%rEx</pattern>! 7 <immediateFlush>false</immediateFlush>! 8 </encoder>! 9 </appender>! 10 ! 11 <!-- Uncomment for logging ALL HTTP request and responses -->! 12 <!-- <logger name="io.gatling.http.ahc" level="TRACE" /> -->! 13 <!-- <logger name="io.gatling.http.response" level="TRACE" /> -->! 14 <!-- Uncomment for logging ONLY FAILED HTTP request and responses -->! 15 <!-- <logger name="io.gatling.http.ahc" level="DEBUG" /> -->! 16 <!-- <logger name="io.gatling.http.response" level="DEBUG" /> -->! 17 ! 18 <root level="WARN">! 19 <appender-ref ref="CONSOLE" />! 20 </root>! 21 ! 22 </configuration>!

1 Request: ! 2 search for pizza place: OK ! 3 =========================! 4 Session:! 5 Session(Pizza Place Searches,1480189078212752348-! 6 19,Map(property -> city, value -> Atlantis),14421! 7 13956702,0,OK,List(),<function1>)! 8 =========================! 9 HTTP request:! 10 GET http://pizza-dev.elasticbeanstalk.com/api/v1! 11 /properties/search?city=Atlantis! 12 headers=! 13 Connection: keep-alive! 14 Host: pizza-dev.elasticbeanstalk.com! 15 Accept: */*! 16 =========================! 17 HTTP response:! 18 status=! 19 200 OK! 20 headers= ! 21 Content-Type: [application/json]! 22 Date: [Sun, 13 Sep 2015 03:14:12 GMT]! 23 Server: [nginx/1.6.2]! 24 X-Content-Type-Options: [nosniff]! 25 Content-Length: [2]! 26 Connection: [keep-alive]! 27 ! 28 body=! 29 []!

Validation 1 class PizzeriasQueryWithCheck extends Simulation {! 2 ! 3 val httpProtocol = http! 4 .baseURL("http://pizza-dev.elasticbeanstalk.com")! 5 ! 6 val searches = csv("pizza_search.csv").circular! 7 ! 8 val scn = scenario("Pizza Place Searches")! 9 .feed(searches)! 10 .exec(http("search for pizza place")! 11 .get("/api/v1/properties/search")! 12 .queryParam("${property}","${value}")! 13 .check(bodyString.not("[]")))! 14 ! 15 setUp(scn.inject(constantUsersPerSec(2) ! 16 during(10 seconds)))! 17 .protocols(httpProtocol)! 18 }! 19 !

Special Cases Or, how to know when your tests aren’t measuring what you think

they’re measuring

• Excessive Queuing

• Testing caching instead of your service

• Cannot assign requested address errors

Excessive Queuing

• Symptom

• You are sending more packets than your system says it’s receiving

• You don’t see any errors

• Fix

• Back off the rate you are sending

Testing caching instead of your service

• Symptom

• system responds faster than expected no matter what load you put on it

• Fix

• vary the types of requests so they are more random

Cannot assign requested address errors

Symptom

• You get “Cannot assign requested address” errors during a load test

• Other out of memory errors

Fix

• see note about how Gatling uses ports on the host OS: http://gatling.io/docs/2.1.7/general/simulation_setup.html?highlight=simulation#injection

• Follow tuning guidelines: http://gatling.io/docs/2.1.7/general/operations.html?highlight=tuning#os-tuning

• use a repeat statement in your scenarios to make connections last longer

How to get help with Gatling

• Documentation: http://gatling.io/docs/2.1.7/index.html

• Gatling User Group:

https://groups.google.com/forum/#!forum/gatling

The Pizza API and my Gatling repo

• pizza api: https://github.com/stevekinney/pizza

• gatling examples: https://github.com/abaird/gatling_examples

• running in Elastic Beanstalk: ??

• script to setup loadtest machine in AWS: ??

Happy Loadtesting!