PowerPoint Presentation

[B2G | Gaia | Firefox OS] Front-end Automation in Python (Marionette)

Stephen Donner

Web QA

March 20, 2013

Overview

Genesis / Introduction

Infrastructure (hardware)

Software stack

Builds in Jenkins / Continuous Integration

Test coverage / Capabilities

By the numbers

Will it blen^H^H run? (demo!)

Contribute!

Questions?

Genesis / Introduction

http://www.flickr.com/photos/daniel_gies/5041121388/sizes/z/in/photostream/

Initially, just a B2G QA thing

Genesis / Introduction

http://memegenerator.net/instance/27635541

Hard to keep good secrets for too long!

...so we expanded coverage + infra

Infrastructure (hardware)

7 Unagis, tethered via USB to 7 Mac Minis, running Ubuntu 12.04 LTS

Fun facts: Must have earbuds plugged in for radio tests to work

For the photo/video tests to work, lens/aperture must not be blocked or in the dark

Software Stack

Marionette (https://developer.mozilla.org/en-US/docs/Marionette) - testing server that sits in B2G Selenium-backed WebDriver

marionette_client Python abstraction layer for Marionette

Gaiatest (https://pypi.python.org/pypi/gaiatest) - wrapper around Python unnitests that runs the tests

Gaia UI Tests test framework comprising setUp, tearDown, API wrappers, and the tests themselves

Builds in Jenkins/Continuous Integration

Web QA helps set up, scale, and provide upkeep for more than just our UI tests

http://qa-selenium.mv.mozilla.com:8080/view/B2G/

By the Numbers

Started on November 11, 2012

25 individual contributors [1] From Taipei, Romania, UK, Canada, USA, Beijing

89 [2] tests, consisting of: ~ 19 unit tests

80 UI tests

~1hour per build, including initial flash

[1] https://github.com/mozilla/gaia-ui-tests/graphs[2] http://qa-selenium.mv.mozilla.com:8080/view/B2G/job/b2g.unagi.gaia.nightly.ui/lastBuild/

Test coverage / Capabilities

Sampling of coverage: Dialer (make/receive calls, test IMEI #)

Contacts (call, delete, create, add photo to, send SMS to)

Send/receive SMS

Camera (add, delete photo/videos)

Clock (add/delete alarms)

Calendar (add event, assert today's date)

Browser (load pages over EDGE, Wi-Fi, bookmark, navigate back)

Play a YouTube video

Marketplace (search, install, load, delete apps)

Everything.me (browse to and load/install an app)

Changing wallpaper

Radio (adding/remove station, dial frequency up/down)

Keyboard

Notification bar

and more...

Test coverage / Capabilities

Capabilities:

we push to device (media files photos, contacts, etc.)

we also capture the app HTML, take screenshots on failure for post-test debugging, and store the latest run's logcat (full, debug)

we test the default/empty statesin-between tests, we use the devicemanager API to iterate through and delete media files, delete all contacts, clear Wi-Fi networks, restore modified settings to default, set volume to 0

manifest.inigives us the ability to tell which tests to run, where

attributes:

antenna, b2g, wifi, lan, camera, xfail, panda, sdcard, qemu, bluetooth

Challenges

Two moving targets: B2G (Gaia/Gonk) + Marionette

Marionette-enabled builds are not identical to non-engineering buildsdifferent online/offline handling (patch pending)

changes to how/when adb / remote debugging work in Gaia

memory leak(s)

feature work

performance refactoring

Event-model changes:.click() .tap()

async JavaScript team's expertise is (still) largely Python

very dynamic, heavy HTML5 apps, with CSS3 usage rotates, transforms, SVG hard to find and test location and state of the app

infrastructure set upacquiring Unagi phones (and keeping them healthy)

SIMs (unlimited plans)

figuring out and maintaining a distributed, consistent format to store each phone's info

Will it blen^H^H^ (demo!)

https://www.youtube.com/watch?v=CC2ywYeP-4s

Repository: https://github.com/mozilla/gaia-ui-tests

Contribute!

How to contribute: https://github.com/mozilla/gaia-ui-tests/blob/master/CONTRIBUTING.md

Open issues up for grabs! https://github.com/mozilla/gaia-ui-tests/issues?direction=desc&labels=difficulty+beginner&sort=created&state=open

IRC: #mozwebqa (core Gaia UI Tests)

#ateam (perf tests)

Questions?

B2G

(Boot to Gecko)

Mozilla Marketplace