leveraging visual testing with your functional tests

Peter Kim 2017

INTRO – H20Dragon Peter Kim Email: h20dragon@outlook.com https://www.linkedin.com/in/peterkim777

Agenda

� Share how visual testing benefited our functional test automation strategy

� Share where visual testing successfully provided coverage to other test types

� Share how to get started with using visual testing

� Demos � Q&A

What is Visual Testing?

What is Visual Testing?

� Verifies the UI/UX �  Test coverage of what was actually rendered

on the UI device

�  Test coverage for responsive app. behaviors

Verifies as though you are using your eyes.

Automated Visual Testing

A need to change because .... �  Catch defects earlier in the SDLC �  Verify the UI/UX across target browsers/UI devices �  Verify the UI/UX across different resolutions �  Existing automation strategy didn’t include “visual” coverage. �  Current process of asserting validations are too static �  Brittle Selenium locators (e.g. XPATH) �  Page Objects aren’t scaling quickly enough �  Customers are reporting defects, in PROD, instead of us �  Inability to scale functional (Regression) test coverage �  Uncertainty of what type of assertions are needed �  Lack domain knowledge in the app. �  Test reports lack depth �  Test reports need meaningful visuals �  Customers feel that your UI/UX is as important as the functionality �  QA team has limited resources �  It’s not 2007 anymore, but your test automation design/strategy

thinks it is. �  You want to repurpose your functional tests for other test types –

e.g. Localization / Accessibility

You need it because ..

Although, we were unable to create assertions to catch potential defects, with visual testing, we now at least have the ability to report and share with others what has changed. Here, you’ve empowered others, or possibly held accountable, that they too were in a position to catch (or see) potential defects or comment on changes (whether intended or not per requirements/user story).

Automation with Selenium

Automation with Selenium �  Challenges

�  Takes time and resources to learn effective

Selenium/WebDriver

�  Selenium is an automation library. You’re responsible for creating and/or piecing together the framework, designs, and strategies that will leverage it.

�  Poor test frameworks/design will mean brittle tests and will most likely hinder your ability to scale – you will be playing “catch-up” with the current versions/releases of your app.

Automation with Selenium � Challenges

�  Assertions are typically “static”, based on the

requirements/user stories/pre-canned queries. E.g. Limited number of assertions used to verify and validate your test case. This means, if your application behaves differently and none of your assertions cover that change .. you are now exposed to risks (defects, false confidence in your findings).

Are You Automating the “Expected”?

“Obvious is the most dangerous word in

mathematics.” - Eric Temple Bell (Mathematician) 1883-1960

Are You Automating the ‘Expected’?

Are You Automating the ‘Expected’?

� Pesticide Paradox Over the course of time, your automated tests will diminish it’s ability to catch defects. Especially, the tests that seem to always pass.

Improving automation

� What was our criteria to advance our existing automation strategy?

Automation Requirements �  Need a technical solution that plays well with Selenium/WebDriver

since all our end-to-end tests are Selenium based.

�  Minimal impact, e.g. refactoring, of our existing automation test suites.

�  Avoid re-inventing designs based on old designs, thinking they’re new designs.

�  Be cautious of “best practices” that have been around since 2007.

�  Short and simple learning curve, with powerful results.

�  Technology/tool must be well supported

�  Technology/tool must scale and be secure (e.g. cloud, multiple device support)

�  Powerful and customizable test reports

Lessons Learned �  .. Be cautious when investing in resources

with strategies which ultimately repurpose the same old approach and processes.

�  Consider leveraging a new design/strategy

�  Consider leveraging proven technologies that integrate with your existing testware.

DEMO – Functional Automated Test

Filters, for advanced search, returns correct set of Porsche 911s.

DEMO – Functional Automated Test

Filters, for advanced search, returns correct set of Porsche 911s.

DEMO I. Create a baseline visual test

The visual baseline will be used for validation of future visual test results.

DEMO I. Create a baseline visual test

The visual baseline will be used for validation of future visual test results.

Demo I. Re-run the test. Visually compare the new run against the baseline

Demo I. Recap �  Created a visual baseline

�  Execute end-to-end with visual coverage �  Specified matchType (e.g. Strict, Content, Layout) �  Specified target page/regions

�  Automated coverage scales with more assertions.

�  Easy to “see” visual results

�  Ability to manage/update assertion criteria

Demo II. Visual Defect

Demo II. Visual Defect Analysis

Demo II. View the results

What was actually demonstrated?

�  Little resources are needed to add test coverage (assertions) if you have a simple, yet powerful, ability to leverage visual testing.

�  Coverage is dramatically improved �  Context (text) �  Layout �  Graphics

�  Less brittleness in your automated Selenium tests, while decreasing coding efforts

�  Forgiveness of ones’ lack of insight of “what should be covered?” or “what’s the scope?”

�  UI (Visual) assertions can facilitate functional testing �  Minimize pesticide paradox �  Fred Flintstone has an awesome car!

Leveraging VT

Localization Test Automation (Regression)

Leveraging VT

�  Localization Testing Ensuring that all targeted and/or served parties, regardless of language, nationality, and culture are equally and fairly respected and presented with the same information and content.

Localization Test Automation � Content �  Images �  Layout

Localization Test Automation � Content

� Headings � Data formats ○  Date ○  Currency

�  Translation Errors �  Images �  Layout

Localization Test Automation � Content

� Headings � Data formats ○  Date ○  Currency

�  Translation Errors �  Images �  Layout

Get more details on automating your Localization tests at https://applitools.com/concur/

Localization Test Auto

Tools –Applitools Eyes �  Visual Test Automation

�  Advanced Visual/Digital Analysis (AI) ○  Mitigates false positives (e.g. floating or shifting of elements) ○  Create/manage regions per inclusion/exclusion

�  Applitools Account (It’s free)

https://applitools.com/sign-up/

�  Applitools API Key

�  Install the SDK (based on your programming language) for your Selenium tests

�  Understanding resolutions – e.g. “viewport” Reference: https://applitools.com/videotutorials/

Summary �  Functional testing can be improved by leveraging visual testing.

�  It’s simple to conduct Visual Test Automation E.g. Applitools Eyes with Selenium/WebDriver based framework

�  Heavy reliance, using the pattern of asserting the “obvious”, being the “master of the obvious”, or “happy path” is a great way to ship defects into PROD.

�  Run and validate your tests on the actual devices your end-users are using.

�  Run and validate your tests on the same resolutions (UI devices) based on the “rendered” pages (not just the DOM).

�  Automate responsive apps visually.

�  Create a plan to deal with Pesticide Paradox

�  Advance your test automation framework with new designs and strategies (it’s not 2007) – also, leverage new technologies that are proven to be reliable and scalable, while being simple yet powerful.

�  For QA/QE teams – leverage “CYA”

Thank You

Peter Kim Email: h20dragon@outlook.com https://www.linkedin.com/in/peterkim777

https://dcast.io