testing the api behind a mobile app - polteq
TRANSCRIPT
2013/7/11
1
ChinaTest, Our TEST!
Testing the API behind a mobile app
Tutorial
Marc van ‘t Veer
ChinaTest, Our TEST!
Content
• What is an API
• Why use an API
• How to use an API
• How-to test an API
• Exercises
– Group exercises (learning concepts – 7)
– Individual exercises (apply concepts – 4)
• Lessons learned
2013/7/11
2
logo van Flair
17-12-2010
Polteq logo_RGB.png
ChinaTest, Our TEST!
What is an API - Definition
• API, stands for application program interface:
– Set of routines, protocols, tools and blocks for building applications (systems of systems) and so reducing complexity
• Definition within the context of this training:
– An internet web service that opens systems and giving standard way of access to external users
4 Marc van ‘t Veer – Polteq
2013/7/11
3
ChinaTest, Our TEST!
What is an API - API model
• Lego bricks are standard building blocks
• Blocks are a library to build
• Figure is showing which resources there are
• Sales box is an example of the possibilities
ChinaTest, Our TEST!
What is an API - API model
2013/7/11
4
ChinaTest, Our TEST!
What is an API – Typical structure
• API is built upon existing internal architecture
Backend (Enterprise Service Bus)
Adapter
Desktop web
Browser
API
Webservices Adapter
app
Mobile web
SoapUI
Adapter
Test app’s app
ChinaTest, Our TEST!
Why use an API - example
Telecom companies provide usage details on their websites. App developers used screen scraping to
get this data into their apps.
Resulting in: • Misinterpretation of data • Malformed data • Telecom company gets blamed when the app provides wrong information
2013/7/11
5
ChinaTest, Our TEST!
Why use an API - example
API provides the telecom companies:
• More control over the data
• Control over the content used in an app
– Key needed to use the API
– App developer needs to prove that it works correct before getting access
ChinaTest, Our TEST!
Why use an API
• Scaling development
– Enhance a program without editing itself
• Converting competitors into partners
– Allow competitors to build on top of your product
• Scaling market reach
– Open up new markets
• Empowering users
– Allow people to use the product for things it wasn’t designed for
2013/7/11
6
ChinaTest, Our TEST!
How to use an API – Mashup
• A mashup is a web page or web application that uses and combines data, presentation or functionality from two or more sources to create new services
– Easy, fast integration
– Frequently using API
– Produce enriched results from the raw source data
ChinaTest, Our TEST!
How to use an API – Demo of API’s
• Local examples
– Command line on Windows or Linux
– Local VB script with Windows API
• Web API examples
– Overview of API’s: Programmableweb.com
– Demo of WordPress blog API
2013/7/11
7
ChinaTest, Our TEST!
How to use an API – Concepts
• Concepts of the API
– Division of data and presentation
– Root (Start URL, http or https)
– Navigation
– View of a command-line
– Hierarchy
– Resources
– Account (private API)
– Structure
ChinaTest, Our TEST!
How to use an API – Who uses it?
• Apps: Android, Iphone, Windows, Blackberry
• Internal organizations
• User via browser version of the API
• Social media: Facebook, twitter, etc.
• (internal/external) developers:
– Documentation
– Base line code for new App (with guidelines)
• Operations (monitoring)
2013/7/11
8
ChinaTest, Our TEST!
Summary – Introduction to an API
• Public available via Internet
• No-frontend
• Interfaces
• API is part of an end-to-end solution
• Architecture / protocol focus
• External users / devices
• Test tools are needed
• Main goal is: a generic API
ChinaTest, Our TEST!
How-to test an API – start from scratch
2013/7/11
9
ChinaTest, Our TEST!
How to test an API – start from scratch
• Architecture
– REST vs. SOAP
– Stateless vs. Session
– Media types vs. WSDL
• text/xml or application/vendor specific
– Verbs (GET) and resources
– Headers and caching results
• Tools
– SoapUI, Fiddler, new add-ons for Firefox
• Architecture
– REST vs. SOAP
– Stateless vs. Session
– Media types vs. WSDL
• text/xml or application/vendor specific
– Verbs (GET) and resources
– Headers and caching results
• Tools
– SoapUI, Fiddler, new add-ons for Firefox
ChinaTest, Our TEST!
How to test an API – RESTful verbs
• Verbs are actions, actions like CRUD on database
• POST: Create a new data element
• GET: Read a data element
• PUT: Update a data element
• DELETE: Delete a data element
2013/7/11
10
ChinaTest, Our TEST!
How to test an API – Headers
• HTTP header fields are components of the message header of requests and responses in the Hypertext Transfer Protocol (HTTP).
• They define the operating parameters of an HTTP transaction
• Request example: • Accept: text/plain
• Cache-Control: no-cache
ChinaTest, Our TEST!
How to test an API – Cache
• A web browser stores web content for reuse
• If the back button is pressed, the local cached version of a page is displayed instead of a new request being sent to the web server.
• Risks
– Old data
– Cached errors
– Layered caching (app – API – server)
2013/7/11
11
ChinaTest, Our TEST!
How to test an API - Typical risks
• Unknown integration
• Big variation of customer data
• No control of the E2E chain
• Load is unknown
• Wrong use of API
• Dynamic scope
ChinaTest, Our TEST!
How to test an API - Strategy
• Early integration test with complete infrastructure
– Integration test and dog fooding during Development and System Testing
– Multiple integration phases
– Prototype app (on Acceptance environment)
2013/7/11
12
ChinaTest, Our TEST!
How to test an API - Dogfooding
• Dogfooding: Don’t try to build the sort of API you think people will want, but use the public API’s your self everywhere
• Examples
– All Google tools use their own public API’s
– Windows 7 is build with windows 7
– Twitter.com uses it’s own public API
ChinaTest, Our TEST!
How to test an API - Test approach
1. Development (with dogfooding)
2. System testing (API with stubs and SIT)
3. Integration testing (API with backend)
4. Acceptance testing (API with prototype app)
5. Production integration (in phases)
6. Regression testing (automated)
2013/7/11
13
ChinaTest, Our TEST!
How to test an API - Test approach
Backend (Enterprise Service Bus)
Adapter
Desktop web
Browser
API
Webservices Adapter
app
Mobile web
SoapUI
Adapter
Test app’s app
ChinaTest, Our TEST!
How to test an API - Mocks and stubs - 1
• API’s are part of a big E2E chain
• Testing is only possible if a stub framework is available
• Stub, driver, mock
• Example of a stub framework
2013/7/11
14
ChinaTest, Our TEST!
How to test an API - Mocks and stubs - 2
• Replacement of a reply from backend system
ChinaTest, Our TEST!
How to test an API - Mocks and stubs - 3
• If there is no connection to a backend system
• Re-routing an interface to a local tool like SoapUI
2013/7/11
15
ChinaTest, Our TEST!
Group exercises
• Minimum of three implementations is needed for a quality API
• Learning
– One implementation: it’s a trick
– Two implementations: it’s a method
– Three implementations: it’s your own way of working
ChinaTest, Our TEST!
Group exercises – learning concepts
• Exercise 1 – Lorum ipsum text
• Exercise 2 – Colour lovers and Color picker
• Exercise 3 – Authorization
• Exercise 4 – Status and error codes
• Exercise 5 – Header injections
• Exercise 6 – Business rules
• Exercise 7 – Current and future version
2013/7/11
16
ChinaTest, Our TEST!
Exercise 1 – Lorum ipsum text
• Description
– Generation of Lorum ipsum text
• Assignment
– Get a number of lines of bacon ipsum text
– Tool: Firefox
ChinaTest, Our TEST!
Exercise 1 details – Lorum ipsum text
• API
– http://baconipsum.com/api/
• API documentation
– Open two tabs
• One with documentation: http://baconipsum.com/api/
• One to enter commands like – ?type=meat-and-filler
2013/7/11
17
ChinaTest, Our TEST!
Exercise 2 – Colour lovers and Color picker
• Description
– API provide content in many different types, like generation of colors
• Assignment
– Find a favorite color with colour lover API and validate the Hex-code with Color picker
– Tool: Firefox
ChinaTest, Our TEST!
Exercise 2 details – Color lovers and Color picker
• API
– http://www.colourlovers.com/api/colors?
– http://www.colorpicker.com/
• API documentation
– http://www.colourlovers.com/api/
2013/7/11
18
ChinaTest, Our TEST!
Exercise 3 – Authorization
• Description
– Difference between public and private API
– Testing of functional security
• Assignment
– Get access to API by using access tokens
– Tools: Firefox
ChinaTest, Our TEST!
Exercise 3 details – Authorization
• API
– http://openexchangerates.org/api/latest.json
– Access key:
• 66e62d8337b545ec9f0508c1215764b5
• API documentation • https://openexchangerates.org/documentation
• https://openexchangerates.org/documentation#app-ids
2013/7/11
19
ChinaTest, Our TEST!
Exercise 4 – Status and error codes
• Description
– Explore what is a good and what is a bad request
• Assignment
– Find status and error codes and rules to trigger them
– Which error code is shown when (priority)
– When do you expect an error code
– Tools: Firefox + Firebug
ChinaTest, Our TEST!
Exercise 4 details – Status and error codes
• API
– http://openexchangerates.org/api/latest.json?app_id=66e62d8337b545ec9f0508c1215764b5
• API documentation
– https://openexchangerates.org/documentation
– https://openexchangerates.org/documentation#errors
– http://en.wikipedia.org/wiki/List_of_HTTP_status_codes
2013/7/11
20
ChinaTest, Our TEST!
Exercise 5 – Header injections
• Description
– Explore what header elements are
• Assignment
– Compare results with and without a header elements
• If-Modified-Since and If-None-Match
– Tools: Firefox + Rest Client
ChinaTest, Our TEST!
Exercise 5 details - Header injections
• API
– http://openexchangerates.org/api/latest.json?app_id=66e62d8337b545ec9f0508c1215764b5
• Documentation
– https://openexchangerates.org/documentation#etags
– http://en.wikipedia.org/wiki/List_of_HTTP_header_fields
2013/7/11
21
ChinaTest, Our TEST!
Exercise 6 – Business rules
• Description
– Compare output between different customer types. Which resource has which output?
• Assignment
– Find output of a prepaid and a business customer
– Find a invoice for a prepaid customer
– Tool: Firefox
ChinaTest, Our TEST!
Exercise 6 details – Business rules
• API
– https://capi.t-mobile.nl/
– Access key
• {Bearer} with: – {StubbedResidentialPostPaid}
– {StubbedBusinessPostPaid}
– {StubbedResidentialPrePaid}
• Documentation
– https://capi.t-mobile.nl/documentation 42 Marc van ‘t Veer – Polteq
2013/7/11
22
ChinaTest, Our TEST!
Exercise 7 – Versions of resources
• Description
– Resources have a life cycle which is organized in versions for now and the future
• Assignment
– Find a resource that is active, have multiple versions and is end of life
– Tool: Firefox
ChinaTest, Our TEST!
Exercise 7 details - Versions of resources
• API
– https://capi.t-mobile.nl/
– Access key
• {Bearer} with: – {StubbedResidentialPostPaid}
– {StubbedBusinessPostPaid}
– {StubbedResidentialPrePaid}
• Documentation
– https://capi.t-mobile.nl/documentation
2013/7/11
23
ChinaTest, Our TEST!
Individual exercises – Apply concepts
• Exercise 8 - Design review
• Exercise 9 – Testing design techniques
• Exercise 10 – Explore test application
• Exercise 11 - Search of the defects
ChinaTest, Our TEST!
Test levels for testing an API
Phase Structure Element Test case
-2. Create designs Review -Perform review on API design
-1. Test case design Design -Create test cases from designs
0. Setup Test application Setup - Stub frame work - Tooling - Interfaces and test data
1. Development Framework Dog fooding Check easy of use framework and how good it functions
2. System testing Functionality with and without stubs
• With and without stubs • Test applications
URL Structure, parameters • Header request : character set, media types, Verbs (GET/POST/UPDATE), HTTP(S), status/error codes • Logging: Support defect analysis (server and database) • HTTP-status: Error and related status Common validation/Authentication/Authorization • Access codes (tokes) • Profiles (subscription active, SIM active, token status) • Customer/contract types • Caching: Duration, authorization, clearing, content, multiple users • Navigation: between resources Resources • Profiles: Business rules for each resource with variation customer data Output • Presentation forms and versions (supported not supported) Monitoring • Reporting and performance (load filter, backend availability Documentation
SIT No stubs Integration with Middleware
Variation in customers, rate plans, prices, contract types, history, options with no stubs
3. Integration testing Internal API with complete backend No regression impact of other projects
4. Acceptance testing External integration API with prototype First integration of app with API
5. Production testing Real customers API with live app Real customers with app on production API
6. Regression Regression Reruns - Manual and Automated regression tests
2013/7/11
24
ChinaTest, Our TEST!
Test application
• JSON API as plugin for WordPress
• CMS system
• Customizable and Open source
• Based on PHP language and MySQL database
• Also has mobile application for all platforms
• See also: http://en.wikipedia.org/wiki/WordPress
ChinaTest, Our TEST!
Test environment setup
Test tool
API
Verification
Request
Reply
Request
Reply
WordPress
server
Data
2013/7/11
25
ChinaTest, Our TEST!
IP address WordPress
ChinaTest, Our TEST!
Exercise 8 – Design review
• Description
– An API design should give all information about how it operates
• Assignment
– Use a checklist to review design of JSON API plugin
2013/7/11
26
ChinaTest, Our TEST!
Exercise 8 details – Design review
• API
– http://wordpress.org/extend/plugins/json-api/other_notes/
– http://<local IP address>/documentation/json-api/
• Documentation
– https://github.com/WhiteHouse/api-standards
– http://<local IP address>/documentation/design-review-checklist/
51 Marc van ‘t Veer – Polteq
ChinaTest, Our TEST!
Exercise 8 - Design review result
• Result of design review on JSON API plugin
–
–
–
–
–
–
–
–
2013/7/11
27
ChinaTest, Our TEST!
Exercise 9 – Test design techniques
• Description
– Normal test techniques are useful
• Assignment
– Create test cases for the JSON API plugin of Wordpress
– Use techniques like
• State transition
• Data combination / classification tree
• Equivalence Class partitioning
• Boundary Value analysis
ChinaTest, Our TEST!
Exercise 9 – Testing design techniques
• Use scrap paper to write test cases
• See example workout
• Documentation
– http://wordpress.org/extend/plugins/json-api/other_notes/
2013/7/11
28
ChinaTest, Our TEST!
Exercise 10 – Explore Test application
• Description
– Before you can test the API you should first explore it
• Assignment
– Open the JSON API and try it out
ChinaTest, Our TEST!
Exercise 10 – Explore Test application details
• API
– http://<local IP address>/?json=
• Documentation
• http://wordpress.org/extend/plugins/json-api/other_notes/
• http://<local IP address>/json-api/
• To comment or post with RestClient use: – “name” = “Content-Type” and “value” = “application/x-www-
form-urlencoded” 56 Marc van ‘t Veer – Polteq
2013/7/11
29
ChinaTest, Our TEST!
Exercise 11 – Search the defects
• Description
– Search defects on the JSON API in WordPress
• Assignment
– Use the design test cases
– Use the learned tooling and techniques
– Analyze the results
– Tool: Firefox + Rest Client
ChinaTest, Our TEST!
Exercise 11 – API details
• API
– http://<local IP address>/?json=
• Documentation
• http://wordpress.org/extend/plugins/json-api/other_notes/
• http://<local IP address>/json-api/
• To comment or post with RestClient use: – “name” = “Content-Type” and “value” = “application/x-www-
form-urlencoded”
2013/7/11
30
ChinaTest, Our TEST!
Exercise 11 - Search the defects
• Which defects are found
–
–
–
–
–
–
–
–
–
–
–
–
ChinaTest, Our TEST!
Evaluation of exercises
• What went well?
–
–
–
• What went wrong?
–
–
–
• Conclusion
–
–
–
2013/7/11
31
ChinaTest, Our TEST!
Lessons learned
• API
• API/app communication
• Testing
ChinaTest, Our TEST!
API
• New test type: production tests
• Command line
• Scope
• Different skills
– Security
– Performance
– Automation
• No backup tricks
2013/7/11
32
ChinaTest, Our TEST!
API/app communication
• Provider of the API is seen as responsible for the presentation of the data in the app
• Presentation errors of data will always occur
• Very tempting to fix defects in app instead of API
• More explanation needed with secured API
ChinaTest, Our TEST!
Testing
• Normal test techniques
• Instrument for defect analysis
• Automated regression tests on production
• New responsibilities
• SOA/Interfaces/HTTP protocol/Tools
• Experience with building interfaces
2013/7/11
33
ChinaTest, Our TEST!
More information
Marc van 't Veer Test consultant Polteq Amersfoort (Netherlands)
+31 (0) 6 46 63 61 48 (mob) http://www.polteq.com [email protected]
ChinaTest, Our TEST!
References
Source Link
[Apple, 2009]
iPhone 3g Commercial "There's An App For That" http://www.youtube.com/watch?v=szrsfeyLzyg
[T-Mobile, 2011] Customer API T-Mobile: https://capi.t-mobile.nl/
[API design, 2013] http://apidesign.com
[SmartBear Software, 2011] SoapUI with REST: http://www.soapui.org/REST-Testing/getting-started.html API testing: http://blog.smartbear.com/software-quality/bid/273613/practicing-safe-sex-with-third-party-apis
[Molen animation,2013] http://www.animatieplaatjes.nl/molens.html
Test design technique http://www.pragmaticmarketing.com/resources/why-api-as-a-strategy
2013/7/11
34
ChinaTest, Our TEST!
References, continued
Source Link
[Tweakers, 2012] More secure public API from NS: http://tweakers.net/nieuws/81389/ns-wil-apps-die-prijs-treinreis-tonen-vooraf-keuren.html
WordPress http://www.wordpress.org http://en.wikipedia.org/wiki/WordPress http://developer.wordpress.com/docs/api/ http://wordpress.org/extend/plugins/json-api/other_notes/
Practical API Design Jaroslav Tulach, chapter 9: Keep Testability in Mind
Firefox add-ons http://www.garethhunt.com/modifyheaders/help/ http://jsonview.com/ http://restclient.net/ https://addons.mozilla.org/en-us/firefox/addon/make-address-bar-font-size-big/
ChinaTest, Our TEST!
References, continued
Source Link
[Wiki,2012] http://en.wikipedia.org/wiki/Application_programming_interface#Use_of_APIs_to_share_content REST architecture: http://en.wikipedia.org/wiki/Representational_state_transfer Internet Media Types: http://en.wikipedia.org/wiki/Internet_media_type#Type_application http://en.wikipedia.org/wiki/Mashup_%28web_application_hybrid%29 http://www.mediawiki.org/wiki/API:Main_page http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods http://en.wikipedia.org/wiki/List_of_HTTP_header_fields http://en.wikipedia.org/wiki/Web_browser
2013/7/11
35
ChinaTest, Our TEST!
References, continued
Source Link
[Wiki,2012], continued http://en.wikipedia.org/wiki/Service-oriented_architecture Sequence diagrams: http://nl.wikipedia.org/wiki/Unified_Modeling_Language Dogfooding: http://en.wikipedia.org/wiki/Eating_your_own_dog_food
Learning REST API
http://www.restapitutorial.com/ http://toc.oreilly.com/2013/02/a-publishers-job-is-to-provide-a-good-api-for-books.html http://net.tutsplus.com/tutorials/other/a-beginners-introduction-to-http-and-rest/ https://github.com/WhiteHouse/api-standards
HTTP design http://www.ietf.org/rfc/rfc2616.txt
App testing http://www.kohl.ca
HTTP Header elements http://themayesfamily.com/blogs/b/2011/05/rest-client-for-firefox-sample-post-request/