building an effective international software qa test strategy

Buildingan EffectiveInternationalSoftware Test Strategy

www.aeontera.comKevin Gee

Presenter

Presentation Notes

Hello, my name is Kevin Gee, from Aeontera. Aeontera is a consulting firm that helps clients design, build and test software that works in multiple countries. In this talk, we will be showing you how to build an effective international software test strategy.

Types of International QA Test

Issues in Functional Testing

Key Recommendations

Presenter

Presentation Notes

We’ll start by outlining the different types of testing done in an international QA process. Then we will describe some common issues encountered during functional testing. Finally, we’ll summarize our key recommendations.

Types ofInternationalQA Testing

Presenter

Presentation Notes

There are four different types of testing used to improve the quality of global software.

Linguistic Test

Cultural Test

Functional Test

Cosmetic Test

Presenter

Presentation Notes

These are linguistic, cultural, cosmetic, and functional testing.

Linguistic Testing

Presenter

Presentation Notes

Linguistic testing finds defects in language translation. It confirms that content has been translated correctly and consistently, given the context of a particular piece of software. This includes finding errors in grammar, spelling, semantics, as well as formatting of names, dates, time, and other units. We generally recommend outsourcing in-country resources for linguistic testing. They tend to be less expensive and more fluent in the local language and customs than in-house staff. Image source: http://auntielynda.blogspot.com/2008/07/funny-foreign-signs.html

GlossariesSpecifications

Translation Memory

Presenter

Presentation Notes

There are several tools available to improve the quality and consistency of translated content during linguistic testing. These include glossaries of terminology, translation specifications, and translation memory software that retains previous translations for reuse.

Cultural Test

Presenter

Presentation Notes

Cultural testing verifies that software conforms to local social, political, and religious norms. This may involve checking for improper images, colors, linguistic idioms, formality of speech, or cultural references. Also, cultural testing ensures that content is consistent with government or industry ratings and censorship laws. If cultural testing is not performed, software can appear 'foreign' to prospective customers, or certain elements of the design may offend users. The solution is to spot-check the design and content throughout the development process using in-house or outside expertise. Finally, the product should be submitted to cultural testing prior to launch.

Cosmetic TestFrench Arabic

Presenter

Presentation Notes

Cosmetic testing looks at the intersection between localized content and software functionality. It ensures that text is readable, and flows correctly within the layout of buttons, menus, or regions of a screen. Words may be improperly hyphenated, or truncated, or may be too large or small to be easily read. You can begin cosmetic testing very early in the development process by using something called pseudo-translated content. This is content that has been automatically translated from the content of the base locale and is used for preliminary testing of the layout and design before the fully translated content is available. As an example of cosmetic testing, we have two screens from a CRM application with one screen in French, the other in Arabic. Since Arabic reads from the right to the left, we should confirm that the logo for the software is on the opposite side of the screen from the French version. Indeed, we find this is the case.

Cosmetic Test

Progress Bar Direction

Reading Direction

Presenter

Presentation Notes

Here we find a minor defect in the progress bar of the installer for the Arabic version of the Firefox 3.6 browser. The progress bar moves from left to right, going against the reading direction of the user.

Presenter

Presentation Notes

On this screen, we see improperly rendered characters on the Google.ae page. This is an example of a functional defect being detected late in the release cycle during cosmetic testing. The meaningless squares are actually text in the Thai character set. The default setup of Arabic Windows does not have the Thai display pack installed so the characters do not render. We believe this defect may have slipped through Google’s QA process because they may have tested the page using a system loaded with multiple language packs (including Thai). This test setup misrepresented almost all end users’ systems in the market.

Functional Test

Features

Base Locale

Presenter

Presentation Notes

Let’s move on to a discussion of functional testing. Standard functional testing verifies that all of the functions of a piece of software work for users in the base locale.

Functional Test

GlobalFeatures

Base Locale

New Locale

Presenter

Presentation Notes

In functional testing of multinational software, we look for defects in global features that only show up for users in certain locales.

Presenter

Presentation Notes

An example of local defects for global features is seen in this screenshot of the support forums for an online shopping cart. Users in certain locales are seeing issues in feature sets that are available for all users.

Functional Test

GlobalFeatures

LocalFeatures

Base Locale

New Locale

New Locale

Presenter

Presentation Notes

The second type of defects we look for in multinational functional testing are in features that are available exclusively to users in certain locales.

Presenter

Presentation Notes

For example, we may allow users in different locales to make payments using methods that are preferred in their area. For instance, users in Japan may be able to pay with a FeliCa NFC card. Users in the Netherlands may pay with Chipknip, while American users could pay by electronic check. Please keep in mind that you should both verify global features for local users, as well as local features for local users during functional testing.

FunctionalTest Issues

Presenter

Presentation Notes

Now lets get into more detail around issues that you will face while performing functional testing for global software.

English French Italian German Spanish Traditional Chinese

Japanese Korean Arabic

Addi

tiona

l Tes

ting

Requ

ired

Scope of Testing is Too Large

LinguisticFunctional

Presenter

Presentation Notes

The single greatest misconception about QA testing for multinational software is that it mainly deals with finding errors in translated text. However, that is only a small part of the problem. You see, most defects in multinational software are functional rather than linguistic in nature. Each new locale exercises code in a new way, revealing functional defects that were not exposed in the base locale. Functional testing for new locales is certainly necessary. However, duplicating all QA tests for each locale is unrealistic due to cost. If functional testing is done in a naïve manner, each new locale requires even more effort to test than the base locale. This means that the scope of testing can grow rapidly, making it too expensive to support new markets.

English French Italian German Spanish Traditional Chinese

Japanese Korean Arabic

Addi

tiona

l Tes

ting

Requ

ired

Prudent Test Strategy

LinguisticFunctional

Group locales by common

characteristics

Focus on key risk areas

Presenter

Presentation Notes

A prudent test strategy is one that finds the same number of defects faster and with fewer resources than a broad-based approach. There are two ways to do this. First, you can reduce the scope of testing by grouping locales together by common characteristics and risks. Second, efforts should be focused on high risks locales and high risk areas within locales. This approach provides good test coverage while wasting fewer resources. Unfortunately, this approach requires experience that may not be available in the organization.

Functional Requirements

Discovered Late in Development Cycle

Presenter

Presentation Notes

It is a truism in software development that defects cost more to find and fix the later they are discovered in the development cycle. This can be a very serious issue in global software development. In worst case scenarios, new functional requirements may only be discovered after release during user acceptance testing, or by real end users. This can severely delay revenues in specific locales as release dates slide out to create time for development of new features or resolution of serious functional defects. What can be done to prevent this? The answer of course, is to document the requirements for global users before development begins. There is no excuse for not understanding local requirements for names, dates, currencies, or address formats.

Juana AguirrePiedras No 623Piso2 Dto.4C1070AAM Capital FederalARGENTINA

U.S. Asis Environmental Partnership in Taiwan(U.S.-AEP Taiwan), American Institute in Taiwan International Trade Tower 32/F, 333 Keelung Road, Section 1Taipei, 105-48TAIWAN, R.O.C.

Ministry of Information and Communication116 Shinmullo 1-gaChongno-guSEOUL 110-700REP OF KOREA

The Senior General ManagerPostal BusinessSA Post OfficeP.O. Box 10 000PRETORIA0083 SOUTH AFRICA

Presenter

Presentation Notes

For example, logistics companies will often validate address information before a parcel is submitting for shipping. Having a valid address avoids the cost of attempting shipment for an improperly addressed package. Here we see some examples of shipping addresses for various locales. Notice that the addresses in Japan and Korea do not include street addresses, but rather blocks and neighborhoods. Notice also that the numerical and character formats for zip codes are not consistent from country to country. The specifications for software that accepts these addresses should define the input validation rules for various locales so unit, integration, and system tests can be developed against the requirements. Source: http://www.bitboost.com/ref/international-address-formats.html http://www.japan-guide.com/e/e2224.html

Functional Tests

Localized Requirements

Localized Test Data

Presenter

Presentation Notes

Requirements such as these should be documented before coding begins, not discovered just prior to release. Ideally, you should use localized data for testing, and write tests that verify that local requirements have been implemented correctly.

Accuracy Consistency Reliability Data Integrity

Presenter

Presentation Notes

Now let’s talk about data integrity. The concept of data integrity involves ensuring the accuracy, consistency, and reliability of data in an application. Most commercial relational databases enforce referential integrity. Also, they support other features that fulfill the so-called ACID properties of atomicity, consistency, isolation, and durability needed for data integrity.

MultinationalBusiness

Logic

External System

External System

In-Country User

Internal DB

In-Country User

Presenter

Presentation Notes

However, achieving data integrity for multinational systems goes beyond maintaining referential integrity within a single database. External systems, 3rd party software, and international users transform data in new ways. Data sources may not only be traditional relational databases, but unstructured or semi-structured data from XML documents or key-value stores. Data can get corrupted if external or legacy systems use non-Unicode character sets. Also, external and internal systems may use validation rules for data that are mutually exclusive.

Levels of Data Integrity

User

System

Process

Content

Presenter

Presentation Notes

For multinational software, we must think about data integrity on multiple levels. These are the content, process, system, and user levels. Source: http://simplecomplexity.net/data-integrity-what-does-it-really-mean-why-is-it-important/

Testing Data Integrity

User

System

Process

ContentMultinational Test Data

Unit and Integration Tests

In-Country UserAcceptance Testing

System Tests

Presenter

Presentation Notes

On the content level, we must verify that data is stored properly by using multinational test data. Data integrity issues with name, address, date and time, as well as currency data are common at this level. On the process level we must check for data integrity with unit and integration tests. On a system level we see quite complex issues.

dateTime09/08/201010:30:19

dateTime09/07/2010 10:30:19

date09/07/2010

Presenter

Presentation Notes

For example, we recently looked at a mobile sync application. It showed a defect that shifted annual recurring events by one day while synchronizing between different databases. This was because in some parts of the system, recurring dates were stored as a date data type, while in other parts dates were stored using dateTime. When data was transformed between date and dateTime objects, they could shift if they crossed a dateline boundary. This is an example of a system level functional defect that causes a loss of data integrity for international users. Image source: http://upload.wikimedia.org/wikipedia/commons/6/61/International_Date_Line.png

Testing Data Integrity

User

System

Process

ContentMultinational Test Data

Unit and Integration Tests

In-Country UserAcceptance Testing

System Tests

Presenter

Presentation Notes

User-level data integrity is crucial for multinational software because different users may have different expectations for the behavior of a system.

System

MM/DD/YY09/08/07DD/MM/YY

Local User

Base Locale

Presenter

Presentation Notes

Here is an example of a loss of data integrity on a user level: Let’s say some users input dates in Month-Day-Year format, and others expect dates in Day-Month-Year format, the data may become corrupted even if the system is implemented correctly according to the design. The system is implemented correctly according to users in the base locale, but users in another locale are inputting data in a way not expected by the system. In this example, the data would even pass input validation tests in some cases. This is why multinational software may not achieve data integrity in practice at a user level, even if it achieves it by design at the system level.

Long Build Times and Failed Build Acceptance Tests

Presenter

Presentation Notes

We now look at how international software can experience long build times and high failure rates during build acceptance testing.

Build Test

Patch

Build

Base Locale

New Locale

Dependency

3rd Party Software

Dependency and 3rd Party Software Cause Delay in Build Process

Test

Release

Release

Presenter

Presentation Notes

Here is a diagram of a serialized build process. In this case, code for the base locale goes through the build process and build acceptance testing before being handed off to another team to build and test the new locale. Once the base locale passes the build acceptance test, the source code is patched to start the build process for the new locale. This serialization of the build process increases cycle times for the whole project. Also, in this case there is a greater risk of failure during build acceptance testing for the new locale due to incompatibility with 3rd party software. These issues can dramatically lengthen build cycles for multiple locales.

GlobalCode

Locale-Specific

CodeInitial Code Base

Presenter

Presentation Notes

The first thing we can do to improve build times is to internationalize the code base. To start, our initial code base mixed international and locale-specific code in some unclear way. Patches and other serial modifications to the code base created the build dependency we looked at earlier.

Internationalization

GlobalCode

Locale-Specific

CodeInitial Code Base

Base Locale Code

I18n Base Code

Locale-Specific Codeis Modular

Locale 2 Code

Locale 5Code

Locale 4Code

Locale 3Code

Localization

Presenter

Presentation Notes

In the software internationalization process, we split the code into an I18n base or internationalization code base, and code that is specific to each locale, including the base locale. This allows some of our build process to be done in parallel.

Build Test

Build

Base Locale

Release

Internationalize Code Base andPre-Qualify 3rd Party Software

ReleaseTest

I18N Base

New Locale

Build

Presenter

Presentation Notes

In our revised build process, we first do builds from our I18n base that are universal for all locales. Then we can build and test for each locale in parallel, reducing overall cycle times.

Presenter

Presentation Notes

Here is an example of the types of problems that can come up during build acceptance testing. Here is a script to run a Java application. One problem with this script is the inclusion of a Java Runtime Engine that does not work with Japanese Windows. Notice also that it is hardcoded to run applications from a specific directory structure.

Presenter

Presentation Notes

The application failed build acceptance tests in the Japanese locale because it was expecting a different directory structure.

Build Test

Build

Base Locale

Release

Internationalize Code Base andPre-Qualify 3rd Party Software

ReleaseTest

I18N Base

Locale 2 Pre-Qualified3rd Party Software

Build

Presenter

Presentation Notes

We can avoid these late stage failures by pre-qualifying 3rd party software even before build acceptance testing begins.

Summaryof Recommendations

Presenter

Presentation Notes

Let’s summarize our recommendations so far.

Key Recommendations• Group locales by region• Focus on key risk areas• Use in-country testers• Use pseudo-translated content to test layout

and UI design• Document requirements• Test early in development cycle• Check data integrity on system and user levels• Pre-qualify 3rd party software

Presenter

Presentation Notes

First, you can dramatically reduce the scope of testing required by grouping locales together by common characteristics, and focusing your test efforts on high risk areas. This will make the incremental testing required for new locales much lower. Use in-country testers for linguistic, cosmetic, and cultural testing. They are a cost effective way to ensure that your product appears to be high quality to users in each locale. Use pseudo-translated content to perform early cosmetic testing and verify that layout and UI design is appropriate for your users. Next, you should document functional requirements that are specific to locales early, so you can test against them with unit, integration, and system tests during development. You should check for data integrity of your software on both the system and user levels. Do not rely on low level features of a relational database to make this happen. Rather, use localized test data and test against user requirements at a system level. You can improve cycle times by pre-qualifying 3rd party code. This step can avoid delays during build acceptance testing for new locales.

Please visit us atwww.aeontera.com

Thank You!

Presenter

Presentation Notes

Thank you for listening. If you would like to learn more about our multinational software consulting or testing services, please visit www.aeontera.com.

building an effective international software qa test strategy

Technology