internationalization testing for a large enterprise product · beta signoff testing will be done on...
TRANSCRIPT
Internationalization testing for a large Enterprise Product
06/23/2015
White Paper
Ramkumar B
Internationalization Testing
2
Table of Content
Abstract ............................................................................................................................................................................. 3
About the Author .............................................................................................................................................................. 4
1. Introduction .............................................................................................................................................................. 5
2. Detailed Walk through .............................................................................................................................................. 5
2.1 Key Activities ..................................................................................................................................................... 5
2.2 Phases ............................................................................................................................................................... 5
2.3 Preparation Phases ........................................................................................................................................... 6
2.3.1 Feature Review ......................................................................................................................................... 6
2.3.2 Test Case Porting ....................................................................................................................................... 7
2.3.3 Test Cases Review ..................................................................................................................................... 7
2.3.4 Test Configuration Matrix ......................................................................................................................... 8
2.3.5 WR Audit ................................................................................................................................................... 8
2.3.6 WR Audit ................................................................................................................................................... 8
2.4 Execution Phases ............................................................................................................................................... 9
2.4.1 Pseudo Localization Test Pass (PLTP) ........................................................................................................ 9
2.4.2 Beta Test Pass ........................................................................................................................................... 9
2.4.3 Beta Sign Off ............................................................................................................................................ 10
2.4.4 Language Test Pass ................................................................................................................................. 11
2.4.5 Final Sign Off ........................................................................................................................................... 11
2.5 Automation ..................................................................................................................................................... 12
2.6 Internationalization testing tools .................................................................................................................... 12
2.6.1 Locale Repository tool ............................................................................................................................. 12
2.6.2 Translation Repository tool ..................................................................................................................... 12
2.6.3 Truncation/Grey out Search tool ............................................................................................................ 13
3. Summary / Key take away ....................................................................................................................................... 14
4. Conclusion ............................................................................................................................................................... 14
5. Acknowledgements ................................................................................................................................................. 15
6. Abbreviations .......................................................................................................................................................... 16
3
Abstract Internationalization testing of an enterprise product handling faster release cycles for the Internationalization versions of the product. The white paper helps in understanding the Internationalization testing process overview, the phase and key milestones involved in Internationalization testing. The Internationalization testing phases involved Preparation and Execution. Preparation phase involves:
Feature review Test Cases review WR audit Test Case Porting
Execution phase involves: Pseudo Localization testing Beta Test Pass Language Test Pass End game testing
The language in scope for the product are categorized as Full, Partial and MUI (Multi User Interface) pack only. Full languages have completely Localized strings and are available in Localized OS format. Partial languages have few strings that are not localized and most of the language strings are localized. These languages are also available in Localized OS format. MUI Pack only languages do not have OS format. The languages will be applied as packs on an English OS or Full language OS. Execution takes place in Virtual Machines on supported OS (W2012R2, W8.1, W8 and Mac). Internationalization testing involves testing of the product on Mobiles and tablet as well. The inflow of requirements will be in the form of features. Every feature has their respective test cases. Test cases are identified as ‘Flagged for Localization’ by the Functional test team for each of the features in scope. These form for the base scope for Internationalization testing. Internationalization test team reviews the test cases and start with the actual test passes.
4
About the Author Ramkumar B, Test lead with 9.5 years of IT industry experience in Tata Consultancy Services Lt., on Manual testing out of which the last 1 year and 8 months has been in Internationalization/Internationalization testing. Expertise in Telecom and Manufacturing domains. Led team of 5-7 tester performing End-to-end testing and System integration testing in Telecom (British Telecomm client) and Manufacturing (General Motors – Client) domains. Currently leading a team of 15 testers performing Internationalization/Localization testing as part of Product testing. Guiding the testing team technically and managing the delivery of the testing releases across 44 languages that are supported by the product. Certified in MB2-700 MS CRM 2013, Microsoft SQL server and HP-QC
5
1. Introduction Internationalization testing being done for the world’s largest product company As part of Internationalization testing Culture, Functional and Cosmetic testing is carried out in 44 supported
languages Desktop, Mobile, On Premise and Online versions of the product are supported and are tested and certified Package and Component validations undertaken for components external to the product 2. Detailed Walk through
Below are the major topics that will be discussed in the course of this white paper
Key Activities Phases Best practices adapted Tools used 2.1 Key Activities Functional Verification – Verification of functionality in Non-English languages through execution of functional
test cases which are marked as Intl impacted Release Criteria (RC) test cases execution - RC test cases are sanity type test cases for each CRM component.
These test cases are executed to confirm the stability of earlier release functionalities on the current release builds First Access (FA) testing - First Access testing (Sanity testing) execution on CRM supported Non-English
languages Package Validation - Installation and Sanity checks on components external to CRM (Outlook, BIDS, Email router,
SRS etc.,)
2.2 Phases
There are two major phases in Internationalization testing into which all the testing milestones are classified:
Preparation phases o Feature review o Test case porting o Test case review o WR Audit
Execution phases
o Pseudo Localization Test Pass (PLTP) o Beta Test Pass (BTP) & Sign Off o Language Test Pass (LTP) o Final Sign Off
6
2.3 Preparation Phases 2.3.1 Feature Review Understand the functionalities implemented in the feature by walk through of functional specs from Test Management tool. The understanding of the feature will be then held in the form of Test scenarios identified
Intl Release activities kicks off with the Features being available for review in the Test Management tool The major input for this phase is the Functional Spec document. Review of a Feature can be done once the Functional spec is available for the feature in Test Management tool. Test cases (in the form of Test case query) will be available for the feature with status ‘Ready for Intl Review’ In case of functional and testing oriented clarifications, Intl testers contact the TCS Core teams. In case required help/information not available from functional teams, respective Feature owner will be contacted
for help. After reviewing the feature our understanding along with test scenarios identified for each feature will be created
Feature Re ie
7
2.3.2 Test Case Porting Port test cases that are flagged for localization from Product DB to Product Intl DB. This is necessary as the test cases will not be authored by Internationalization test team and the possible test scope will be flagged by Core functional team Test cases will be available for a feature in the form of Test case queries in Test Management tool Test Case porting is done from Product DB to Product Internationalization Specific DB Testers ensure that the Path available for a test case getting ported is available in Product Internationalization
Specific DB. If not, tester creates it before porting test cases. Feature wise Test case query for both Product DB and Product Internationalization specific DB added to Sync tool
in the form of an entry Run the Sync tool to find mis syncs (differences) between the test cases in Product DB and Product
Internationalization specific DB These differences will be identified and manually corrected in Test Management tool
2.3.3 Test Cases Review Test cases flagged for localization from each feature will be executed first on English by Internationalization test team to understand the test cases flow and clarify any issues with functional test team. At the end of the exercise, team will record the Time to complete the execution of each FFL test case Test cases are ported from Product DB to Product Internationalization specific DB and are ready for
Internationalization review. Testers execute the feature test cases (from Product Internationalization specific DB) in English to start with and
confirm if they are able to run the test steps. Testers update the ‘Time to Complete Case’ field with the actual time taken to execute the test case steps. In case of any clarifications, Internationalization testers contact the TCS Core functional team. If the required information is not available, the testers reach out to the Feature owner for help. Test Case Execution happens in Server or Client VM built through Test Environment tool using respective build
path
Test Case Porting
Test Cases Re ie
8
2.3.4 Test Configuration Matrix Test configuration matrix comprises of Locale (Language and Culture) to be tested, System configurations
including Server OS, Client OS and Browser detail. Test configuration matrix plays a vital role in deciding the approach and strategy for Internationalization testing. Test Configuration matrix provides an optimized way to decide the Language, Browser, Office, Device
configurations that are to be tested.
2.3.5 WR Audit
2.3.6 WR Audit
Perform Audit on the functional features from WR perspective (from 2 languages) and decide whether each feature passes or fails the audit
WR Audit takes place once the PLOC testing is completed for a feature. A bug qualified as WR audit if it is raised with ‘How found’ field set as ‘WR Audit’ and is not resolved as By
Design, Not Repro or External. A feature will be termed ‘WR Audit Passed’ as per specific Criteria A feature will be termed ‘WR Audit Failed’ as per specific Criteria
WR Audit
9
2.4 Execution Phases
2.4.1 Pseudo Localization Test Pass (PLTP)
Verify that product is localized along with pseudo strings substituted before actual translation is received for all the new functional features
PLOC testing happens on Pseudo builds. Execution of test cases is done on WR languages Test suite prepared by Test Leads Test Results will be logged in Test Suite path. Test Management tool is used in this phase. Execution happens in VM built through Test Environment tool
2.4.2 Beta Test Pass
Verify product is properly localized on beta languages. This execution will be normally done on the Beta builds
The input for this phase is the test cases (available as Test suites for all in scope languages) in Product Internationalization specific DB.
Beta Test Pass will be done on the beta branch of Product builds for the particular release. Test cases of the release will be executed on all Product supported beta languages Test Case Execution happens in Server or Client VM built through Test Environment tool.
Pseudo Localization
Test Pass
Beta Test Pass
10
2.4.3 Beta Sign Off
Verify that the sanity (Release Candidate) scenarios are stable on Beta builds (builds from which the beta build for the product release will be done
The input for this phase is the RC Test Cases. Beta Signoff testing will be done on Beta languages. LOC Beta Signoff testing is done once the Beta branch is created in Product Builds for the particular release. RC test cases are a Sanity set of test cases from a particular release that covers the End to End flow from all
features. The RC test cases will be prepared for each component and for each release. RC Test cases prepared for Release n (Current) appended with n-1 release. The appended RC test case will be the
RC Test cases for Release n. Test Case Execution happens on the Language Organizations that are provisioned and available on reference
environment. In scenarios where Organizations cannot be used, testing is done on Internet Enabled Deployment OnPrem servers There will be official Signoff mail from TCS Internationalization Test Leads stating if the Beta release is good for
Signoff with Exceptions
LOC Beta Sign Off
11
2.4.4 Language Test Pass
Verify product is properly localized on delta languages (excluding WR and beta languages). This execution will be normally done on the RTM (Release to Market) build of the product
The input for this phase is the test cases (available as Test suites for all in scope languages) in Product Internationalization specific DB.
Language Test Pass will be done on the Main branch of Product builds for the particular release. Test cases of the release will be executed on all Product supported languages excluding beta & WR languages Test Case Execution happens in Server or Client VM built through Test Environment tool.
2.4.5 Final Sign Off Verify that all Product packages deploy and are localized properly. Execute Release candidate test cases on the final RTM builds to confirm the basic functionalities are localized
Package validation, RC execution and Sanity Validation of components external to product Basic sanity of Components Test suite prepared for the RC test cases RC execution on Organizations refreshed RC test results logged RC Execution on Ref environment Organizations Sign up of Organizations and Sanity
Language Test Pass
Final Sign Off
12
2.5 Automation
The major activities involved in automation include Schedule LOC automated jobs, Report results, Failure analysis and debug Automation tools
Preparation of Schedule on Automation runs for a release Schedule Automation runs Pass comparison with base language If failed test cases fails again, run them manually For Product bug, Core functional bug will be logged For framework issues, Automation bug is logged Debug Automation and Test results related tools 2.6 Internationalization testing tools
2.6.1 Locale Repository tool
Centralized tool for checking locale verification points Generate localized test data with the help of tool Strings generated are stored in excel sheet for each the Product Modules Benefits: This tool is centralized place to find locale specific risk areas, strings for all Product supported languages This tool gives to find out locale specifications from both Language and risk area wise Test Planning can be done based on the set of languages and locale specifications those to be verified This tool generates localized strings which are useful to provide as inputs in Product 2.6.2 Translation Repository tool Obtain translated strings in all required languages Identify translated strings for all languages tested Specify if a particular string should be translated for a particular string Translation strings identification is easy when performing User Acceptance testing on huge volume of help content Benefits: String translations in all supported languages can be viewed in a single place instead of referring various sources All the approved translated string can be viewed in the tool Avoid Invalid bugs which are related to string translations Easily Adoptable, retrieves search results very fast and user friendly Easy to access: No installation is needed for this tool, just accessing the link
13
2.6.3 Truncation/Grey out Search tool To search strings in Language Code files of Localized Product Builds If a localized string/text appears truncated in product screens, tester can use this tool to identify the actual string If a localized string/text appears greyed out in product screens, tester can use this tool to identify the actual string
Benefits:
This tool searches quickly for strings in Language code files of Product build Expected results can be set clearly while raising bugs for truncated/greyed strings
14
3. Summary / Key take away Detailed walk through of the process involved in Internationalization testing Key activities Phases Key challenges 4. Conclusion Below are the points that will help in setting up and managing an Internationalization testing project: Internationalization test team ramp up involves estimation of the number of languages in scope Require recruiting linguistic expert to perform smoke testing on builds before actual testing kicks off Plan for utilizing specific resources for environment preparation to enable environment availability for testing and
effective utilization of resources’ productive hours Plan for usage of Regional locale related tools which gives complete information related to Locales
15
5. Acknowledgements Thanks to my Manager Srinivas Bachina for his relentless reviews that enabled me in identifying the improvement areas in the white paper. Special Thanks to my Director Sandru Veerabhadraiah for his microscopic reviews of this white paper
16
6. Abbreviations
Acronym Description
BTP Beta Test Pass
EULA End User License Agreement
FFL Flagged for Localization
FVT Functional Verification Testing
LCL Language Coded List
LTP Language Test Pass
MoCA Mobile Client Application
PLTP Pseudo Localization Test Pass
RC Release Criteria
SDET Software Development Engineer in Test
TTC Time to Complete Case
UA User Acceptance
VM Virtual Machine
WR World Readiness
Thank You