test for success! for success v2.pdf · 5 our testing model • we’ll present several testing...
Post on 08-Jul-2020
3 Views
Preview:
TRANSCRIPT
Produced by Wellesley Information Services, LLC, publisher of SAPinsider. © 2016 Wellesley Information Services. All rights reserved.
Test for Success! A deep dive into effective SAP BusinessObjects BI 4 test strategies
Alan Mayer Solid Ground Technologies, Inc.
1
In This Session
• Understand the role that testing plays in BI 4.1 environments
• Learn an overall model for testing
• Witness individual techniques via live demonstrations
• Gauge the effectiveness of each technique
• Create your own test plan from the strategies presented
2
What We’ll Cover
• Introduction
• Testing at the atomic level
• Comparing reports
• Verifying workflows
• Reconstructing scenarios
• Stressing systems
• Promotion strategies
• Wrap-up
3
Why Test?
• Catch behavior that software vendors may not have considered
• Identify and fix badly behaving reports / visualizations
• Guarantee to users reliable access to their content
• Verify that “advertised” changes are the only ones …
• Predict how your system will react under load
4
When to Test
• Before migrating to a new release
XI 3.1 to BI 4.x (Major)
BI 4.1 to BI 4.2 (Minor)
• Before applying a service pack or patch
• Before promoting content from between environments
Dev to Test
Test to Production
• Prior to changing any system that our BI systems
depend upon
Databases
Networks
5
Our Testing Model
• We’ll present several testing techniques in this session
• Each provides a measurable benefit based on your testing needs
• Knowing when to use the test is crucial
Look for the following labels in the upper right hand corner of each slide
• After presenting each technique, we’ll revisit the tests needed per event
Major / Minor
SP / Patch
Lifecycle
Other
6
When Time and Skills Matter
• Some companies opt out of testing for various reasons:
Takes too long
Don’t have the right number of people who know the technology
• We’ll help soften those excuses:
Each testing method will have an accelerated version
We’ll also post the skill level needed to perform the test
Skill levels rated from 1 – 10
Based on the application being tested
(Webi, IDT, …)
7
What We’ll Cover
• Introduction
• Testing at the atomic level
• Comparing reports
• Verifying workflows
• Reconstructing scenarios
• Stressing systems
• Promotion strategies
• Wrap-up
8
Atomic Level Testing Defined
• For each application used within your company:
Check every toolbar button
Check every menu level command
• Typical BI 4.1 applications include
Web Intelligence
Crystal Reports
Universe Designer
Information Design Tool
Major / Minor
9
Atomic Testing – Why?
• This seems to be a bit extreme
Surely a software vendor has validated their applications to this level
• The answer to the question depends on the application … and YOU
Many defects were found in BI 4.0 using this simple technique
Several were still found in BI 4.1
It also tests your environment and setup
Can you launch the Java applet required for editing?
Can you export to a shared filestore that all users
should have access to?
Can you send email notifications?
10
Atomic Testing – Documenting your Results
• Create an Excel workbook to capture the results
Add the menu-level command hierarchies
Add buttons by toolbar
This particular test revealed several interface
behaviors added “by design” that had to be
reworked
11
Atomic Testing – Accelerated
• If you don’t have time to test every command and option:
Test the most widely used buttons and menus
Use your workflows as a guide
Test the buttons / commands needed to execute the workflow
Ignore others for the sake of time
Accelerated: 1 week
Normal: 2 weeks
12
Atomic Testing – Skill Level
Execute and record the results of
invoking a toolbar button or command
Should have used the software before
to know what to expect
2
Create Workbook
4
Conduct Atomic Tests
Create the workbook of toolbar
buttons and menu commands
Have to know what is available
from the application
Reference guides help here
Skill levels range from 1 (Green – easiest)
to 10 (Red – most advanced)
14
What We’ll Cover
• Introduction
• Testing at the atomic level
• Comparing reports
• Verifying workflows
• Reconstructing scenarios
• Stressing systems
• Promotion strategies
• Wrap-up
15
Report Comparison - Defined
• Refresh the report in both versions at the same time
Scheduling the refresh to process more reports at a time
• Compare the report contents in each version
Data
Format
• Identify and resolve differences
May involve reworking the report
Defect-related differences should be resolved
by the vendor
Major / Minor
SP / Patch
16
Report Comparison – Why?
• Differences in versions can affect report content
Calculation engine changes can easily be caught
• If this testing step is skipped, users might catch the difference
Usually occurs after go-live
Lowers confidence in your newly upgraded software
17
Report Comparison – Manual Inspection
• Compile a list of inspection points that must match between versions
Queries
Headers / footers
Tables
Graphs
Fonts
Alignment
Sections
Number of pages
…
18
Report Comparison – Accelerated
• Focus on the mission-critical reports first
• Third party tools can greatly accelerate the spotting of problems
Schedule reports to both Excel and PDF formats
Use an Excel comparison tool to spot any data-related problems
Us a PDF comparison tool to spot format-related issues
These tools may not accelerate the resolution of problem reports
They will separate the good one from the bad, however
Accelerated: < 1 week
Normal: 4 weeks
19
Report Compare – Skill Level
Skill level drops if
additional tools used
2
Collect critical content Compare
Should know what is important
Made easier by looking for
critical event-driven schedules
5 2
Manual Automated
Discover what is wrong
Correct content -or-
contact vendor
Diagnose
7
20
Report Comparison – Additional Notes 1
• DO NOT use the Administrator account for testing
This is a common (but bad) practice at many companies
Avoids the time needed to apply proper security
Administrator accounts will not correctly test all environment features
Example: Universe restrictions
Not applied when using the Administrator account
Not applied to members of the Administrators group
• Gather a list of typical users by role for each business area
• Use their accounts for more realistic testing
21
Report Comparison – Additional Notes 2
• DO use production data to test production reports
Testing production reports against test data has its own issues
Dev and test databases may not be able to refresh production reports
Query conditions within those reports must be rewritten to reflect
the available data
The final result may not match reality
• A production break-fix environment comes in very handy here
A production environment that has access to the same data users have
Not widely available to all users
Usually just administrators
23
What We’ll Cover
• Introduction
• Testing at the atomic level
• Comparing reports
• Verifying workflows
• Reconstructing scenarios
• Stressing systems
• Promotion strategies
• Wrap-up
24
Workflow Definition
• Test the common actions a user conducts with this application
Log in / out
Create a document
Create a data provider (query)
Refresh all data providers
Schedule a report
Add breaks to a report
Fold / unfold the breaks
Format a report (shading / fonts)
Create a document link
…
Major / Minor
SP / Patch
Other
25
Workflows – Why?
• Workflows test series of actions
Unlike atomic testing which tests an individual action
Unlike report comparisons which test overall data / formatting accuracy
• Workflow testing will uncover other problems
• Particularly suited for quick tests
All workflows can usually be tested within
an hour or two
26
Workflows – Accelerated
• Testing workflows is fairly quick as it is
• Narrow the workflows even further to the most popular for your company
Don’t test element linking between blocks if this feature is not used
Accelerated: < 2 hours
Normal: 1 day
27
Workflows – Skill Level
Tester should have the skills of an
average users for that application
5
Document Workflows
5
Execute Workflows
28
Workflows – Additional Notes
• The same notes mentioned for report comparisons apply to workflows
• DO NOT use the Administrator account for testing
• Gather a list of typical users by role for each business area
30
What We’ll Cover
• Introduction
• Testing at the atomic level
• Comparing reports
• Verifying workflows
• Reconstructing scenarios
• Stressing systems
• Promotion strategies
• Wrap-up
31
Scenario Testing - Definition
• Rebuilding a complicated document or visualization from scratch
• The goal is to document any trouble spots while building
It is NOT to create a picture-perfect duplicate report
Major / Minor
SP / Patch
32
Scenario Testing – Why?
• Trying to reconstruct a finished document will uncover
other problems
We are providing a comprehensive testing strategy
but are still human
Might have missed an important workflow that will be
uncovered by this step
33
Scenario Testing – Accelerated
• Try this testing step with at least ONE complicated report
• Do not skimp on the level of complexity
That is the entire reason this step was included
Accelerated: < 1 week
Normal: 2 weeks
34
Scenario Testing – Skill Level
Recreate a complicated report or
visualization from scratch
9
Recreate Content
36
What We’ll Cover
• Introduction
• Testing at the atomic level
• Comparing reports
• Verifying workflows
• Reconstructing scenarios
• Stressing systems
• Promotion strategies
• Wrap-up
37
Stress Testing - Definition
• Simulate a very busy processing day for your BI system
This is done by subjecting to high levels of activity
Adhoc queries and reports
Schedules
Publications
Visualizations
• Measure how well your system responds under load
• Use those results to reconfigure the system and retest
• Use free performance testing tools like jmeter to
configure and run these simulations
Major / Minor
SP / Patch
38
Stress Testing – Why?
• Stress testing allows you to see how your system will respond under high load
• This event will occur whether you conduct this test or not
It’s called GOING LIVE
39
• Rather than trying to simulate a complete BI environment, focus on a
simple workflow
Login and refresh a report
• Script that stress test and simulate hundreds of users running it
• Batches of schedules / publications can be created and launched via custom events
This will simulate high periods of scheduled activity
Accelerated: 2 – 4 days
Normal: 2 weeks
Stress Testing – Accelerated
Accelerated: 2 – 4 days
Normal: 2 weeks
40
Stress Testing – Skill Level
Batch schedules and trigger by custom
event
9
Create Volume Test Script
6
Create Scheduling Payloads
Simulates high levels of
concurrent activity
42
What We’ll Cover
• Introduction
• Testing at the atomic level
• Comparing reports
• Verifying workflows
• Reconstructing scenarios
• Stressing systems
• Promotion strategies
• Wrap-up
43
Promotion Testing - Definition
• Test changes to reports, universes, and visualizations before
• promoting them
Typical promotions include:
Dev to Test
Test to Production
• Know what changes were made in the current
environment (Dev, Test)
• Compare that against the current copy in the
next higher level (Test, Prod)
Lifecycle
44
Promotion Testing – Why?
• Without evaluating changes to reports and universes, other changes may
slip through
• These may or may not be intentional
• Those additional changes can have unintended consequences
• Other great reasons besides validating the number of changes
Acts as a kind of “code review”
Evaluate changes:
For reports:
Does the report refresh / calculate correctly with those changes in place?
For universes:
Is the correct SQL being generated?
Are query contexts still being generated correctly?
45
Promotion Testing – Accelerated
• Metadata tools can help identify which changes were made
SAP Information Steward
Other third party metadata tools
• Need the “deep scan” of the report or universe in question
Deep metadata scans will dissect the object into
component pieces
Reports: Formulas, variables, tables,
charts, …
Universes: Folders, objects, tables, joins,
contexts, …
• Once deep scanned, the component pieces can be
verified before / after the change Accelerated: 1 hour
Normal: 1 day
46
Promotion Testing – Skill Level
Provide sample reports that
can be run before / after the
change
Report comparison
techniques can be used to
evaluate the results
2
Identify changes Compare sample content
List all components that
have changed
5 7
Manual Automated
48
What We’ll Cover
• Introduction
• Testing at the atomic level
• Comparing reports
• Verifying workflows
• Reconstructing scenarios
• Stressing systems
• Promotion strategies
• Wrap-up
49
Testing by Activity
• Here’s the summarized list of tests that should be run by activity:
Major / Minor Releases SP / Patch Lifecycle Other (Network / DB)
• Atomic
• Report Comparison
• Workflow
• Scenario
• Stress
• Report Comparison
• Workflow
• Scenario
• Stress
• Promotion • Workflow
50
Where to Find More Information
• Alan Mayer, “SAP BusinessObjects Web Intelligence 4.x Calculation Engine Changes”,
(Reporting and Analytics INTERACTIVE, November 2015)
• Carly Thomas, “Calculation Engine Changes for Web Intelligence BI4 and XI31”, (SCN
http://tinyurl.com/lq5g7vw)
• Brown and Rapp, “Crank Your BI Performance up to 11 – Sizing, Tuning, & Performance
Testing”, (ASUG SABOUG 2015, September 2015)
• Brown, Mayer, and NS, “BI 4.1 Admin Myths and Legends”, (ASUG SABOUG 2015,
September 2015)
• James Rapp, “Creating Your First Test Plan”, (SCN, http://tinyurl.com/gnvn7lc)
• James Rapp, “Performance Testing in BI 4.1”, (SCN http://tinyurl.com/h48ya8l)
51
7 Key Points to Take Home
• Realize the importance testing plays to maintain or upgrade BI 4.1 systems
• Understand the various testing techniques available
• Justify the use of each technique and the ramifications if testing is avoided
• Know when to test
• Use accelerated techniques when necessary to reduce overall time involved
• Involve the right personnel based on skill level needed
• Create personalized test plans based on your own requirements
52
Your Turn!
How to contact me:
Alan Mayer
Email: alan.mayer@solidgrounded.com
Twitter: @solidgrounded
Please remember to complete your session evaluation
53
SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other
countries. All other product and service names mentioned are the trademarks of their respective companies. Wellesley Information Services is neither owned nor controlled by SAP SE.
Disclaimer
top related