Test apps 1
Creating manual tests using the web portal 4
Customizing Code Coverage Analysis 12
Developer testing tools scenarios and capabilities 21
Exploratory test web apps directly in your browser 28
Exploratory Testing - Capture, Create and Collaborate 32
How to Configure Unit Tests to Target An Earlier Version of the .NET Frame 35
IntelliTest Reference Manual 40
Introduction to Coded UI Tests with Visual Studio Enterprise 2015 43
Introduction to IntelliTest with Visual Studio Enterprise 2015 67
Manual and exploratory testing scenarios and capabilities 85
Parallel and Context Sensitive Test Execution 92
Unit testing native code with Test Explorer 98
Unit Testing, Code Coverage and Code Clone Analysis with Visual Studio 2015 110
Manual and exploratory testing
Overview (manual-exploratory-
testing/manual-testing)
Get started (../work/kanban/add-run-
update-tests)
Test with Visual Studio Team Services
(manual-exploratory-testing/getting-
started/create-a-test-plan)
Test with Team Foundation Server
Performance testing
Overview (performance-
testing/performance-testing)
Get started (performance-
testing/getting-started/getting-
started-with-performance-testing)
Performance test your app
(performance-testing/run-
performance-tests-app-before-release)
Test apps early and often
Team Services | Visual Studio 2015 | Previous version (https://msdn.microsoft.com/library
/ms182409%28v=vs.120%29.aspx)
Test continuously while you code, build, and deploy your app. Find problems before launching your
app or updates into production. You can then better assess whether your app meets your customers'
needs and is ready for release.
Team Services (/docs/overview) > (/docs/test/overview)
Test apps early and often | Visual Studio Team Services and Team Founda... https://www.visualstudio.com/docs/test/overview
1 of 3 02.09.2016 13:24
(https://msdn.microsoft.com/library
/dd380763%28v=vs.140%29.aspx)
Test with Microsoft Test Manager
(https://msdn.microsoft.com/library
/jj635157%28v=vs.140%29.aspx)
URL-based load tests (performance-
testing/getting-started/get-started-
simple-cloud-load-test)
Test Azure web apps (performance-
testing/app-service-web-app-
performance-test)
Continuous testing
Overview (continuous-
testing/continuous-testing)
Get started (continuous-
testing/getting-started/getting-
started-with-continuous-testing)
Run tests with your builds (continuous-
testing/test-build)
Run Selenium tests (continuous-
testing/getting-started/continuous-
test-selenium)
Test Java apps (continuous-
testing/getting-started/continuous-
test-java)
Developer tools and unit testing
Overview (developer-
testing/developer-testing)
Get started (developer-testing/getting-
started/getting-started-
with-developer-testing)
Unit test your code
(https://msdn.microsoft.com/library
/dd264975)
Perform system tests with Visual Studio
(https://msdn.microsoft.com/library
/jj620889%28v=vs.140%29.aspx)
Automate system tests
(https://msdn.microsoft.com/library
/ff472576%28v=vs.140%29.aspx)
Set up test environments
Set up test machines
(https://msdn.microsoft.com/library
/dd293551%28v=vs.140%29.aspx)
Administer your test lab
(https://msdn.microsoft.com/library
/dd936084%28v=vs.140%29.aspx)
Troubleshoot lab management
(https://msdn.microsoft.com/library
/ee853230%28v=vs.140%29.aspx)
Extend your test tools
API reference for test tools
(https://msdn.microsoft.com/library
/dd465178.aspx)
Help and support
Submit bugs through Connect (https://connect.microsoft.com/visualstudio), make suggestions on
Uservoice (http://visualstudio.uservoice.com/forums/121579-visual-studio), and send quick thoughts
Test apps early and often | Visual Studio Team Services and Team Founda... https://www.visualstudio.com/docs/test/overview
2 of 3 02.09.2016 13:24
using the Send-a-Smile link in the Visual Studio, Team Services, or TFS title bar. We look forward to
your feedback.
© 2016 Microsoft
Test apps early and often | Visual Studio Team Services and Team Founda... https://www.visualstudio.com/docs/test/overview
3 of 3 02.09.2016 13:24
Creating manual tests using the web portal
Create manual test cases to check that each of the deliverables meet your users' needs. You organize your test cases by
adding test cases to test suites.
Create manual test cases
If you haven't already, create a test plan and requirement-based test suites.1.
From the Test hub in the web portal, choose a requirement-based test suite. Then create a test case for that suite.
A requirement-based test suite is created from a backlog item. If you add a test case to this type of suite, it is
automatically linked to that backlog item.
2.
Add test steps with actions and expected results so that any team member can run the test. You can add
attachments to a step if you want.
3.
Creating manual tests using the web portal https://msdn.microsoft.com/en-us/library/dd286729(d=printer).aspx
1 of 8 02.09.2016 14:22
Now you have created a test case that you can run.
Q&AQ: Can I add an extra line into a test step?
A: Yes. Press Shift+Enter in the action or expected results field to add an extra line.
Q: How do I insert a test step into a test case?
A: Select a test step. Press Alt+P to insert a test step above the selected step.
Q: Is there a way to quickly add multiple test cases at the same time?
A: Yes. Use the grid view when you add test cases to the test suite.
Creating manual tests using the web portal https://msdn.microsoft.com/en-us/library/dd286729(d=printer).aspx
2 of 8 02.09.2016 14:22
In the grid view, use the context menu to insert, clear or delete a row.
Q: Can I bulk edit multiple test cases?
A: Yes. Switch the view from list to grid. All the test cases for the current test suite will be displayed in the grid with their
test steps. This is a helpful view if you want to review your test cases with other team members jointly. When you review,
you can update and add new test cases.
Creating manual tests using the web portal https://msdn.microsoft.com/en-us/library/dd286729(d=printer).aspx
3 of 8 02.09.2016 14:22
Or you can filter and sort the test cases. Then select only the ones that you want to bulk edit.
To return to the test suite view, switch the view from grid back to list.
Q: Can I copy test cases and test steps from an existing Excel worksheet?
A: Yes. Copy the columns from Excel that you want to use for the title, action and expected results fields. Any formatting
in the columns, other than multiline, will not be copied into the worksheet. Paste these columns into the grid view, edit if
necessary, and save them. (This is only supported for Internet Explorer and Chrome browsers.)
Q: Can I copy test cases from the grid view to an Excel worksheet?
A: Yes. Copy the data and paste it into your Excel worksheet. Any formatting in the test steps, other than multiline, will not
be copied into the worksheet. (This is only supported for Internet Explorer and Chrome browsers.)
Q: Can I tag test cases so that I can view only tests with specific tags?
A: Yes. Tag test cases in a suite with any tag that is important to you. For example, tag all the tests related to login so that
Creating manual tests using the web portal https://msdn.microsoft.com/en-us/library/dd286729(d=printer).aspx
4 of 8 02.09.2016 14:22
you can rerun these tests if a bug is fixed for the login page. Then you can filter on that tag from the test hub. You can add
and edit tags when you edit a test case, or bulk edit tags in the grid view.
Q: Can I share test steps between test cases?
A: Yes. Choose the steps that you want to share. More details are here.
Q: Can I add parameters to a test case to run multiple times with different data?
A: Yes. Choose a test step and then add the parameter. Scroll down to add values for a parameter. More details are here.
Creating manual tests using the web portal https://msdn.microsoft.com/en-us/library/dd286729(d=printer).aspx
5 of 8 02.09.2016 14:22
Q: Can I share the data for parameters between test cases?
A: Yes. That way test cases that use the same parameters can be run with the same data for consistent results. Convert
existing parameters to shared parameters.
After you have created a shared parameter set, open another test case and add the shared parameter set to that test case.
More details are here.
Add, edit and rename your shared parameter sets from the parameters tab. View the test cases that reference them in the
test cases pane.
Creating manual tests using the web portal https://msdn.microsoft.com/en-us/library/dd286729(d=printer).aspx
6 of 8 02.09.2016 14:22
Each shared parameter set is a work item. From the Properties tab, you can view or make changes to this work item. For
example, you can assign owners and track changes.
From Microsoft Test Manager using Test Professional or Visual Studio Enterprise, you can view shared parameters for a
test case but you cannot update them. To update shared parameters, you must use the web portal. There is a hyperlink in
Microsoft Test Manager that takes you to the correct location to edit the shared parameters.
Q: How do I reassign a test to another tester?
A: Select the tests that you want to reassign. Then choose the tester who you want to run the tests.
Q: How can I find out if a test case has been added to other test suites?
A: Select a test case, then choose to view the test suites details. The associated test suites pane shows you any test suite
Creating manual tests using the web portal https://msdn.microsoft.com/en-us/library/dd286729(d=printer).aspx
7 of 8 02.09.2016 14:22
for any test plan that contains this test case. This includes all team projects. Click the test suite to view it. Hover over the
test suite in the pane to view the team project and the test plan for that test suite.
Q: What happens when I delete a test case from a requirement-based test suite?
A: The test case still exists in your team project, but the test case is removed from the test suite. In addition, it is no longer
linked to the backlog item for that test suite.
Try this nextRunning manual tests using the web portal
Or, dig deeper:Share steps between test cases
Test configurations: specifying test platforms
Repeat a test with different data
Explore your app without planning in advance.
© 2016 Microsoft
Creating manual tests using the web portal https://msdn.microsoft.com/en-us/library/dd286729(d=printer).aspx
8 of 8 02.09.2016 14:22
Customizing Code Coverage Analysis
By default, the Visual Studio Code Coverage tool analyzes all solution assemblies (.exe/.dll) that are loaded during unit tests.
We recommend that you retain this default, because it works well most of the time. For more information, see Using Code
Coverage to Determine How Much Code is being Tested.
Before customizing the code coverage behavior, consider some alternatives:
I want to exclude the test code from the code coverage results and include only the application code.
Add the ExcludeFromCodeCoverage Attribute to your test class.
I want to include assemblies that are not part of my solution.
Obtain the .pdb files for these assemblies and copy them into the same folder as the assembly .dll files.
To customize the code coverage behavior, copy the sample at the end of this topic and add it to your solution using the file
extension .runsettings. Edit it to your own needs, and then on the Test menu, choose Test Settings, Select Test Settings
file. The remainder of this topic describes this procedure in more detail.
The .runsettings fileAdvanced code coverage settings are specified in a .runsettings file. This is the configuration file used by unit testing
tools. We recommend you copy the sample at the end of this topic and edit it to suit your own needs.
What happened to the .testsettings file I used in Visual Studio 2010?
In Visual Studio 2010, the .testsettings file applies only to unit tests based on the MSTest framework. In Visual
Studio 2012, the testing tools apply not only to MSTest, but also other frameworks such as NUnit and xUnit.net.
The .testsettings file will not work with these. The .runsettings file is designed to customize the test tools in a way
that works with all testing frameworks.
To customize code coverage, you will need to add a .runsettings file to your solution:
Add an .xml file as a solution item with the extension .runsettings:
In Solution Explorer, on the shortcut menu of your solution, choose Add, New Item, and select XML File. Save the
file with a name ending such as CodeCoverage.runsettings
1.
Add the content given in the sample at the end of this topic, and then customize it to your needs as described in
the following sections.
2.
On the Test menu, choose Test Settings, Select Test Settings File and select the file.3.
Visual Studio 2015
Customizing Code Coverage Analysis https://msdn.microsoft.com/en-us/library/jj159530(d=printer).aspx
1 of 9 02.09.2016 13:45
Now when you run Analyze Code Coverage, this .runsettings file will control its behavior. Don’t forget that you
must run code coverage again: your previous coverage results and code coloring aren’t automatically hidden when
you run tests or update your code.
4.
To turn the custom settings off and on, deselect or select the file in the Test, Test Settings menu.5.
Other aspects of unit tests can be configured in the same .runsettings file. For more information, see Unit Test Your Code.
Specifying symbol search paths
Code coverage requires symbols (.pdb files) for assemblies to be present. For assemblies built by your solution, symbol
files are generally present alongside the binary files, and code coverage works automatically. But in some cases, you
might want to include referenced assemblies in your code coverage analysis. In such cases, the .pdb files might not be
adjacent to the binaries, but you can specify the symbol search path in the .runsettings file.
Warning
Symbol resolution can take time, especially when using a remote file location with a lot of assemblies. Therefore,
consider copying remote .pdb files to the same local location as the binary (.dll and .exe) files.
Excluding and including
You can exclude specified assemblies from code coverage analysis. For example:
<SymbolSearchPaths>
<Path>\\mybuildshare\builds\ProjectX</Path>
<!‐‐More paths if required‐‐>
</SymbolSearchPaths>
<ModulePaths>
<Exclude>
<ModulePath>Fabrikam.Math.UnitTest.dll</ModulePath>
<!‐‐ Add more ModulePath nodes here. ‐‐>
XML
minterastlib
Customizing Code Coverage Analysis https://msdn.microsoft.com/en-us/library/jj159530(d=printer).aspx
2 of 9 02.09.2016 13:45
As an alternative, you can specify which assemblies should be included. This approach has the drawback that when you
add more assemblies to the solution, you have to remember to add them to the list:
If <Include> is empty, then code coverage processing includes all assemblies (.dll and .exe files) that are loaded and
for which .pdb files can be found, except for items that match a clause in an <Exclude> list.
Include is processed before Exclude.
Regular expressions
Include and exclude nodes use regular expressions. For more information, see Using Regular Expressions in Visual
Studio. Regular expressions are not the same as wildcards. In particular:
.* matches a string of any characters1.
\. matches a dot ".")2.
\( \) matches parentheses "( )"3.
\\ matches a file path delimiter "\"4.
^ matches the start of the string5.
$ matches the end of the string6.
All matches are case-insensitive.
For example:
</Exclude>
</ModulePaths>
<ModulePaths>
<Include>
<ModulePath>Fabrikam.Math.dll</ModulePath>
<!‐‐ Add more ModulePath nodes here. ‐‐>
</Include>
</ModulePaths>
<ModulePaths>
<Include>
<!‐‐ Include all loaded .dll assemblies (but not .exe assemblies): ‐‐>
<ModulePath>.*\.dll$</ModulePath>
</Include>
<Exclude>
<!‐‐ But exclude some assemblies: ‐‐>
minterastlib
XML
Customizing Code Coverage Analysis https://msdn.microsoft.com/en-us/library/jj159530(d=printer).aspx
3 of 9 02.09.2016 13:45
Warning
If there is an error in a regular expression, such as an unescaped and unmatched parenthesis, then code coverage
analysis will not run.
Other ways to include or exclude elements
See the sample at the end of this topic for examples.
ModulePath – Assemblies specified by assembly file path.
CompanyName – matches assemblies by the Company attribute.
PublicKeyToken – matches signed assemblies by the public key token. For example to match all Visual Studio
components and extensions, use <PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>.
Source – matches elements by the path name of the source file in which they are defined.
Attribute – matches elements to which a particular attribute is attached. Specify the full name of the attribute,
including "Attribute" at the end of the name.
Function – matches procedures, functions, or methods by fully qualified name.
Matching a function name
Your regular expression must match the fully qualified name of the function, including namespace, class name, method
name and parameter list. For example,
C# or Visual Basic: Fabrikam.Math.LocalMath.SquareRoot(double)
C++: Fabrikam::Math::LocalMath::SquareRoot(double)
<ModulePath>.*\\Fabrikam\.MyTests1\.dll$</ModulePath>
<!‐‐ Exclude all file paths that contain "Temp": ‐‐>
<ModulePath>.*Temp.*</ModulePath>
</Exclude>
</ModulePaths>
<Functions>
<Include>
<!‐‐ Include methods in the Fabrikam namespace: ‐‐>
<Function>^Fabrikam\..*</Function>
<!‐‐ Include all methods named EqualTo: ‐‐>
<Function>.*\.EqualTo\(.*</Function>
XML
Customizing Code Coverage Analysis https://msdn.microsoft.com/en-us/library/jj159530(d=printer).aspx
4 of 9 02.09.2016 13:45
How to specify .runsettings files while running tests
To customize runsettings in Visual Studio tests
Choose Test, Test Settings, Select Test Settings File and select the .runsettings file. The file appears on the Test
Settings menu, and you can select or cancel it. While selected, your .runsettings file applies whenever you use Analyze
Code Coverage.
To customize run settings in a command line test
To run tests from the command line, use vstest.console.exe. The settings file is a parameter of this utility. For more
information, see Using VSTest.console from the command line.
Launch the Visual Studio Developer Command Prompt:
On Windows Start, choose All Programs, Microsoft Visual Studio, Visual Studio Tools, Developer
Command Prompt.
1.
Run:
vstest.console.exe MyTestAssembly.dll /EnableCodeCoverage
/Settings:CodeCoverage.runsettings
2.
To customize run settings in a build definition
You can get code coverage data from a team build.
</Include>
<Exclude>
<!‐‐ Exclude methods in a class or namespace named UnitTest: ‐‐>
<Function>.*\.UnitTest\..*</Function>
</Exclude>
</Functions>
Customizing Code Coverage Analysis https://msdn.microsoft.com/en-us/library/jj159530(d=printer).aspx
5 of 9 02.09.2016 13:45
Make sure your .runsettings file is checked in.1.
In Team Explorer, open Builds, and then add or edit a build definition.2.
On the Process page, expand Automated Tests, Test Source, Run Settings. Select your .runsettings file.
But Test Assembly appears instead of Test Source. When I try to set the Run Settings field, I can only
select .testsettings files.
Under Automated Tests, select Test Assembly, and choose [...] at the end of the line. In the Add/Edit
Test Run dialog box, set Test Runner to Visual Studio Test Runner.
3.
The results are visible in the summary section of the build report.
Sample .runsettings fileCopy this code and edit it to suit your own needs. This is the default .runsettings file.
(For other uses of the .runsettings file, see Configure unit tests by using a .runsettings file.)
<?xml version="1.0" encoding="utf‐8"?>
<!‐‐ File name extension must be .runsettings ‐‐>
<RunSettings>
<DataCollectionRunSettings>
XML
Customizing Code Coverage Analysis https://msdn.microsoft.com/en-us/library/jj159530(d=printer).aspx
6 of 9 02.09.2016 13:45
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft
/CodeCoverage/2.0"
assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector,
Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral,
PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<!‐‐
Additional paths to search for .pdb (symbol) files. Symbols must be found for modules
to be instrumented.
If .pdb files are in the same folder as the .dll or .exe files, they are automatically
found. Otherwise, specify them here.
Note that searching for symbols increases code coverage runtime. So keep this small and
local.
‐‐>
<!‐‐
<SymbolSearchPaths>
<Path>C:\Users\User\Documents\Visual Studio 2012\Projects\ProjectX
\bin\Debug</Path>
<Path>\\mybuildshare\builds\ProjectX</Path>
</SymbolSearchPaths>
‐‐>
<!‐‐
About include/exclude lists:
Empty "Include" clauses imply all; empty "Exclude" clauses imply none.
Each element in the list is a regular expression (ECMAScript syntax). See
http://msdn.microsoft.com/library/2k3te2cs.aspx.
An item must first match at least one entry in the include list to be included.
Included items must then not match any entries in the exclude list to remain included.
‐‐>
<!‐‐ Match assembly file paths: ‐‐>
<ModulePaths>
<Include>
<ModulePath>.*\.dll$</ModulePath>
<ModulePath>.*\.exe$</ModulePath>
</Include>
<Exclude>
<ModulePath>.*CPPUnitTestFramework.*</ModulePath>
</Exclude>
</ModulePaths>
<!‐‐ Match fully qualified names of functions: ‐‐>
<!‐‐ (Use "\." to delimit namespaces in C# or Visual Basic, "::" in C++.)
‐‐>
<Functions>
<Exclude>
<Function>^Fabrikam\.UnitTest\..*</Function>
<Function>^std::.*</Function>
<Function>^ATL::.*</Function>
<Function>.*::__GetTestMethodInfo.*</Function>
<Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*
Customizing Code Coverage Analysis https://msdn.microsoft.com/en-us/library/jj159530(d=printer).aspx
7 of 9 02.09.2016 13:45
</Function>
<Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
</Exclude>
</Functions>
<!‐‐ Match attributes on any code element: ‐‐>
<Attributes>
<Exclude>
<!—Don't forget "Attribute" at the end of the name ‐‐>
<Attribute>^System\.Diagnostics\.DebuggerHiddenAttribute$</Attribute>
<Attribute>^System\.Diagnostics
\.DebuggerNonUserCodeAttribute$</Attribute>
<Attribute>^System\.Runtime
\.CompilerServices.CompilerGeneratedAttribute$</Attribute>
<Attribute>^System\.CodeDom
\.Compiler.GeneratedCodeAttribute$</Attribute>
<Attribute>^System\.Diagnostics
\.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>
</Exclude>
</Attributes>
<!‐‐ Match the path of the source files in which each method is defined: ‐‐>
<Sources>
<Exclude>
<Source>.*\\atlmfc\\.*</Source>
<Source>.*\\vctools\\.*</Source>
<Source>.*\\public\\sdk\\.*</Source>
<Source>.*\\microsoft sdks\\.*</Source>
<Source>.*\\vc\\include\\.*</Source>
</Exclude>
</Sources>
<!‐‐ Match the company name property in the assembly: ‐‐>
<CompanyNames>
<Exclude>
<CompanyName>.*microsoft.*</CompanyName>
</Exclude>
</CompanyNames>
<!‐‐ Match the public key token of a signed assembly: ‐‐>
<PublicKeyTokens>
<!‐‐ Exclude Visual Studio extensions: ‐‐>
<Exclude>
<PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>
<PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>
<PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>
<PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>
<PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>
<PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>
<PublicKeyToken>^E361AF139669C375$</PublicKeyToken>
</Exclude>
</PublicKeyTokens>
Customizing Code Coverage Analysis https://msdn.microsoft.com/en-us/library/jj159530(d=printer).aspx
8 of 9 02.09.2016 13:45
See AlsoUsing Code Coverage to Determine How Much Code is being Tested
Unit Test Your Code
© 2016 Microsoft
<!‐‐ We recommend you do not change the following values: ‐‐>
<UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
<CollectFromChildProcesses>True</CollectFromChildProcesses>
<CollectAspDotNet>False</CollectAspDotNet>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
Customizing Code Coverage Analysis https://msdn.microsoft.com/en-us/library/jj159530(d=printer).aspx
9 of 9 02.09.2016 13:45
Developer testing tools scenarios andcapabilities
Visual Studio 2015 | Previous version (https://msdn.microsoft.com/library
/dd264943%28v=vs.120%29.aspx)
Maintain code health with unit testing. Visual Studio and Visual Studio Team Services provide a wide
range of powerful tools and techniques for developers to use when testing applications:
Scenarios and capabilities:
Avoid regressions and achieve code coverage with IntelliTest
User interface testing with Coded UI and Selenium
Effective unit testing with Visual Studio Code Coverage
Unit testing with any framework using the high performance Test Explorer
Get started with developer testing tools (getting-started/getting-started-with-developer-
testing)
Avoid regressions and achieve code coverage with IntelliTest
In traditional unit test suites, each test case represents an exemplary usage scenario, and the
assertions embody the relationship between the input and output. Verifying a few such scenarios
might well be enough, but experienced developers know that bugs lurk even in well-tested code,
when correct but untested inputs provoke wrong responses.
Improve coverage, and avoid regressions with IntelliTest. IntelliTest dramatically reduces the effort to
create and maintain unit tests for new or existing code.
Team Services (/docs/overview) > (/docs/test/overview)
Maintain code health with unit testing in Visual Studio Team Services https://www.visualstudio.com/docs/test/developer-testing/developer-testing
1 of 7 02.09.2016 13:26
Introduction to IntelliTest with Visual Studio Enterprise 2015 (http://download.microsoft.com
/download/6/2/B/62B60ECE-B9DC-4E8A-A97C-EA261BFB935E
/Docs/Introduction%20to%20IntelliTest%20with%20Visual%20Studio%20Enterprise%202015.docx
IntelliTest – One Test to rule them all (http://blogs.msdn.com/b/visualstudioalm/archive/2015/07
/05/intellitest-one-test-to-rule-them-all.aspx)
IntelliTest Videos (https://channel9.msdn.com/series/test-tools-in-visual-studio)
Get Started with IntelliTest (https://msdn.microsoft.com/library/dn823749.aspx)
IntelliTest reference manual (intellitest-manual/index)
User interface testing with Coded UI and Selenium
Test your user interface (UI) with best of breed or community approved UI Testing. Coded UI tests
provide a way to create fully automated tests to validate the functionality and behavior of your
application’s user interface. They can automate UI testing across a variety of technologies, including
XAML-based Windows Store apps, browser apps, and SharePoint apps.
Whether you choose best of breed Coded UI Tests or generic browser based UI testing with Selenium,
Visual Studio provides all the tools you need.
Maintain code health with unit testing in Visual Studio Team Services https://www.visualstudio.com/docs/test/developer-testing/developer-testing
2 of 7 02.09.2016 13:26
Use UI Automation To Test Your Code (https://msdn.microsoft.com/library/dd286726.aspx)
Get started creating, editing and maintaining a coded UI test (https://msdn.microsoft.com
/library/ff977233.aspx)
Test Windows Store 8.1 Apps with Coded UI Tests (https://msdn.microsoft.com/library
/dn305948.aspx)
Test Windows Phone 8.1 Apps with Coded UI Tests (https://msdn.microsoft.com/library
/dn747198.aspx)
Test SharePoint 2010 Applications with Coded UI Tests (https://msdn.microsoft.com/library
/jj715713.aspx)
Introduction to Coded UI Tests with Visual Studio Enterprise 2015 (Lab)
(http://download.microsoft.com/download/6/2/B/62B60ECE-B9DC-4E8A-A97C-EA261BFB935E
/Docs/Introduction%20to%20Coded%20UI%20Tests%20with%20Visual%20Studio%20Enterprise%202015.docx
Effective unit testing with Visual Studio Code Coverage
To determine what proportion of your project’s code is actually being tested by coded tests such as
unit tests, you can use the code coverage feature of Visual Studio. To guard effectively against bugs,
your tests should exercise or ‘cover’ a large proportion of your code.
Maintain code health with unit testing in Visual Studio Team Services https://www.visualstudio.com/docs/test/developer-testing/developer-testing
3 of 7 02.09.2016 13:26
Code coverage analysis can be applied to both managed (CLI) and unmanaged (native) code.
Code coverage is an option when you run test methods using Test Explorer. The results table shows
the percentage of the code that was run in each assembly, class, and method. In addition, the source
editor shows you which code has been tested.
Using Code Coverage to Determine How Much Code is being Tested
(https://msdn.microsoft.com/library/dd537628.aspx)
Unit Testing, Code Coverage and Code Clone Analysis with Visual Studio 2015 (Lab)
(http://download.microsoft.com/download/6/2/B/62B60ECE-B9DC-4E8A-A97C-EA261BFB935E
/Docs/Unit%20Testing,%20Code%20Coverage%20and%20Code%20Clone%20Analysis%20with%20Visual%20Studio%202
Customizing Code Coverage Analysis (https://msdn.microsoft.com/library/jj159530.aspx)
Unit testing with any framework using the high performanceTest Explorer
Test Explorer help developers create, manage, and get maximum benefit from unit testing.
Maintain code health with unit testing in Visual Studio Team Services https://www.visualstudio.com/docs/test/developer-testing/developer-testing
4 of 7 02.09.2016 13:26
Get started with Unit Testing (https://msdn.microsoft.com/library/dd264975.aspx)
Run unit tests with Test Explorer (https://msdn.microsoft.com/library/hh270865.aspx)
Writing Unit tests for C/C++ (https://msdn.microsoft.com/library/hh598953.aspx)
Install third-party unit test frameworks (https://msdn.microsoft.com/library/hh598952.aspx)
Visual Studio is also extensible and opens the door for third-party unit testing adapters such as NUnit
and xUnit.net. In addition, the code clone capability goes hand-in-hand with delivering high quality
software by helping you identify blocks of semantically similar code which may be candidates for
common bug fixes or refactoring.
Maintain code health with unit testing in Visual Studio Team Services https://www.visualstudio.com/docs/test/developer-testing/developer-testing
5 of 7 02.09.2016 13:26
Also see
Get started with developer testing tools (getting-started/getting-started-with-developer-
testing)
Speeding up Unit Test Execution in Team Foundation Server (http://blogs.msdn.com
/b/visualstudioalm/archive/2015/07/30/speeding-up-test-execution-in-tfs.aspx)
Parallel and Context Sensitive Unit Test Execution (https://blogs.msdn.microsoft.com
/visualstudioalm/2016/02/08/parallel-and-context-sensitive-test-execution-with-visual-studio-
2015-update-1/)
Unit Testing, Code Coverage and Code Clone Analysis with Visual Studio 2015 (Lab)
(http://download.microsoft.com/download/6/2/B/62B60ECE-B9DC-4E8A-A97C-EA261BFB935E
/Docs/Unit%20Testing,%20Code%20Coverage%20and%20Code%20Clone%20Analysis%20with%20Visual%20Studio%202
Maintain code health with unit testing in Visual Studio Team Services https://www.visualstudio.com/docs/test/developer-testing/developer-testing
6 of 7 02.09.2016 13:26
Help and support
Submit bugs through Connect (https://connect.microsoft.com/visualstudio), make suggestions on
Uservoice (http://visualstudio.uservoice.com/forums/121579-visual-studio), and send quick thoughts
using the Send-a-Smile link in the Visual Studio, Team Services, or TFS title bar. We look forward to
your feedback.
© 2016 Microsoft
Maintain code health with unit testing in Visual Studio Team Services https://www.visualstudio.com/docs/test/developer-testing/developer-testing
7 of 7 02.09.2016 13:26
Exploratory test web apps directly in your browserLast Updated: 8/12/2016
IN THIS ARTICLE
Team Services | TFS 2015
The Exploratory Testing extension lets everyone in the team, such as developers, testers, product owners, leads, and managers, contribute to thequality of the product.
What is the Exploratory Testing extension?
The Exploratory Testing extension is a simple browser‐based extension you can use to test web apps anytime and anywhere, and is simple enoughfor everyone in the team to use. It helps to improve productivity by allowing you to spend more time finding issues, and less time filing them.
Using the extension is a simple, three step process.
ShareTeam Services > Test > Exploratory test your web app in Visual…
Table of contents
Testing scenarios
Manual and exploratory testing
Overview
Get started with manual testing
Advanced manual testingtechniques
Manual testing with MicrosoftTest Manager
Get started with exploratorytesting
Install the Exploratory Testingextension
Test in Connected mode
Test in Standalone mode
Advanced exploratory testingtechniques
Exploratory testing with MicrosoftTest Manager
Performance testing
Continuous testing
Developer testing tools
Test lab management
API reference for test tools
Free Visual Studio
Page 1 of 4 Assembled by RunPDF.com
Capture your findings quickly and easily using the tools in the extension. The Exploratory Testing extension helps capture a large volume and widevariety of information that clearly highlights issues, and includes rich diagnostic data that can serve as a starting point for debugging. In Connectedmode, it can capture screenshots, notes, and screen recordings; which can be annotated. The extension automatically records user actions as animage action log; system information about the browser, operating system, memory, and more; and page load data including resource usage andnavigation timelines. In Standalone mode, the extension can capture screenshots and inline annotations.
Create work items such as bugs, tasks, and test cases directly from the Exploratory Testing extension. The captured findings automatically become apart of the work item. Users can file a bug to report an issue with the product, or create a task that indicates a new work requirement. The extensioncan also be used to create test cases for scenarios discovered during exploration.
Collaborate with your team by sharing your findings. In Connected mode, the powerful integration with team systems maximizes traceability, andsimplifies bug or task tracking and management. In Standalone mode you can export your session report and share it with the rest of the team.
Install the Exploratory Testing extension
1. Download and install Chrome, then open it, if you haven't already.
2. Go to Visual Studio Marketplace > Exploratory Testing and install the extension.
3. Add the Exploratory Testing extension to Chrome.
Select an exploratory testing mode
1. Go to Chrome and open the extension.
Page 2 of 4 Assembled by RunPDF.com
2. Decide if you want to use the extension in Connected or Standalone mode.
Connected mode is a full functional mode that supports Visual Studio Team Services and Team Foundation Server 2015 and above. In thismode, users can quickly capture all their issues using annotated screenshots, notes, and screen recordings; create bugs, tasks, and test casesdirectly; and collaborate with the rest of the team using the integrated experience to provide traceability, avoid duplication, update existingwork items, and get insights.
Standalone mode is a self‐contained mode with no dependency on Visual Studio Team Services or Team Foundation Server. Any team, largeor small, can use this mode to capture issues using screenshots with inline annotations and notes, then share the results using a session reportand manually create work items based on the results.
Try this next
Use the Exploratory Testing extension in Connected modeUse the Exploratory Testing extension in Standalone mode
Q&A
Q: Does the extension support other browsers?
A: The Exploratory Testing extension is currently available only for Chrome. Edge and Firefox support is planned.
Q: Does the extension support Team Foundation Server?
A: See extension details here.
Help and support
Submit bugs through Connect, make suggestions on Uservoice, and send quick thoughts using the Send‐a‐Smile link in the Visual Studio, TeamServices, or TFS title bar. We look forward to your feedback.
Visual StudioMy Visual Studio
Manage Visual Studio
Marketplace
Integrate
Related SitesVisual Studio documentation
MSDN Home
Channel 9
Azure
Visual Studio Blog
ProductsVisual Studio
SupportGet Support
Page 3 of 4 Assembled by RunPDF.com
Visual Studio Team Services
Visual Studio Code
Download
Compare
How to Buy
Submit a Bug
Submit an Idea
Forums
United States ﴾English﴿ Contact us Jobs Privacy Terms of use Trademarks
© 2016 Microsoft
Is this page helpful?
YES NO
Page 4 of 4 Assembled by RunPDF.com
| Marketplace
Visual Studio Team Services > Test > Exploratory Testing ‐ Capture, Create & Collaborate
Exploratory Testing ‐ Capture, Create & CollaborateMicrosoft | 14250 installs | ﴾27﴿
Exploratory Testing for everyone
Install
Categories
Test
Tags
adhoc testing exploratory testing manual testing
test execution
Works withVisual Studio Team Services
ResourcesSupportGet StartedLicense
More InfoVersion 1.0.105.1
Last updated Friday, September 02, 2016 06:53:31
Publisher Microsoft
Report Report Abuse
Sign in
Exploratory Testing now in 3 easy steps – capture, create & collaborate.
Everyone in the team, be it developers, testers, product owners, UX, leads/managers etc., can perform exploratorytesting of web‐apps, right from the Chrome browser on any platform ﴾Windows, Mac or Linux﴿. Support for Edgebrowser coming soon.
Capture
Use various capture formats‐ notes, screenshots with annotations, image action logs ﴾user actions﴿, and screenrecordings. Test your applications on real devices using cloud providers like Perfecto, or test them on browser basedemulators.
Create
Quick creation of bugs, tasks and test cases with all the captured information automatically attached for you.
Page 1 of 3 Assembled by RunPDF.com
Collaborate
Export your session report in the standalone mode and share your findings with the rest of the team. Additionallyconnect to your account on Visual Studio Team Services / Team Foundation Server ﴾2015 RTM or above﴿ to leveragean integrated experience spanning E2E traceability, avoiding duplicate bugs, simplified tracking & triaging of bugs/tasksand gathering rich insights across exploratory testing sessions.
Features and Pricing
Free during Preview*
Only Visual Studio Team Services users ﴾not stakeholders﴿ can be assigned the Exploratory Testing extension to beused in Connected ﴾Team﴿ mode. Learn more about Visual Studio Team Services user pricing.
After the Preview period is over, the Exploratory Testing extension will be included with the Test Manager extensionand with these Visual Studio subscriptions:
Visual Studio Enterprise ‐ monthly
Page 2 of 3 Assembled by RunPDF.com
© 2016 Microsoft
Write a review
See more
User Reviews
﴾27﴿
Fredrik Almén 8/24/2016
Love the concept! Really nice work!Have some questions,1. I cant find the export button under session timeline, and i cant find any sessions i have done either, I get forwardedto tfs but dont see ... Read More
Microsoft8/24/2016
Thanks Fredrik for appreciation!
About your questions ‐ ﴾1﴿ ‐ Export button is visible in Standalone mode ﴾From Connection page﴿, which is not connected toTFS/VSTS. It is for easy access of all s... Read More
Nimesh Sankhalpara 8/12/2016
Great little product.. Any chances of adding this extension to IE??
Microsoft8/17/2016
Thanks Nimesh for the appreciation! We do have plans to get this extension on Edge and Firefox, howeverwe have no plans around getting it on IE.
Valdir Viana 8/9/2016
Awesome!!
Carly Rimmer 8/8/2016
Can you advise where this tool saves the session reports? there is no indication as to where this is?
Microsoft8/9/2016
You can view your completed sessions using the Exploratory Testing ‐ Insights page. There are 2 ways to get to the insights page:a. From within the extension, click on “View completed exploratory s... Read More
sphereqube13 7/28/2016
We are trying to use Exploratory Testing chrome Plugin. We trying to connect our project TFS URL with Exploratory, weare not getting our team project into Exploratory. We can see its just connecting... Read More
Microsoft8/3/2016
Hi
Thanks for reporting this issue. What is the TFS version you are on? We support TFS 2015 onwards. In caseit is TFS 2015+, please write to vsoxt [at] microsoft [dot] com, and we will take this d... Read More
Contact us Jobs Privacy Terms of use Trademarks
Visual Studio Enterprise ‐ annualVisual Studio Enterprise with MSDNMSDN PlatformsVisual Studio Test Professional with MSDN
Page 3 of 3 Assembled by RunPDF.com
How to: Configure Unit Tests to Target AnEarlier Version of the .NET Framework
When you create a test project in Microsoft Visual Studio, the most recent version of the .NET Framework is set as the target,
by default. Additionally, if you upgrade test projects from previous versions of Visual Studio, they are upgraded to target
the most recent version of the .NET Framework. By editing the project properties, you can explicitly re-target the project to
earlier versions of the .NET Framework.
You can create unit test projects that target specific versions of the .NET Framework. The targeted version must be 3.5 or
later, and cannot be a client version. Visual Studio enables the following basic support for unit tests that target specific
versions:
You can create unit test projects and target them to a specific version of the .NET Framework.
You can run unit tests that target a specific version of the .NET Framework from Visual Studio on your local machine.
You can run unit tests that target a specific version of the .NET Framework by using MSTest.exe from the command
prompt.
You can run unit tests on a build agent as part of a build.
Testing SharePoint Applications
The capabilities listed above also enable you to write unit tests and integration tests for SharePoint applications using Visual
Studio. For more information about how to develop SharePoint applications using Visual Studio, see Create SharePoint
Solutions, Building and Debugging SharePoint Solutions and Verifying and Debugging SharePoint Code.
Limitations
The following limitations apply when you re-target your test projects to use earlier versions of the .NET Framework:
In the .NET Framework 3.5, multitargeting is supported for test projects that contain only unit tests. The .NET
Framework 3.5 does not support any other test type, such as coded UI or load test. The re-targeting is blocked for
test types other than unit tests.
Execution of tests that are targeted at an earlier version of the .NET Framework is supported only in the default host
adapter. It is not supported in the ASP.NET host adapter. ASP.NET applications that have to run in the ASP.NET
Development Server context must be compatible with the current version of the .NET Framework.
Data collection support is disabled when you run tests that support .NET Framework 3.5 multitargeting. You can run
code coverage by using the Visual Studio command-line tools.
Unit tests that use .NET Framework 3.5 cannot run on a remote machine.
You cannot target unit tests to earlier client versions of the framework.
Visual Studio 2015
How to: Configure Unit Tests to Target An Earlier Version of the .NET F... https://msdn.microsoft.com/en-us/library/gg601487(d=printer).aspx
1 of 5 02.09.2016 13:53
Re-targeting to a specific version of the .NET Framework for Visual
Basic Unit Test Projects
Create a new Visual Basic unit test project. On the File menu, choose New and then choose Project.
The New Project dialog box is displayed.
1.
Under Installed Templates, expand Visual Basic. Select Test and then select the Test Project template.2.
In the Name text box, type a name for your Visual Basic test project and then choose OK.3.
In Solution Explorer, choose Properties from the shortcut menu of the new Visual Basic test project.
The properties for your Visual Basic test project are displayed.
4.
On the Compile tab choose Advanced Compile Options as shown in the following illustration.5.
Use the Target framework (all configurations) drop-down list to change the target framework to .NET
Framework 3.5 or a later version as shown in callout B in the following illustration. You should not specify a client
version.
6.
How to: Configure Unit Tests to Target An Earlier Version of the .NET F... https://msdn.microsoft.com/en-us/library/gg601487(d=printer).aspx
2 of 5 02.09.2016 13:53
Re-targeting to a specific version of the .NET Framework for Visual
C# Unit Test Projects
Create a new Visual C# unit-test project. On the File menu, choose New and then choose Project.
The New Project dialog box is displayed.
1.
Under Installed Templates, expand Visual C#. Select Test and then select the Test Project template.2.
In the Name text box, type a name for your Visual C# test project and then choose OK.3.
In Solution Explorer, choose Properties from the shortcut menu of your new Visual C# test project.
The properties for your Visual C# test project are displayed.
4.
On the Application tab choose Target framework and then choose .NET Framework 3.5 or a later version from
the drop-down list to change the target framework.as shown in the following illustration. You should not specify a
client version.
5.
How to: Configure Unit Tests to Target An Earlier Version of the .NET F... https://msdn.microsoft.com/en-us/library/gg601487(d=printer).aspx
3 of 5 02.09.2016 13:53
Re-targeting to a specific version of the .NET Framework for
C++/CLI Unit Test Projects
Create a new C++ unit-test project. On the File menu, select New and then click Project.
The New Project dialog box is displayed.
Warning
To build C++/CLI unit tests for a previous version of the .NET framework for Visual C++, you must use the
corresponding version of Visual Studio. For example, to target the .NET Framework 3.5, you must install Visual
Studio 2008 and Visual Studio 2008 Service Pack 1.
1.
Under Installed Templates, expand Visual C ++. Select Test and then select the Test Project template.2.
In the Name text box, type a name for your Visual C++ test project and then click OK.3.
In Solution Explorer, choose Unload Project from your new Visual C++ test project.4.
In Solution Explorer, choose the unloaded Visual C++ test project and then choose Edit <project name>.vcxproj.
The .vcxproj file opens in the editor.
5.
Set the TargetFrameworkVersion to version 3.5 or a later version in the PropertyGroup labeled "Globals". You
should not specify a client version:
6.
<PropertyGroup Label="Globals">
<TargetName>DefaultTest</TargetName>
<ProjectTypes>{3AC096D0‐A1C2‐E12C‐1390‐A8335801FDAB};{8BC9CEB8‐8B4A‐
11D0‐8D11‐00A0C91BC942}</ProjectTypes>
<ProjectGUID>{CE16D77A‐E364‐4ACD‐948B‐1EB6218B0EA3}</ProjectGUID>
<TargetFrameworkVersion>3.5</TargetFrameworkVersion>
<Keyword>ManagedCProj</Keyword>
How to: Configure Unit Tests to Target An Earlier Version of the .NET F... https://msdn.microsoft.com/en-us/library/gg601487(d=printer).aspx
4 of 5 02.09.2016 13:53
Save and close the .vcxproj file.7.
In Solution Explorer, choose select Reload Project from the shortcut menu of your new Visual C++ test project.8.
See AlsoCreating and Running Unit Tests for Existing Code
Create SharePoint Solutions
Building and Debugging SharePoint Solutions
Advanced Compiler Settings Dialog Box (Visual Basic)
© 2016 Microsoft
<RootNamespace>CPP_Test</RootNamespace>
</PropertyGroup>
How to: Configure Unit Tests to Target An Earlier Version of the .NET F... https://msdn.microsoft.com/en-us/library/gg601487(d=printer).aspx
5 of 5 02.09.2016 13:53
IntelliTest Reference ManualLast Updated: 8/4/2016
IN THIS ARTICLE
| Visual Studio 2015 |
Contents
Overview of IntelliTest
The Hello World of IntelliTestLimitations
NondeterminismConcurrencyNative codePlatformLanguageSymbolic reasoningIncorrect stack traces
Further reading
Get started with IntelliTest
Important attributesImportant static helper classes
Test Generation
Test generatorsParameterized unit testingGeneric parameterized unit testingAllowing exceptionsTesting internal typesAssumptions and assertionsPreconditionPostconditionTest failuresSetup and tear downFurther reading
Input Generation
Constraint solverDynamic code coverageIntegers and floatsObjectsInstantiating existing classesVisibilityParameterized mocksStructsArrays and stringsObtaining additional inputs
ShareTeam Services > Test > IntelliTest Reference Manual | Microsoft…
Table of contents
Testing scenarios
Manual and exploratory testing
Performance testing
Continuous testing
Developer testing tools
Overview
Get started
Create Unit Tests command
Generate tests with IntelliTest
Run tests with Test Explorer
Determine code coverage
System test with Visual Studio
Create system tests with VS
Run system tests with VS
Submit bugs in VS
Submit bugs in TFS
IntelliTest reference manual
Overview
Get started
Index to sections
Test generation
Dynamic symbolic execution
Exploration bounds
Attribute glossary
Settings waterfall
Static helper classes
Warnings and errors
Test lab management
API reference for test tools
Free Visual Studio
Page 1 of 3 Assembled by RunPDF.com
Further reading
Exploration Bounds
MaxConstraintSolverTimeMaxConstraintSolverMemoryMaxBranchesMaxCallsMaxStackMaxConditionsMaxRunsMaxRunsWithoutNewTestsMaxRunsWithUniquePathsMaxExceptionsTestExcludePathBoundsExceededTestEmissionFilterTestEmissionBranchHits
Attribute Glossary
PexAssumeNotNullPexClassPexGenericArgumentsPexMethodPexExplorationAttributeBasePexAssemblySettingsPexAssemblyUnderTestPexInstrumentAssemblyAttributePexUseTypePexAllowedExceptionPexAllowedExceptionFromAssemblyPexAllowedExceptionFromTypePexAllowedExceptionFromTypeUnderTest
Settings Waterfall
Static Helper Classes
PexAssumePexAssertPexChoosePexObservePexSymbolicValue
Warnings and Errors
MaxBranches exceededMaxConstraintSolverTime exceededMaxConditions exceededMaxCalls exceededMaxStack exceededMaxRuns exceededMaxRunsWithoutNewTests exceededCannot concretize solutionNeed help to construct objectNeed help to find typesUsable type guessedUnexpected failure during explorationTargetInvocationExceptionUninstrumented method calledExternal method calledUninstrumentable method called
Page 2 of 3 Assembled by RunPDF.com
Testability issueLimitationObserved call mismatchValue stored in static field
Got feedback?
Post your ideas and feature requests on UserVoice.
Tell us what you think with Send‐a‐Smile on the title bar.
Visual StudioMy Visual Studio
Manage Visual Studio
Marketplace
Integrate
Related SitesVisual Studio documentation
MSDN Home
Channel 9
Azure
Visual Studio Blog
ProductsVisual Studio
Visual Studio Team Services
Visual Studio Code
Download
Compare
How to Buy
SupportGet Support
Submit a Bug
Submit an Idea
Forums
United States ﴾English﴿ Contact us Jobs Privacy Terms of use Trademarks
© 2016 Microsoft
Is this page helpful?
YES NO
Page 3 of 3 Assembled by RunPDF.com
Hands-On LabIntroduction to Coded UI Tests with Visual Studio Enterprise 2015
Lab version: 14.0.23107.0
Last updated: 9/11/2015
TABLE OF CONTENT
INTRODUCTION TO CODED UI TESTS WITH VISUAL STUDIO ENTERPRISE 2015 ...................................................... 1
........................................................................................................................................................................... 1
OVERVIEW .......................................................................................................................................................... 4
CODED UI TESTS PROVIDE A WAY TO CREATE FULLY AUTOMATED TESTS TO VALIDATE THE FUNCTIONALITY AND BEHAVIOR OF YOUR APPLICATION’S USER INTERFACE. IN THIS LAB, YOU WILL GAIN A BASIC UNDERSTANDING OF CODED UI TESTS BY CREATING A NEW TEST AND ADDING VALIDATION LOGIC TO IT. ............................................. 4
PREREQUISITES ................................................................................................................................................... 4
IN ORDER TO COMPLETE THIS LAB YOU WILL NEED THE VISUAL STUDIO 2015 VIRTUAL MACHINE PROVIDED BY MICROSOFT. FOR MORE INFORMATION ON ACQUIRING AND USING THIS VIRTUAL MACHINE, PLEASE SEE THIS BLOG POST. ......................................................................................................................................................... 4
EXERCISES ........................................................................................................................................................... 4
THIS HANDS-ON LAB INCLUDES THE FOLLOWING EXERCISES: ............................................................................... 4
INTRODUCTION TO CODE GENERATION FROM ACTION RECORDINGS ................................................................... 4
INTRODUCTION TO CODE GENERATION USING CODED UI TEST BUILDER .............................................................. 4
DATA DRIVEN DEMONSTRATION FOR CODED UI TEST ........................................................................................... 4
ESTIMATED TIME TO COMPLETE THIS LAB: 60 MINUTES. ...................................................................................... 4
EXERCISE 1: CODE GENERATION FROM ACTION RECORDINGS .............................................................................. 5
LOG IN AS JULIA (VSALM\JULIA). ALL USER PASSWORDS ARE P2SSW0RD. ............................................................ 5
LAUNCH VISUAL STUDIO 2015 FROM THE TASKBAR AND OPEN TEAM EXPLORER. ................................................ 5
SELECT THE CONNECT TO TEAM PROJECTS BUTTON. ............................................................................................ 5
........................................................................................................................................................................... 5
FIGURE ............................................................................................................................................................... 5
CONNECTING TO A DIFFERENT TEAM PROJECT ..................................................................................................... 5
IN TEAM EXPLORER – CONNECT, DOUBLE-CLICK ON THE TAILSPIN TOYS PROJECT. ................................................ 5
........................................................................................................................................................................... 5
FIGURE ............................................................................................................................................................... 5
LOADING THE TAILSPIN TOYS PROJECT ................................................................................................................. 5
START A NEW TESTING PROJECT (FILE | NEW | PROJECT…). .................................................................................. 5
IN THE NEW PROJECT WINDOW, SELECT THE CODED UI TEST PROJECT TEMPLATE FROM VISUAL C# | TEST, THEN SELECT THE OK BUTTON TO CREATE THE TEST PROJECT. ....................................................................................... 5
........................................................................................................................................................................... 5
FIGURE ............................................................................................................................................................... 5
CREATING A CODED UI TEST PROJECT .................................................................................................................. 5
THERE ARE TWO WAYS TO GENERATE CODE FOR THIS NEW CODED UI TEST. THE FIRST AND DEFAULT OPTION IS TO USE THE CODED UI TEST BUILDER, WHICH ALLOWS YOU TO GENERATE TEST CODE BY MANUALLY WALKING THROUGH A TEST SCENARIO. THE SECOND OPTION IS TO USE AN EXISTING ACTION RECORDING. SELECT THE SECOND OPTION TO USE AN EXISTING ACTION RECORDING AND SELECT THE OK BUTTON TO CONTINUE. ............ 5
........................................................................................................................................................................... 5
FIGURE ............................................................................................................................................................... 5
USING AN EXISTING ACTION RECORDING FOR TEST GENERATION ........................................................................ 5
SELECT THE IDS RADIO BUTTON AND ENTER 41. FOR THE PURPOSES OF THIS LAB, ASSUME THAT WE ALREADY KNOW THAT THIS IS THE ID OF A TEST CASE WITH AN ACTION RECORDING. ......................................................... 6
SELECT THE FIND BUTTON TO EXECUTE THE WORK ITEM QUERY. ......................................................................... 6
........................................................................................................................................................................... 6
FIGURE ............................................................................................................................................................... 6
FINDING A TEST CASE WITH ACTION RECORDING ................................................................................................. 6
SELECT THE OK BUTTON TO GENERATE A CODED UI TEST FROM THE ACTION RECORDING. ................................... 6
........................................................................................................................................................................... 6
FIGURE ............................................................................................................................................................... 6
SELECTING A TEST CASE WITH ACTION RECORDING ............................................................................................. 6
NAVIGATE TO THE CODEDUITESTMETHOD1 METHOD IN THE GENERATED CODEDUITEST1.CS FILE. EACH LINE REPRESENTS A STEP FROM THE ACTION RECORDING USED DURING TEST GENERATION. ....................................... 6
PLACE THE CURSOR SOMEWHERE ON THE LINE OF THE FIRST METHOD CALL, OPENHTTPWWWTAILSPINTOYSCOM(), AND THEN PRESS ALT + F12 TO PEEK AT ITS DEFINITION. THIS WILL LOAD THE UIMAP CLASS FROM THE UIMAP.DESIGNER.CS FILE, WHICH CONTAINS THE GENERATED TEST LOGIC. THIS GENERATED METHOD LAUNCHES INTERNET EXPLORER AND NAVIGATES TO A SPECIFIED URL. .............................. 6
........................................................................................................................................................................... 6
FIGURE ............................................................................................................................................................... 6
USING PEEK DEFINITION ...................................................................................................................................... 6
IN THE PEEK DEFINITION WINDOW, SCROLL DOWN TO THE CLICKFOURTHCOFFEEFLYER METHOD. THIS GENERATED METHOD TESTS CLICKING ON A “FOURTH COFFEE FLYER” HYPERLINK THAT IS IN THE TAILSPIN TOYS WEB APPLICATION. ............................................................................................................................................. 6
........................................................................................................................................................................... 6
FIGURE .............................................................................................................................................................. 6
GENERATED TEST METHOD EXAMPLE .................................................................................................................. 6
THE CLICKFOURTHCOFFEEFLYER TEST METHOD DOES NOT SPECIFY THE HYPERLINK PARAMETERS DIRECTLY, BUT INSTEAD REFERS TO THE
“UIBLANKPAGEWINDOWSINTEWINDOW.UIHOMETAILSPINTOYSDOCUMENT1.UIFOURTHCOFFEEFLYERHYPERLINK” PROPERTY. SELECT THE “UIFOURTHCOFFEEFLYERHYPERLINK” PROPERTY (MAY NEED TO SCROLL TO THE RIGHT) AND THEN PRESS ALT + F12 TO VIEW ITS DEFINITION. .......................................................................................... 6
........................................................................................................................................................................... 6
FIGURE ............................................................................................................................................................... 6
DEFINITION OF HYPERLINK PROPERTY ................................................................................................................. 6
NOTE: THE HTMLHYPERLINK INSTANCE THAT IS CREATED FOR THE UIFOURTHCOFFEEFLYERHYPERLINK PROPERTY HAS A NUMBER OF SEARCH AND FILTER PROPERTIES APPLIED THAT AID THE TEST FRAMEWORK IN LOCATING THECORRECT HTML HYPERLINK. IN THE EVENT THAT THE WEB APPLICATION CHANGES SOME OF THE LINK PROPERTIES, SUCH AS THE INNER TEXT, THE TEST HARNESS MAY STILL BE ABLE TO FIND THE HYPERLINK USING THE REMAINING SEARCH PROPERTIES. ................................................................................................................ 7
PRESS ESCAPE TO CLOSE THE CODE PEEK WINDOW AND RETURN TO THE CODEDUITEST1.CS FILE. ....................... 7
RIGHT-CLICK SOMEWHERE WITHIN THE CODEDUITEST1.CS SOURCE FILE AND SELECT RUN TESTS. DO NOT TOUCH THE MOUSE OR KEYBOARD DURING THE TESTS. .................................................................................................. 7
........................................................................................................................................................................... 7
FIGURE ............................................................................................................................................................... 7
LOCATION OF RUN TESTS COMMAND .................................................................................................................. 7
AS THE TESTS RUN, AN INSTANCE OF INTERNET EXPLORER WILL BE OPENED AND ACTIONS AUTOMATICALLY TAKEN AS THEY ARE DEFINED IN THE CODED UI TEST. THE TEST RUNS MORE THAN ONCE BECAUSE THE ORIGINAL MANUAL TEST THAT THIS CODED UI TEST WAS GENERATED FROM HAD MULTIPLE ROWS OF TEST PARAMETERS. . 7
........................................................................................................................................................................... 7
FIGURE .............................................................................................................................................................. 7
EXAMPLE OF CODED UI TEST RUNNING ............................................................................................................... 7
VERIFY THAT THE TEST PASSED BY VIEWING THE TEST EXPLORER WINDOW. IN THIS CASE, HOWEVER, WE ARE NOT PERFORMING ANY VALIDATION AFTER ANY OF THE STEPS. ........................................................................... 7
........................................................................................................................................................................... 7
FIGURE ............................................................................................................................................................... 7
TEST EXPLORER WINDOW SHOWING PASSED TEST .............................................................................................. 7
EXERCISE 2: CODE GENERATION USING CODED UI TEST BUILDER .......................................................................... 7
IN THIS EXERCISE, YOU WILL LEARN HOW TO USE THE CODED UI TEST BUILDER TO GENERATE TEST CODE FOR THETAILSPIN TOYS WEB APPLICATION AND MODIFY THE GENERATED CODE IN ORDER TO ENABLE DATA DRIVEN TESTING. ............................................................................................................................................................. 7
OPEN INTERNET EXPLORER AND SELECT THE TAILSPIN TOYS BUTTON FROM THE FAVORITES BAR. ........................ 7
SELECT THE MODEL AIRPLANES LINK. .................................................................................................................. 7
SELECT THE FOURTH COFFEE FLYER LINK. ............................................................................................................. 7
SELECT THE ADD TO CART LINK TO LOAD THE SHOPPING CART. ............................................................................ 7
RETURN TO VISUAL STUDIO, LOCATE THE CODEDUITESTMETHOD1 METHOD IN THE CODEDUITEST1.CS FILE, AND ADD A BLANK LINE AFTER THE CALL TO THE “THIS.UIMAP.CLICKONWHITESPACEINWEBSITE” METHOD. ................ 8
........................................................................................................................................................................... 8
FIGURE ............................................................................................................................................................... 8
ADDING BLANK LINE TO TEST SOURCE ................................................................................................................. 8
RIGHT-CLICK AT THE LOCATION OF THE BLANK LINE AND SELECT GENERATE CODE FOR CODED UI TEST | USE CODED UI TEST BUILDER… FROM THE CONTEXT MENU. THIS WILL LOAD THE CODED UI TEST BUILDER WINDOW (WHICH IS ALWAYS DISPLAYED OVER OTHER WINDOWS) AND THE INTERNET EXPLORER INSTANCE THAT WE PREVIOUSLY LEFT OPEN. ...................................................................................................................................... 8
........................................................................................................................................................................... 8
FIGURE ............................................................................................................................................................... 8
STARTING THE CODED UI TEST BUILDER ............................................................................................................... 8
NOTE: THE CODED UI TEST BUILDER IS USED TO RECORD ACTIONS AND ASSERTIONS WITHIN THE USER INTERFACE WHICH ARE THEN CONVERTED TO CODE. ........................................................................................... 8
NOW WE WILL ADD AN ASSERTION TO VERIFY THAT THE QUANTITY TEXTBOX IS EQUAL TO A VALUE OF 1. DRAG AND DROP THE TARGET ICON FROM THE CODED UI TEST BUILDER TOOL WINDOW ONTO THE QUANTITY TEXTBOX IN INTERNET EXPLORER. THIS ACTION WILL LOAD THE CODED UI TEST BUILDER WINDOW. ................... 8
........................................................................................................................................................................... 8
FIGURE ............................................................................................................................................................... 8
SELECTING AN ELEMENT TO USE FOR ASSERTION ................................................................................................ 8
IN THE CODED UI TEST BUILDER WINDOW, SELECT THE TEXT PROPERTY AND CLICK ON ADD ASSERTION. THIS WILL LOAD A DIALOG TO FINALIZE THE ASSERTION OPTIONS TO USE. .................................................................. 8
........................................................................................................................................................................... 8
FIGURE ............................................................................................................................................................... 8
CODED UI TEST BUILDER WINDOW ...................................................................................................................... 8
USE THE AREEQUAL COMPARATOR AND A COMPARISON VALUE OF ‘1’. SELECT THE OK BUTTON TO CONTINUE. ... 8
........................................................................................................................................................................... 8
FIGURE ............................................................................................................................................................... 8
SELECTING THE COMPARATOR TYPE AND VALUE .................................................................................................. 8
VERIFY THAT A CHECKBOX HAS BEEN ADDED TO THE TEXT PROPERTY ROW. ........................................................ 9
........................................................................................................................................................................... 9
FIGURE ............................................................................................................................................................... 9
TEXT PROPERTY SHOWING ASSERTION CHECKBOX .............................................................................................. 9
SELECT THE GENERATE CODE BUTTON FROM THE CODED UI TEST BUILDER TOOL WINDOW. ................................ 9
........................................................................................................................................................................... 9
FIGURE ............................................................................................................................................................... 9
GENERATE CODE BUTTON LOCATION ................................................................................................................... 9
IN THE GENERATE CODE WINDOW, USE QUANTITYEQUALSONE FOR THE METHOD NAME AND SELECT THE ADD AND GENERATE BUTTON TO GENERATE THE VALIDATION CODE. .......................................................................... 9
........................................................................................................................................................................... 9
FIGURE ............................................................................................................................................................... 9
GENERATING ASSERTION CODE ........................................................................................................................... 9
REMOVE THE “FOURTH COFFEE FLYER” ITEM FROM THE SHOPPING CART. ........................................................... 9
CLOSE THE CODED UI TEST BUILDER. .................................................................................................................... 9
CLOSE THE INTERNET EXPLORER WINDOW AND RETURN TO VISUAL STUDIO. ...................................................... 9
NOTE THAT THE ASSERTION CODE GENERATION HAS ADDED THE NEW VALIDATION STEP. .................................... 9
........................................................................................................................................................................... 9
FIGURE ............................................................................................................................................................... 9
NEWLY CREATED ASSERTION STEP FOR CODED UI TEST ........................................................................................ 9
RIGHT-CLICK AND SELECT RUN TESTS TO RUN THE TESTS WITH THE NEW VALIDATION STEPS. THE TEST SHOULD COMPLETE SUCCESSFULLY. ................................................................................................................................... 9
EXERCISE 3: DATA DRIVEN DEMONSTRATION FOR CODED UI TEST ...................................................................... 10
IN THIS EXERCISE, YOU WILL ADD ANOTHER SET OF TEST PARAMETER VALUES TO THE TEST CASE IN ORDER TO DEMONSTRATE THAT THESE TEST PARAMETERS ARE HOOKED UP TO THE CODED UI TEST AND THAT THE VALIDATION THAT WE RECENTLY ADDED IN IS PERFORMING AS EXPECTED. ........................................................ 10
LAUNCH MICROSOFT TEST MANAGER FROM THE TASKBAR. ............................................................................... 10
IF NOT ALREADY CONNECTED, CONNECT TO THE ITERATION 2 TEST PLAN FROM THE TAILSPIN TOYS TEAM PROJECT. YOU CAN CHANGE THIS SELECTION BY CLICKING ON THE HOME BUTTON. ........................................... 10
......................................................................................................................................................................... 10
FIGURE ............................................................................................................................................................. 10
CONNECTING TO TAILSPIN TOYS TEAM PROJECT ................................................................................................ 10
......................................................................................................................................................................... 10
FIGURE ............................................................................................................................................................. 10
CONNECTING TO ITERATION 2 TEST PLAN .......................................................................................................... 10
......................................................................................................................................................................... 10
FIGURE ............................................................................................................................................................. 10
LOCATION OF CURRENTLY SELECTED TEAM PROJECT AND TEST PLAN ................................................................. 10
SELECT THE TEST TAB AND SELECT TEST SUITE 7 NAMED “AS A CUSTOMER I SHOULD BE ABLE TO REMOVE ITEMS FROM MY SHOPPING CART”. ............................................................................................................................. 10
......................................................................................................................................................................... 10
FIGURE ............................................................................................................................................................. 10
TEST SUITE 7 HAS BEEN SELECTED ...................................................................................................................... 10
SELECT THE FIRST TEST CASE WITH ID = 41 AND SELECT THE OPEN TEST CASE BUTTON. ...................................... 10
......................................................................................................................................................................... 10
FIGURE ............................................................................................................................................................. 10
OPENING TEST CASE WITH ID=41 ....................................................................................................................... 11
IN THE PARAMETER VALUES SECTION AT THE BOTTOM, ADD A NEW ROW WITH QUANTITY 10. ......................... 11
NOTE: A VALUE OF 10 IS A LEGITIMATE VALUE FOR THE SHOPPING CART, SO THE CART WILL REFRESH TO SHOW AQUANTITY OF 10 WHEN THIS VALUE IS ENTERED. HOWEVER, SINCE THE PURPOSE OF THIS EXERCISE IS TO SHOW WHAT HAPPENS WHEN A TEST ITERATION FAILS, WE WILL PRETEND THAT THIS VALUE CAUSES THE TEST TO FAIL IN ORDER TO DEMONSTRATE A TEST CASE FAILURE. THE ASSERTION THAT WE ADDED EXPECTS THAT THE QUANTITY WILL REMAIN AT 1. ........................................................................................................................... 11
......................................................................................................................................................................... 11
FIGURE ............................................................................................................................................................. 11
PARAMETER VALUE CHANGE ............................................................................................................................. 11
SAVE THE CHANGES TO THE TEST CASE AND RETURN TO VISUAL STUDIO. .......................................................... 11
RUN THE TESTS AGAIN AND NOTE THAT THE TEST FAILS ON THE FOURTH ITERATION. ........................................ 11
SELECT THE FAILED TEST WITHIN THE TEST EXPLORER WINDOW TO SEE THE DETAILS. AT THE BOTTOM OF THE TEST EXPLORER WINDOW WE ARE NOTIFIED THAT 3 OUT OF 4 TESTS PASSED, WITH THE FOURTH DATA ROW FAILING. ............................................................................................................................................................ 11
......................................................................................................................................................................... 11
FIGURE ............................................................................................................................................................. 11
TEST RESULTS SHOWING FAILED ASSERTION ...................................................................................................... 11
Overview
Coded UI tests provide a way to create fully automated tests to validate the functionality and behavior ofyour application’s user interface. In this lab, you will gain a basic understanding of coded UI tests by creating a new test and adding validation logic to it.
Prerequisites
In order to complete this lab you will need the Visual Studio 2015 virtual machine provided by Microsoft.For more information on acquiring and using this virtual machine, please see this blog post.
Exercises
This hands-on lab includes the following exercises:
Introduction to Code Generation from Action Recordings
Introduction to Code Generation using Coded UI Test Builder
Data Driven Demonstration for Coded UI Test
Estimated time to complete this lab: 60 minutes.
Exercise 1: Code Generation from ActionRecordings
In this exercise, you will be introduced to the code generation features in Visual Studio that allow testers to quickly and easily create coded UI tests directly from existing action recordings. Action recordings contain the steps taken during manual testing of an application. To learn more about manual testing and action recordings, please see the “Authoring and Running Manual Tests using Microsoft Test Manager 2015” lab.
Log in as Julia (VSALM\Julia). All user passwords are P2ssw0rd.
Launch Visual Studio 2015 from the taskbar and open Team Explorer.
Select the Connect to Team Projects button.
Figure 1Connecting to a different team project
In Team Explorer – Connect, double-click on the Tailspin Toys project.
Figure 2Loading the Tailspin Toys project
Start a new testing project (File | New | Project…).
In the New Project window, select the Coded UI Test Project template from Visual C# | Test, then select the OK button to create the test project.
Figure 3Creating a Coded UI Test Project
There are two ways to generate code for this new coded UI test. The first and default option is to use the Coded UI Test Builder, which allows you to generate test code by manually walking through a test scenario. The second option is to use an existing action recording. Select the second option to use an existing action recording and select the OK button to continue.
Figure 4Using an existing action recording for test generation
Select the IDs radio button and enter 41. For the purposes of this lab, assume that we already know that this is the ID of a test case with an action recording.
Select the Find button to execute the work item query.
Figure 5Finding a test case with action recording
Select the OK button to generate a coded UI test from the action recording.
Figure 6Selecting a test case with action recording
Navigate to the CodedUITestMethod1 method in the generated CodedUITest1.cs file. Each line represents a step from the action recording used during test generation.
Place the cursor somewhere on the line of the first method call, Openhttpwwwtailspintoyscom(), and then press Alt + F12 to peek at its definition. This will load the UIMap class from the UIMap.Designer.cs file, which contains the generated test logic. This generated method launches Internet Explorer and navigates to a specified URL.
Figure 7Using Peek Definition
In the peek definition window, scroll down to the ClickFourthCoffeeFlyer method. This generated method tests clicking on a “Fourth Coffee Flyer” hyperlink that is in the Tailspin Toys web application.
Figure 8 Generated test method example
The ClickFourthCoffeeFlyer test method does not specify the hyperlink parameters directly, but instead refers to the “UIBlankPageWindowsInteWindow.UIHomeTailspinToysDocument1.UIFourthCoffeeFlyerHyperlink” property. Select the “UIFourthCoffeeFlyerHyperlink” property (may need to scroll to the right) and then press Alt + F12 to view its definition.
Figure 9Definition of hyperlink property
Note: The HtmlHyperlink instance that is created for the UIFourthCoffeeFlyerHyperlink property has a number of search and filter properties applied that aid the test framework in locating the correct HTML hyperlink. In the event that the web application changes some of the link properties, such as the inner text, the test harness may still be able to find the hyperlink using the remaining search properties.
Press Escape to close the code peek window and return to the CodedUITest1.cs file.
Right-click somewhere within the CodedUITest1.cs source file and select Run Tests. Do not touch the mouse or keyboard during the tests.
Figure 10Location of Run Tests command
As the tests run, an instance of Internet Explorer will be opened and actions automatically taken as they are defined in the coded UI test. The test runs more than once because the original manualtest that this coded UI test was generated from had multiple rows of test parameters.
Figure 11 Example of coded UI test running
Verify that the test passed by viewing the Test Explorer window. In this case, however, we are not performing any validation after any of the steps.
Figure 12Test Explorer window showing passed test
Exercise 2: Code Generation using Coded UI Test Builder
In this exercise, you will learn how to use the Coded UI Test Builder to generate test code for the Tailspin Toys Web application and modify the generated code in order to enable data driven testing.
Open Internet Explorer and select the Tailspin Toys button from the favorites bar.
Select the Model Airplanes link.
Select the Fourth Coffee Flyer link.
Select the Add To Cart link to load the shopping cart.
Return to Visual Studio, locate the CodedUITestMethod1 method in the CodedUITest1.cs file, and add a blank line after the call to the “this.UIMap.Clickonwhitespaceinwebsite” method.
Figure 13Adding blank line to test source
Right-click at the location of the blank line and select Generate Code for Coded UI Test | Use CodedUI Test Builder… from the context menu. This will load the Coded UI Test Builder window (which is always displayed over other windows) and the Internet Explorer instance that we previously left open.
Figure 14Starting the Coded UI Test Builder
Note: The Coded UI Test Builder is used to record actions and assertions within the user interface which are then converted to code.
Now we will add an assertion to verify that the Quantity textbox is equal to a value of 1. Drag and drop the target icon from the Coded UI Test Builder tool window onto the Quantity textbox in Internet Explorer. This action will load the Coded UI Test Builder window.
Figure 15Selecting an element to use for assertion
In the Coded UI Test Builder window, select the Text property and click on Add Assertion. This will load a dialog to finalize the assertion options to use.
Figure 16Coded UI Test Builder window
Use the AreEqual comparator and a comparison value of ‘1’. Select the OK button to continue.
Figure 17Selecting the comparator type and value
Verify that a checkbox has been added to the Text property row.
Figure 18Text property showing assertion checkbox
Select the Generate Code button from the Coded UI Test Builder tool window.
Figure 19Generate Code button location
In the Generate Code window, use QuantityEqualsOne for the Method Name and select the Add and Generate button to generate the validation code.
Figure 20Generating assertion code
Remove the “Fourth Coffee Flyer” item from the shopping cart.
Close the Coded UI Test Builder.
Close the Internet Explorer window and return to Visual Studio.
Note that the assertion code generation has added the new validation step.
Figure 21Newly created assertion step for coded UI test
Right-click and select Run Tests to run the tests with the new validation steps. The test should complete successfully.
Exercise 3: Data Driven Demonstration for Coded UI Test
In this exercise, you will add another set of test parameter values to the test case in order to demonstrate that these test parameters are hooked up to the coded UI test and that the validation that we recently added in is performing as expected.
Launch Microsoft Test Manager from the taskbar.
If not already connected, connect to the Iteration 2 test plan from the Tailspin Toys team project. You can change this selection by clicking on the Home button.
Figure 22
Connecting to Tailspin Toys team project
Figure 23Connecting to Iteration 2 test plan
Figure 24Location of currently selected team project and test plan
Select the Test tab and select test suite 7 named “As a customer I should be able to remove items from my shopping cart”.
Figure 25Test suite 7 has been selected
Select the first test case with ID = 41 and select the Open Test Case button.
Figure 26Opening test case with ID=41
In the Parameter Values section at the bottom, add a new row with quantity 10.
Note: A value of 10 is a legitimate value for the shopping cart, so the cart will refresh to show a quantity of 10 when this value is entered. However, since the purpose of this exerciseis to show what happens when a test iteration fails, we will pretend that this value causes the test to fail in order to demonstrate a test case failure. The assertion that we added expects that the quantity will remain at 1.
Figure 27Parameter Value change
Save the changes to the test case and return to Visual Studio.
Run the tests again and note that the test fails on the fourth iteration.
Select the Failed test within the Test Explorer window to see the details. At the bottom of the Test Explorer window we are notified that 3 out of 4 tests passed, with the fourth data row failing.
Figure 28Test results showing failed assertion
Hands-On LabIntroduction to IntelliTest with Visual Studio Enterprise 2015
Lab version: 14.0.23107.0
Last updated: 9/23/2015
TABLE OF CONTENT
INTRODUCTION TO INTELLITEST WITH VISUAL STUDIO ENTERPRISE 2015 ............................................................. 1
OVERVIEW .......................................................................................................................................................... 3
INTELLITEST EXPLORES YOUR .NET CODE TO GENERATE TEST DATA AND A SUITE OF UNIT TESTS. FOR EVERY STATEMENT IN THE CODE, A TEST INPUT IS GENERATED THAT WILL EXECUTE THAT STATEMENT. A CASE ANALYSIS IS PERFORMED FOR EVERY CONDITIONAL BRANCH IN THE CODE. FOR EXAMPLE, IF STATEMENTS, ASSERTIONS, AND ALL OPERATIONS THAT CAN THROW EXCEPTIONS ARE ANALYZED. THIS ANALYSIS IS USED TO GENERATE TEST DATA FOR A PARAMETERIZED UNIT TEST FOR EACH OF YOUR METHODS, CREATING UNIT TESTS WITH HIGH CODE COVERAGE. ................................................................................................................................................ 3
WHEN YOU RUN INTELLITEST, YOU CAN EASILY SEE WHICH TESTS ARE FAILING AND ADD ANY NECESSARY CODE TO FIX THEM. YOU CAN SELECT WHICH OF THE GENERATED TESTS TO SAVE INTO A TEST PROJECT TO PROVIDE A REGRESSION SUITE. AS YOU CHANGE YOUR CODE, RERUN INTELLITEST TO KEEP THE GENERATED TESTS IN SYNC WITH YOUR CODE CHANGES. ............................................................................................................................... 3
PREREQUISITES ................................................................................................................................................... 4
IN ORDER TO COMPLETE THIS LAB YOU WILL NEED THE VISUAL STUDIO 2015 VIRTUAL MACHINE PROVIDED BY MICROSOFT. FOR MORE INFORMATION ON ACQUIRING AND USING THIS VIRTUAL MACHINE, PLEASE SEE THIS BLOG POST. ......................................................................................................................................................... 4
EXERCISES ........................................................................................................................................................... 4
THIS HANDS-ON LAB INCLUDES THE FOLLOWING EXERCISES: ............................................................................... 4
ESTIMATED TIME TO COMPLETE THIS LAB: 30 MINUTES. ...................................................................................... 4
EXERCISE 1: INTRODUCTION TO INTELLITEST ........................................................................................................ 5
IN PRACTICAL TERMS, WHITE BOX UNIT TEST DEVELOPMENT INCLUDES AN ITERATIVE WORKFLOW INFORMED BYCODE COVERAGE - WRITE A UNIT TEST, SEE WHAT PARTS OF THE CODE ARE NOT COVERED BY THE TEST, WRITE MORE TESTS TO COVER THOSE PARTS, REPEAT UNTIL ALL OF THE CODE IS COVERED. THIS WORKFLOW IS SIMILARTO WHAT WE WOULD USE WHILE WORKING WITH INTELLITEST, AS YOU WILL SEE IN THIS EXERCISE. ................... 5
TASK 1: RUNNING INTELLITEST ............................................................................................................................. 5
........................................................................................................................................................................... 5
........................................................................................................................................................................... 5
........................................................................................................................................................................... 5
TASK 2: UNDERSTANDING INTELLITEST WARNINGS .............................................................................................. 5
........................................................................................................................................................................... 5
........................................................................................................................................................................... 5
........................................................................................................................................................................... 5
........................................................................................................................................................................... 6
........................................................................................................................................................................... 6
........................................................................................................................................................................... 6
........................................................................................................................................................................... 6
........................................................................................................................................................................... 6
........................................................................................................................................................................... 6
TASK 3: PROVIDING MOCK IMPLEMENTATIONS .................................................................................................... 6
........................................................................................................................................................................... 6
........................................................................................................................................................................... 7
........................................................................................................................................................................... 7
........................................................................................................................................................................... 7
........................................................................................................................................................................... 7
........................................................................................................................................................................... 7
........................................................................................................................................................................... 7
TASK 4: FOCUSING ON ‘JUST MY CODE’ ................................................................................................................ 8
........................................................................................................................................................................... 8
........................................................................................................................................................................... 8
........................................................................................................................................................................... 8
TASK 5: MODIFYING THE PARAMETERIZED UNIT TEST TO INCREASE CODE COVERAGE .......................................... 8
IN ORDER TO EXERCISE THE CODE-UNDER-TEST FURTHER, WE NEED TO MODIFY THE PARAMETERIZED UNIT TEST METHOD IN ORDER TO RETURN DATA FROM CALLS TO THE GETDATA METHOD. IN THE PUT, THE ‘TARGET’ IS THE OBJECT THAT CONTAINS DATA TO BE RETURNED BY CALLS TO GETDATA<T>. MORE SPECIFICALLY, T IS EITHER A LOOKUPGAMEPLAYERSDTO ARRAY OR A GETPLAYERSTATISTICSDTO ARRAY. OUR TASK NOW IS TO FILL UP FAKEQUERYDATASTORE WITH CONCRETE INSTANCES OF THESE TYPES. ................................................................ 8
........................................................................................................................................................................... 9
......................................................................................................................................................................... 10
......................................................................................................................................................................... 10
......................................................................................................................................................................... 10
......................................................................................................................................................................... 10
......................................................................................................................................................................... 10
Overview
IntelliTest explores your .NET code to generate test data and a suite of unit tests. For every statement in the code, a test input is generated that will execute that statement. A case analysis is performed for every conditional branch in the code. For example, if statements, assertions, and all operations that can throw exceptions are analyzed. This analysis is used to generate test data for a parameterized unit test for each of your methods, creating unit tests with high code coverage.
When you run IntelliTest, you can easily see which tests are failing and add any necessary code to fix them. You can select which of the generated tests to save into a test project to provide a regression suite. As you change your code, rerun IntelliTest to keep the generated tests in sync with your code changes.
Prerequisites
In order to complete this lab you will need the Visual Studio 2015 virtual machine provided by Microsoft.For more information on acquiring and using this virtual machine, please see this blog post.
Exercises
This hands-on lab includes the following exercises:
Introduction to IntelliTest
Estimated time to complete this lab: 30 minutes.
Exercise 1: Introduction to IntelliTest
In practical terms, white box unit test development includes an iterative workflow informed by code coverage - write a unit test, see what parts of the code are not covered by the test, write more tests to cover those parts, repeat until all of the code is covered. This workflow is similar to what we would use while working with IntelliTest, as you will see in this exercise.
Task 1: Running IntelliTest
Log in as Adam Barr (VSALM\Adam). All user passwords are P2ssw0rd.
Open an Explorer window and then open PokerLeagueManager.sln in Visual Studio from c:\samples\pokerleaguemanager\src. This application tracks stats for a weekly poker league. It has table that has the stats on each player (Games Played, Total Winnings, Total Profit, and so on).
In Solution Explorer, search for GetPlayerStatisticsHandler.cs in the search box and then open the associated file in the code editor.
Scroll down and locate the Handle method that takes a GameDeletedEvent parameter. When a Game is deleted, this method is responsible for updating the stats of the affected players. As with most real-world code, this code interacts with other objects and layers. Our goal with this demonstration is to enable IntelliTest reach 100% code coverage on the Handle method.
Right-click somewhere within this Handle method and then select Run IntelliTest.
Task 2: Understanding IntelliTest Warnings
After IntelliTest runs, only two tests are generated and there is low coverage of the code (6/42 blocks). In addition, there are 5 warnings reported.
Click on the Warnings button.
The first warning is a Runtime Warning, and it indicates that IntelliTest has discovered, and will use, “PokerLeagueManager.Queries.Core.QueryDataStore” as IQueryDataStore. Browsing through the code, we can discover that IQueryDataStore is the type returned by the getter from the QueryDataStore property on the base class BaseHandler. In order to unit test this method, a concrete instantiation of this type is required. However, this may not be the type that you want to use for testing.
Select the Object Creation warning category. IntelliTest has also discovered publicly accessible APIs though which to instantiate QueryDataStore (in this case that happens to be the public constructor). The APIs need to be publicly accessible because IntelliTest needs to actually call them to instantiate the type.
Select the first Object Creation warning. This warning alerts us about the APIs that it discovered. If we prefer, those calls could be persisted as a Factory method by clicking the Fix button, althoughwe will not do so now.
Select the second Object Creation warning. This warning alerts us that IntelliTest was not automatically able to instantiate the object and indicates that we have some more work to do inorder to get this working.
Next, select the Uninstrumented Method warning category followed by the only warning from the list.
It turns out that the QueryDataStore constructor ends up calling into some, as of yet, uninstrumented code, which you can see if you take a quick look through the provided stack trace.
This information is important to note, because IntelliTest works by instrumenting code and monitoring execution. However, it does not instrument the entire universe of code for two reasons, 1) it cannot know a priori what comprises that universe of code and 2) that would makethe system very slow. Therefore, that is why we see this “uninstrumented method” warning.
Select the Boundary warning category followed by the only warning from the list.
When the number of branches in the code path that IntelliTest is exploring is large, it can trip an internal boundary that has been configured for fast interactive performance. Hence, it raises a warning and stops the exploration.
Task 3: Providing Mock Implementations
To proceed further, we need to answer that first question: is that the type you want to use? To unit test the method, we need to provide a mock implementation of IQueryDataStore. Browsing through the solution, we can discover a FakeQueryDataStore. Let’s tell IntelliTest to use that (instead of the QueryDataStore that it discovered).
To start assisting IntelliTest like this, we first need to setup the Parameterized Unit Test (PUT). Click on the Warnings button once again to toggle it off.
Select the two tests and then click the Save button.
IntelliTest will generate a new project named PokerLeagueManager.Queries.Core.Tests, with the generated Parameterized Unit Test found in GetPlayerStatisticsHandlerTest.cs.
In Solution Explorer, expand GetPlayerStatisticsHandlerTest.cs and delete the generated unit test file ending with .g.cs.
In the test project, right-click on the References node and select Add Reference.
In the Reference Manager window, select the Projects node and then add a reference to PokerLeagueManager.Common.Tests. Click OK.
Open GetPlayerStatisticsHandlerTest.cs in the code editor.
Add the following Using statements to the top of the file:
using Microsoft.Pex.Framework.Using;
using PokerLeagueManager.Common.Tests;
To specify that IntelliTest should use FakeQueryDataStore, add the following attribute to the Handle method:
[PexUseType(typeof(FakeQueryDataStore))]
In the IntelliTest Exploration Results window, click the Run button.
Once the IntelliTest run completes, note that the bounds exceeded warning is gone.
Task 4: Focusing on ‘Just my Code’
Click the Warnings button.
The Object Creation warning now shows that IntelliTest has discovered how to instantiate FakeQueryDataStore. The Details alert us about the APIs it can use to instantiate it, and if we prefer, we can persist this as a factory method.
Select the Uninstrumented Method category and note the warnings shown. These indicate that IntelliTest has ended up calling into uninstrumented code once again. If you inspect the stack trace associated with these warnings, it is possible to see that the calls where execution transitions into uninstrumented code is at the constructor and GetData<T> methods.
Since we are not testing the mock, let us suppress these uninstrumented method warnings. Select one of the warnings and then click the Supress button.
Open PexAssemblyInfo.cs from the test project and note the added assembly attribute, PexSupressUninstrumentedMethodFromType.
Run IntelliTest once again and verify that only the object creation warning remains.
Task 5: Modifying the Parameterized Unit Test to Increase Code Coverage
In order to exercise the code-under-test further, we need to modify the parameterized unit test method in order to return data from calls to the GetData method. In the PUT, the ‘target’ is the object that contains data to be returned by calls to GetData<T>. More specifically, T is either a LookupGamePlayersDto array or a GetPlayerStatisticsDto array. Our task now is to fill up FakeQueryDataStore with concrete instances of these types.
Since IntelliTest can synthesize data values, we will add this to the PUT’s signature. We want 2 instances of LookupGamePlayersDto, and 1 instance of GetPlayerStatisticsDto. Further, we want to associate the statistics for the first player.
Add a reference to PokerLeageManager.Common.DTO in the test project (right-click on References, Add Reference…)
Add the following Using statements to the top of GetPlayerStatisticsHandlerTest.cs:
using PokerLeagueManager.Common.DTO;
using PokerLeagueManager.Common.DTO.DataTransferObjects.Lookups;
Modify the signature of the PUT method by adding the following parameters:
LookupGamePlayersDto[] lookupGamePlayers,
GetPlayerStatisticsDto[] getPlayerStatistics
The signature of the PUT method should now look like the following screenshot.
We should also add in some additional hints to IntelliTest about the assumptions we would like to make about the input parameters. Insert the following code snippet to the beginning of the PUT:
// assume
PexAssume.IsNotNull(lookupGamePlayers);
PexAssume.IsTrue(lookupGamePlayers.Length == 2);
PexAssume.IsNotNull(lookupGamePlayers[0]);
PexAssume.IsNotNull(lookupGamePlayers[1]);
PexAssume.IsNotNull(getPlayerStatistics);
PexAssume.IsTrue(getPlayerStatistics.Length == 1);
PexAssume.IsNotNull(getPlayerStatistics[0]);
PexAssume.IsTrue(lookupGamePlayers[0].PlayerName == getPlayerStatistics[0].PlayerName);
Next, we will prime the target with these steps (add this code just after the previous code that you inserted):
// arrange
foreach (var lookupGamePlayer in lookupGamePlayers)
{
target.QueryDataStore.Insert<LookupGamePlayersDto>(lookupGamePlayer);
}
target.QueryDataStore.Insert<GetPlayerStatisticsDto>(getPlayerStatistics[0]);
The next step is to exercise the code under test, but the code to do that is already in place:
target.Handle(e);
At the very end of the PUT, we simply query for the statistics and then assert the observed value of its fields. Add a Using statement to the top of the file to System.Linq and then add the following snippet to the very end of the PUT:
// assert
var playerStats = target.QueryDataStore.GetData<GetPlayerStatisticsDto>().Single();
PexObserve.ValueAtEndOfTest("playerStats", playerStats);
Delete the .g.cs file once again, since we changed the signature of the PUT.
Run IntelliTest and note that we now have full code coverage (52/52 blocks), with 3 passing tests, 4 failing tests, and a number of warnings.
Take a quick look at the warnings, and note that none is related to the code-under-test. Therefore, go ahead and Suppress all of the warnings.
Run IntelliTest and verify that the warnings are gone.
Two of the tests fail because they uncover a NullReferenceException when the ‘e’ parameter is null.
One of the tests uncovers a potential DivideByZeroException. This will happen if stats.GamesPlayed has a value of ‘1’. In this case, the statement stats.GamesPlayed-- will make it ‘0’, and subsequently stats.Profit / stats.GamesPlayed will raise the exception.
To see where in code the DivideByZeroException was thrown, select the test in the IntelliTest Exploration Results window, expand the Stack Trace on the right-hand side, and then double-click on the first line shown.
Another test uncovered an OverflowException.
Select the failed test in the IntelliTest Exploration Results window and take a moment to scroll through the Details section. This shows the specific test and parameters that were used against the code-under-test in order to generate the exception.
This shows that IntelliTest has generated tests that uncovered previously unknown errors in the code. If we were to add additional assertions about the expected behavior of the code-under-test, then it would generate tests for validating that as well.
Manual and exploratory testing scenarios andcapabilities
Visual Studio 2015 | Team Services | TFS 2015
Drive quality and collaboration throughout the development process.
Scenarios and capabilities:
Manual testing
Exploratory testing
User acceptance testing
Reports and dashboards
Browser-based testing
Rich testing environment
Manual testing
Visual Studio Test Professional provides access to the Test hub in Team Foundation Server and Team
Services. Use this to coordinate all test management activities including test planning, authoring,
execution, and tracking from a central location. The Test hub gives product owners and business
analysts critical insight into progress against defined acceptance criteria and quality metrics.
Team Services (/docs/overview) > (/docs/test/overview)
Manual and exploratory testing of your applications - Visual Studio Team ... https://www.visualstudio.com/en-us/docs/test/manual-exploratory-testing...
1 of 7 02.09.2016 13:29
Get started with test case management (getting-started/create-a-test-plan)
Web-based test case management with TFS (https://msdn.microsoft.com/magazine
/dn890369.aspx)
Videos: Test case management (https://channel9.msdn.com/series/test-tools-in-visual-studio)
Exploratory testing
Visual Studio Test Professional enables you to perform ad-hoc and exploratory tests without requiring
predefined test cases or test steps. The exploratory testing tool and browser extensions include action
recording, commenting with annotations, rich data capture, and bug-reporting capabilities. Use it to
quickly and easily create test cases from exploratory test recordings and add them to test plans and
suites for future manual or automated test passes.
Manual and exploratory testing of your applications - Visual Studio Team ... https://www.visualstudio.com/en-us/docs/test/manual-exploratory-testing...
2 of 7 02.09.2016 13:29
Get started with exploratory testing (getting-started/perform-exploratory-tests)
Learn more and download about exploratory testing extension now
(https://marketplace.visualstudio.com/items?itemname=ms.vss-exploratorytesting-web)
Videos: Exploratory testing (http://aka.ms/vstest)
User acceptance testing
With the today's faster development pace, tools that enable test teams to more easily verify that value
customers have requested has been delivered, and is of high quality, have become increasingly
important. This type of testing is often referred to as user acceptance testing (UAT). Visual Studio Team
Services and Team Foundation Server have been enhanced to make this easy.
Manual and exploratory testing of your applications - Visual Studio Team ... https://www.visualstudio.com/en-us/docs/test/manual-exploratory-testing...
3 of 7 02.09.2016 13:29
Get started with user acceptance testing (getting-started/user-acceptance-testing)
Acceptance testing (https://msdn.microsoft.com/library/ff649646.aspx)
Video: User acceptance testing (http://aka.ms/vstest)
Create actionable test results with reports and dashboards
Get end-to-end traceability across all artifacts by running tests and logging defects from your browser,
or use the Microsoft Test Manager (MTM) client. Track and assess quality throughout your testing
lifecycle, and verify requirements with user acceptance testing. All your quality engineers and user
acceptance testing groups can use the same test tools, and you can assign pre-defined test cases to
your user acceptance testers. You pay for the tools only when you need them.
Manual and exploratory testing of your applications - Visual Studio Team ... https://www.visualstudio.com/en-us/docs/test/manual-exploratory-testing...
4 of 7 02.09.2016 13:29
Get started with reports and dashboards (getting-started/track-test-status)
Web-based test case management with TFS (https://msdn.microsoft.com/magazine
/dn890369.aspx)
Video: Create actionable test results (http://aka.ms/vstest)
Browser-based testing
Execute manual and acceptance tests with only a browser. Run manual tests and record test results for
each test step using a toolset optimized around the needs of testers. The web-based test runner
enables pass-fail results, tracking of test steps, rich commenting, and bug reporting capabilities. You
can also use the test runner for your user acceptance testing needs to run and report test case
execution by multiple acceptance testers.
Manual and exploratory testing of your applications - Visual Studio Team ... https://www.visualstudio.com/en-us/docs/test/manual-exploratory-testing...
5 of 7 02.09.2016 13:29
Get started with browser-based testing (getting-started/run-manual-tests)
Create manual tests using the web portal (https://msdn.microsoft.com/library/dd286729.aspx)
Video: Browser based testing video (http://aka.ms/vstest)
Rich experience for running, recording, and repeating manualtests
Microsoft Test Manager (MTM) offers a fully configurable test runner that captures detailed records of
steps performed, behaviors observed, and the status of each test step. Pause testing to report a bug,
even if it’s not directly related to the current test. High-fidelity defect reports go directly to the
development team, including all the technical data needed to reproduce and determine the root
cause. You can record tests for later playback to speed test sessions and enable automation.
Manual and exploratory testing of your applications - Visual Studio Team ... https://www.visualstudio.com/en-us/docs/test/manual-exploratory-testing...
6 of 7 02.09.2016 13:29
Download Microsoft Test Manager (https://marketplace.visualstudio.com
/items?itemname=ms.vss-testmanager-web)
Plan manual tests with Microsoft Test Manager (https://msdn.microsoft.com/library
/dd286659.aspx)
Video: Manual testing (http://aka.ms/vstest)
Help and support
Submit bugs through Connect (https://connect.microsoft.com/visualstudio), make suggestions on
Uservoice (http://visualstudio.uservoice.com/forums/121579-visual-studio), and send quick thoughts
using the Send-a-Smile link in the Visual Studio, Team Services, or TFS title bar. We look forward to
your feedback.
© 2016 Microsoft
Manual and exploratory testing of your applications - Visual Studio Team ... https://www.visualstudio.com/en-us/docs/test/manual-exploratory-testing...
7 of 7 02.09.2016 13:29
Parallel and Context Sensitive Test Execution withVisual studio 2015 Update 1
0 2 37
February 8, 2016 by Charles Sterling // 1 Comments
﴾Editors Note: One of the most popular series of blog posts on the ALM Blog was Terje’s posts on Unit Testing. Sowhen he asked to republish his Norwegian post on the ALM blog the answer was of course an enthusiastic YES!. Alittle off topic this post also highlights we need a post on executing parallel test runs with the new “Run Functional TestTask” as that paradigm is different using separate machines and “test containers” ﴾think assemblies﴿﴿
Visual studio 2015 Update 1 : Parallel andContext Sensitive Test ExecutionVisual Studio 2015 Update 1 contains a bunch of improvements and bug fixes. In this post we will have a look at twoof the performance improvements, that may give a significant impact on your overall development performance.
Parallel test executionIn Visual Studio 2015 Update 1 the test engine can execute test assemblies in parallel. This can improve your testperformance significantly. This is off by default, so you need to enable it.
The system runs the test assemblies in parallel on as many cores as you specify, and of course, up to the maximumnumber of cores on your machine. If you only have a single test project you will of course have no benefit of this. Thisis a solution for the slightly larger projects.
It is independent of the type of test framework you use, and therefore works with both MSTest, NUnit 2, NUnit 3 andXUnit.
This mechanism is completely independent on whether the underlying framework supports parallelism or not.
To turn on the feature, you must use a runsettings file, which you must enable in Visual Studio. The easy way to get acorrect runsettings file in, is to use one of these templates, note that you need at least version 3.1 of these. When youhave this item extension installed, you have 3 templates, and if you’re just after the parallelism, use the one namedParallel.
Do as follow:
Select Solution, Right click and choose Add/New item
You will now see a list like the one below:
Subscribe
Download Visual Studio
Download TFS 2015
Visual Studio Team Services
Search this blogSearch all blogs
Share This Post
Tags
Administration Agile ALMAnnouncementAutomated Testing AzureBuild Process cloud load testing
Coded UI Test Coded UITesting Code Visualization
Customizing Debugging
DevOps
DiagnosticsExtensibility Feedback GitGuidance insightsIntelliTrace Load TestingManaging Technical DebtManual Testing Process Templates
Rangers ReleaseManagement SonarQube
Team Foundation Build
★★★★★★★★★★★★★★★
Search MSDN with Bing
Server & Tools Blogs > Developer Tools Blogs > Microsoft Application Lifecycle Management
Microsoft Application LifecycleManagementLearn to use Visual Studio, Visual Studio Team Services, Application Insights and Team Foundation Server to decreaserework, increase transparency into your application and increase the rate at which you can ship high quality softwarethroughout the application lifecycle
ExecutiveBloggers
Visual Studio ApplicationLifecycleManagement
Languages .NET PlatformDevelopment
Sign in
Page 1 of 6 Assembled by RunPDF.com
Select Parallel and you will see an instance of it under Solution Items in your solution.
That’s all it takes to enable parallel execution !
Just remember to activate it under the Visual Studio Test menu, do like this:
﴾1﴿ Go to Select Test Settings file from the top menu Test/Test settings.
﴾2﴿ Then select the wanted Runsettings file, in this case I named it Parallel2.runsettings.
﴾3﴿ Verify on the same menu that it is selected and checked.
The parameters:Looking at the content of the runsettings file, we see this:
There is only one item here we can change, the MaxCpuCount value. It controls how many parallel processes we canrun. The value 0 means “run as much as you can, limited by availability of cores on your machine”. The value 1 meansit will run sequentially, that is the same as the default with only one process activated. Any other number will limit atthat number, or the number of available cores on your machine.
Why don’t we include Code Coverage?You might have noticed that in the description for Parallel it says code coverage is not included. This means that whenyou activate this runsetting, you will not active code coverage during the same run. The other two templates do exactlythat, it doesn’t really give you any benefits.
Team Foundation SDK TeamFoundationServer Testing Test
Management TFS TFServiceUnit Testing Version Control
Visual Studio Visual Studio
2010 Visual Studio 2012 Visual
studio 2013 VisualStudio ALM VisualStudio Online Visual
Studio Team Services WorkItems
Archives
September 2016 ﴾1﴿August 2016 ﴾21﴿July 2016 ﴾15﴿June 2016 ﴾20﴿May 2016 ﴾17﴿April 2016 ﴾18﴿March 2016 ﴾24﴿February 2016 ﴾13﴿January 2016 ﴾22﴿All of 2016 ﴾151﴿All of 2015 ﴾186﴿All of 2014 ﴾173﴿All of 2013 ﴾154﴿All of 2012 ﴾207﴿All of 2011 ﴾81﴿All of 2010 ﴾28﴿All of 2009 ﴾1﴿All of 2008 ﴾5﴿All of 2007 ﴾30﴿All of 2006 ﴾76﴿All of 2005 ﴾22﴿
Subscribe Blog via Email
Subscribe to this blog and receivenotifications of new posts by email.
Email Address
Subscribe! Unsubscribe
Page 2 of 6 Assembled by RunPDF.com
If you activate the Code Coverage the performance will be reduced, because code coverage also needs to run, and itruns sequentially after the test runs, and will therefore increase the overall time significantly. And, it will not give youanything you really need.
The fact that you don’t active Code Coverage does not imply that you can’t run code coverage from Team Explorer –you can! Further, the results from this code coverage run is not used in the Test Explorer – strangely enough. Theactivation done in the two other templates just activate a background code coverage run, which results are just storedin a code coverage result file under the Test Results folder.
This file doesn’t really give you any benefits in Visual Studio, but is useful if you want these coverage data into anotherprogram , one good example is NDepend.
One benefit is that this coverage section not only enables the background run of code coverage, but also sets up thefiltering of the code coverage data, and this might be interesting for you. These filters will also take effect when you runthe Test Explorer “Analyze code coverage from selected tests”
In the linked article above there is a bunch of settings for code coverage, instead of using these, just use the templateCompleteRunSettings, which includes this, plus the other settings. The XML in the linked article is not up to date.
The templaten CompleteRunsettings has the MaxCpuCount set to 0, it will then run as parallel as possible.
The templaten CoverageNoParallel is nearly identical, with exactly the same fields and settings, except theMaxCpuCount which is set to 1, thus sequential test running.
There is also an MSDN artikkel about configuration of unit tests which are more correct,m but it doesn’t contain bthesame detailed information as the link above. In this however, all fields are explained.
Test project for parallel runsI have made a simple test project for parallel runs. It has 4 test projects, with one test each, and each test runs for 5seconds. You can download the source code from here, and then run the tests yourself.
Page 3 of 6 Assembled by RunPDF.com
﴾1﴿ Running without settings at all. It is then runs sequentially, and we see the total time is 26 seconds. Since each testuses 5 seconds we have a 6 seconds overhead.
﴾2﴿ Running with CodeCoverageNoParallel, which is identical to the old runsettings file, we will, get code coverage run inaddition, but no parallel runs. The total time increases to 31 seconds.
﴾3﴿ We then uses the complete settings, using both CodeCoverage and Parallel. The total time then decreases to 18seconds.
﴾4﴿ And finally using only Parallel without code coverage, we’re down in 12 seconds.
So from ﴾4﴿ to ﴾1﴿, we gain more than a 2:1 in performance. That is pretty good!
The machine this was tested on has more cores – 8 in all, but it also runs a bunch of other programs, so not all corescan be made available for the testing processes. You will gain some, but not equal to the number of cores – in practice. Anyway, a gain of 2:1 is pretty good and pretty significant in your daily work.
Parallel on build serverDoes this work on a build server? The answer is Yes, in a build definition you can define which runsettingfile to run, sothis will also have an effect on the build server run. But, do you really want that? That all depends on how you manybuild servers you have, how they are configured, and how many check‐ins/commits you have. That si, how loadedyour build servers are. Normally you set up a build agent to run one per core on the machine. If you have many builddefinitions and many commits/Checkins happening continuously during the day, the build agents will be pretty busy.,then there will be little benefit in setting up the tests to run in parallel in addition, as they will “steal” cores from theother builds. There can also be an overhead that eat up all your gain.
If you, on the other hand, have a builds machine that is not that much loaded, and not so many commits continuously,then by all means – why not? Grab the extra performance by running your tests in parallel.
There are also other ways to run in parallel, see this article for a different approach.
Context sensitive test executionContext sensitive test execution is new in Update 1 – so what is that? Under given circumstances, the new TestExplorer will only test the assemblies that it detects tests code that you have changed.
This means that if you work with a certain piece of code in a given module of your system, and your code is distributedover multiple modules, then for each build, only the code that you changes is being build. That is the normalincremental build which is by default. New is that also only the tests touching this code is being run, and then not thetests for code that has not been touched. It is pretty obvious that for larger systems this can mean a pretty dramatic
Page 4 of 6 Assembled by RunPDF.com
speedup in the test run performance.
There is one condition for this to work: You must activate “Run tests after build”.
You do this either in the Test Explorer ﴾1﴿ or from the top menu Test/Test Settings/Run Tests after Build ﴾2﴿.
The effect of this can be seen in the screenshots below:
﴾1﴿ Activate “Run tests after build”
﴾2﴿ Do some code changes
﴾3﴿ Just select Build, or use the keyboard shortcut
﴾4﴿ Just the touched test are being run, the other ones are shown dimmed green. They ran green the last time theywere run, but they were not run this time, because no code what changed that affected them. .
If you have larger projects this can give a significant gain in performance, and make your turn around from buildthrough test back to edit mode much faster – and that is what is important during development, right?
TestingTags
7 months agoScott Harmen
How does this work with running CodedUI tests in parallel?
From Charles SterlingHello Scott,As CodedUI Tests expect to the only keyboard input they need to be run in series.Running a CodedUI Test in parallel on a single machine would not work as the tests wouldinterfere with each other.﴾This regularly happens when developers accidently add CodedUI Tests to a Load Test with
Back totop
Page 5 of 6 Assembled by RunPDF.com
multiple users﴿.That said if you do want to scale out your Coded UI Tests there is a new build task called “RunFunctional Test” and “Deploy Visual Studio Test Agent”. If you partition your tests into separateassemblies this would enable you to run your CodedUI Test in parallel…using computers.ThanksChuck
Comments are closed.
© 2016 Microsoft Corporation. Terms of Use Trademarks Privacy & Cookies
Page 6 of 6 Assembled by RunPDF.com
Unit testing native code with Test Explorer
In Visual Studio, you can create unit tests for unmanaged code written in C++. Unmanaged code is sometimes referred to as
native code.
The following procedure contains the essential information that will get you started. The later sections provide a
walkthrough that describes the steps in more detail.
To write unit tests for an unmanaged code DLL
Use the Native Test Project template to create a separate Visual Studio project for your tests.
The project contains some sample test code.
1.
Make the DLL accessible to the test project:
#include a .h file that contains declarations of the DLL’s externally-accessible functions.
The .h file should contain function declarations marked with _declspec(dllimport). Alternatively, you can
export the methods using a DEF file. For more information, see Importing and Exporting.
Your unit tests can access only functions that are exported from the DLL under test.
Add the DLL project to the References of the test project:
In the Properties of the test project, expand Common Properties, Framework and References, and choose
Add Reference.
2.
In the test project, create test classes and test methods by using the TEST macros and Assert class in the following
way:
Assert contains several static functions that you can use to verify the result of a test.
3.
#include "stdafx.h"
#include <CppUnitTest.h>
#include "..\MyProjectUnderTest\MyCodeUnderTest.h"
using namespace Microsoft::VisualStudio::CppUnitTestFramework;
TEST_CLASS(TestClassName)
{
public:
TEST_METHOD(TestMethodName)
{
// Run a function under test here.
Assert::AreEqual(expectedValue, actualValue, L"message", LINE_INFO());
}
}
C++
Unit testing native code with Test Explorer https://msdn.microsoft.com/en-us/library/hh270864(d=printer).aspx
1 of 12 02.09.2016 13:47
The LINE_INFO() parameter is optional. In cases where there is no PDB file, it allows the test runner to identify
the location of a failure.
You can also write test setup and cleanup methods. For more information, open the definition of the
TEST_METHOD macro, and read the comments in CppUnitTest.h
You cannot nest test classes.
Use Test Explorer to run the tests:
On the View menu, choose Other Windows, Test Explorer.a.
Build the Visual Studio solution.b.
In Test Explorer, choose Run All.c.
To investigate any test in more detail in Test Explorer:
Select the test name to see more details such as a failure message and stack trace.i.
Open the test name (for example by double-clicking) to go to the failure location or to the test code.ii.
On the shortcut menu for a test, choose Debug Selected Test to run the test in the debugger.iii.
d.
4.
Walkthrough: Developing an unmanaged DLL with Test ExplorerYou can adapt this walkthrough to develop your own DLL. The principal steps are as follows:
Create a Native Test Project. The tests are created in a separate project from the DLL that you are developing.1.
Create a DLL Project. This walkthrough creates a new DLL, but the procedure for testing an existing DLL is similar.2.
Make the DLL functions visible to the tests.3.
Iteratively augment the tests. We recommend a "red-green-refactor" cycle, in which development of the code is led
by the tests.
4.
Debug failing tests. You can run tests in debug mode.5.
Refactor while keeping the tests unchanged. Refactoring means improving the structure of the code without
changing its external behavior. You can do it to improve the performance, extensibility, or readability of the code.
Because the intention is not to change the behavior, you do not change the tests while making a refactoring
change to the code. The tests help make sure that you do not introduce bugs while you are refactoring. You can
therefore make such changes with much more confidence than if you did not have the tests.
6.
Check coverage. Unit tests are more useful when they exercise more of your code. You can discover which parts of
your code have been used by the tests.
7.
Isolate units from external resources. Typically, a DLL is dependent on other components of the system that you are
developing, such as other DLLs, databases, or remote subsystems. It is useful to test each unit in isolation from its
dependencies. External components can make tests run slowly. During development, the other components might
not be complete.
8.
Unit testing native code with Test Explorer https://msdn.microsoft.com/en-us/library/hh270864(d=printer).aspx
2 of 12 02.09.2016 13:47
Create a native unit test project
On the File menu, choose New, Project.
In the dialog box, expand Installed, Templates, Visual C++, Test.
Choose the Native Test Project template.
In this walkthrough, the test project is named NativeRooterTest.
1.
In the new project, inspect unittest1.cpp
Notice that:
Each test is defined by using TEST_METHOD(YourTestName){...}.
You do not have to write a conventional function signature. The signature is created by the macro
TEST_METHOD. The macro generates an instance function that returns void. It also generates a static
function that returns information about the test method. This information allows the test explorer to find
the method.
2.
Unit testing native code with Test Explorer https://msdn.microsoft.com/en-us/library/hh270864(d=printer).aspx
3 of 12 02.09.2016 13:47
Test methods are grouped into classes by using TEST_CLASS(YourClassName){...}.
When the tests are run, an instance of each test class is created. The test methods are called in an unspecified
order. You can define special methods that are invoked before and after each module, class, or method. For
more information, see Organizing C++ Tests.
Verify that the tests run in Test Explorer:
Insert some test code:
Notice that the Assert class provides several static methods that you can use to verify results in test
methods.
a.
On the Test menu, choose Run , All Tests.
The test builds and runs.
Test Explorer appears.
The test appears under Passed Tests.
b.
3.
Create an Unmanaged DLL project
Create a Visual C++ project by using the Win32 Project template.
In this walkthrough, the project is named RootFinder.
1.
TEST_METHOD(TestMethod1)
{
Assert::AreEqual(1,1);
}
C++
Unit testing native code with Test Explorer https://msdn.microsoft.com/en-us/library/hh270864(d=printer).aspx
4 of 12 02.09.2016 13:47
Select DLL and Export Symbols in the Win32 Application Wizard.
The Export Symbols option generates a convenient macro that you can use to declare exported methods.
2.
Declare an exported function in the principal .h file:
The declarator __declspec(dllexport) causes the public and protected members of the class to be visible
3.
Unit testing native code with Test Explorer https://msdn.microsoft.com/en-us/library/hh270864(d=printer).aspx
5 of 12 02.09.2016 13:47
outside the DLL. For more information, see Using dllimport and dllexport in C++ Classes.
In the principal .cpp file, add a minimal body for the function:4.
Couple the test project to the DLL project
Add the DLL project to the project references of the test project:
Open the properties of the test project and choose Common Properties, Framework and References.a.
Choose Add New Reference.
In the Add Reference dialog box, select the DLL project and choose Add.
b.
1.
In the principal unit test .cpp file, include the .h file of the DLL code:2.
Add a basic test that uses the exported function:3.
// Find the square root of a number.
double CRootFinder::SquareRoot(double v)
{
return 0.0;
}
#include "..\RootFinder\RootFinder.h"
TEST_METHOD(BasicTest)
C++
C++
C++
Unit testing native code with Test Explorer https://msdn.microsoft.com/en-us/library/hh270864(d=printer).aspx
6 of 12 02.09.2016 13:47
Build the solution.
The new test appears in Test Explorer.
4.
In Test Explorer, choose Run All.5.
You have set up the test and the code projects, and verified that you can run tests that run functions in the code project.
Now you can begin to write real tests and code.
Iteratively augment the tests and make them pass
Add a new test:
Tip
1.
{
CRootFinder rooter;
Assert::AreEqual(
// Expected value:
0.0,
// Actual value:
rooter.SquareRoot(0.0),
// Tolerance:
0.01,
// Message:
L"Basic test failed",
// Line number ‐ used if there is no PDB file:
LINE_INFO());
}
TEST_METHOD(RangeTest)
{
CRootFinder rooter;
for (double v = 1e‐6; v < 1e6; v = v * 3.2)
{
double actual = rooter.SquareRoot(v*v);
Assert::AreEqual(v, actual, v/1000);
}
}
C++
Unit testing native code with Test Explorer https://msdn.microsoft.com/en-us/library/hh270864(d=printer).aspx
7 of 12 02.09.2016 13:47
We recommend that you do not change tests that have passed. Instead, add a new test, update the code so that
the test passes, and then add another test, and so on.
When your users change their requirements, disable the tests that are no longer correct. Write new tests and
make them work one at a time, in the same incremental manner.
Build the solution, and then in Test Explorer, choose Run All.
The new test fails.
Tip
Verify that each test fails immediately after you have written it. This helps you avoid the easy mistake of writing
a test that never fails.
2.
Enhance the code under test so that the new test passes:3.
Build the solution and then in Test Explorer, choose Run All.
Both tests pass.
4.
#include <math.h>
...
double CRootFinder::SquareRoot(double v)
{
double result = v;
double diff = v;
while (diff > result/1000)
{
double oldResult = result;
result = result ‐ (result*result ‐ v)/(2*result);
diff = abs (oldResult ‐ result);
}
return result;
}
C++
Unit testing native code with Test Explorer https://msdn.microsoft.com/en-us/library/hh270864(d=printer).aspx
8 of 12 02.09.2016 13:47
Tip
Develop code by adding tests one at a time. Make sure that all the tests pass after each iteration.
Debug a failing test
Add another test:1.
#include <stdexcept>
...
// Verify that negative inputs throw an exception.
TEST_METHOD(NegativeRangeTest)
{
wchar_t message[200];
CRootFinder rooter;
for (double v = ‐0.1; v > ‐3.0; v = v ‐ 0.5)
{
try
{
// Should raise an exception:
double result = rooter.SquareRoot(v);
_swprintf(message, L"No exception for input %g", v);
Assert::Fail(message, LINE_INFO());
}
catch (std::out_of_range ex)
{
continue; // Correct exception.
}
catch (...)
{
_swprintf(message, L"Incorrect exception for %g", v);
Assert::Fail(message, LINE_INFO());
}
}
}
C++
Unit testing native code with Test Explorer https://msdn.microsoft.com/en-us/library/hh270864(d=printer).aspx
9 of 12 02.09.2016 13:47
Build the solution and choose Run All.2.
Open (or double-click) the failed test.
The failed assertion is highlighted. The failure message is visible in the detail pane of Test Explorer.
3.
To see why the test fails, step through the function:
Set a breakpoint at the start of the SquareRoot function.a.
On the shortcut menu of the failed test, choose Debug Selected Tests.
When the run stops at the breakpoint, step through the code.
b.
4.
Insert code in the function that you are developing:5.
All tests now pass.6.
#include <stdexcept>
...
double CRootFinder::SquareRoot(double v)
{
// Validate parameter:
if (v < 0.0)
{
throw std::out_of_range("Can't do square roots of negatives");
}
C++
Unit testing native code with Test Explorer https://msdn.microsoft.com/en-us/library/hh270864(d=printer).aspx
10 of 12 02.09.2016 13:47
Refactor the code without changing tests
Simplify the central calculation in the SquareRoot function:1.
Build the solution and choose Run All, to make sure that you have not introduced an error.
Tip
A good set of unit tests gives confidence that you have not introduced bugs when you change the code.
Keep refactoring separate from other changes.
2.
Next steps
Isolation. Most DLLs are dependent on other subsystems such as databases and other DLLs. These other
components are often developed in parallel. To allow unit testing to be performed while the other components are
not yet available, you have to substitute mock or
Build Verification Tests. You can have tests performed on your team’s build server at set intervals. This ensures
that bugs are not introduced when the work of several team members is integrated.
Checkin tests. You can mandate that some tests are performed before each team member checks code into
source control. Typically this is a subset of the complete set of build verification tests.
You can also mandate a minimum level of code coverage.
// old code:
// result = result ‐ (result*result ‐ v)/(2*result);
// new code:
result = (result + v/result)/2.0;
Unit testing native code with Test Explorer https://msdn.microsoft.com/en-us/library/hh270864(d=printer).aspx
11 of 12 02.09.2016 13:47
See AlsoTasks
Walkthrough: Creating and Using a Dynamic Link Library (C++)
Concepts
Importing and Exporting
Other Resources
An Overview of Managed/Unmanaged Code Interoperability
Debugging Native Code
© 2016 Microsoft
Unit testing native code with Test Explorer https://msdn.microsoft.com/en-us/library/hh270864(d=printer).aspx
12 of 12 02.09.2016 13:47
Hands-On LabUnit Testing, Code Coverage and Code Clone Analysis with Visual Studio 2015
Lab version: 14.0.23107.0
Last updated: 8/12/2015
TABLE OF CONTENT
UNIT TESTING, CODE COVERAGE AND CODE CLONE ANALYSIS WITH VISUAL STUDIO 2015 .................................... 1
OVERVIEW .......................................................................................................................................................... 4
1.IN THIS LAB, YOU WILL LEARN ABOUT HOW THE UNIT TESTING ENGINE IN VISUAL STUDIO IS EXTENSIBLE AND OPENS THE DOOR FOR 3RD PARTY UNIT TESTING ADAPTERS SUCH AS NUNIT AND XUNIT.NET. IN ADDITION, YOU WILL SEE SOME OF THE IMPROVEMENTS MADE TO CODE COVERAGE SUPPORT. YOU WILL ALSO LEARN ABOUT HOW THE CODE CLONE CAPABILITY GOES HAND-IN-HAND WITH DELIVERING HIGH QUALITY SOFTWARE BY HELPING YOU IDENTIFY BLOCKS OF SEMANTICALLY SIMILAR CODE WHICH MAY BE CANDIDATES FOR COMMON BUG FIXES OR REFACTORING. .............................................................................................................................. 4
PREREQUISITES ................................................................................................................................................... 4
1.IN ORDER TO COMPLETE THIS LAB YOU WILL NEED THE VISUAL STUDIO 2015 VIRTUAL MACHINE PROVIDED BY MICROSOFT. FOR MORE INFORMATION ON ACQUIRING AND USING THIS VIRTUAL MACHINE, PLEASE SEE THIS BLOG POST. ......................................................................................................................................................... 4
ABOUT THE FABRIKAM FIBER SCENARIO .............................................................................................................. 4
1.THIS SET OF HANDS-ON-LABS USES A FICTIONAL COMPANY, FABRIKAM FIBER, AS A BACKDROP TO THE SCENARIOS YOU ARE LEARNING ABOUT. FABRIKAM FIBER PROVIDES CABLE TELEVISION AND RELATED SERVICES TO THE UNITED STATES. THEY ARE GROWING RAPIDLY AND HAVE EMBRACED WINDOWS AZURE TO SCALE THEIR CUSTOMER-FACING WEB SITE DIRECTLY TO END-USERS TO ALLOW THEM TO SELF-SERVICE TICKETS AND TRACK TECHNICIANS. THEY ALSO USE AN ON-PREMISES ASP.NET MVC APPLICATION FOR THEIR CUSTOMER SERVICE REPRESENTATIVES TO ADMINISTER CUSTOMER ORDERS. ..................................................................................... 4
EXERCISES ........................................................................................................................................................... 4
1.THIS HANDS-ON LAB INCLUDES THE FOLLOWING EXERCISES: ............................................................................ 4
2.UNIT TESTING ............................................................................................................................................................ 4 3.CODE COVERAGE ........................................................................................................................................................ 4 4.CODE CLONE ANALYSIS ................................................................................................................................................. 4
1.ESTIMATED TIME TO COMPLETE THIS LAB: 30 MINUTES. ................................................................................... 4
EXERCISE 1: UNIT TESTING ................................................................................................................................... 5
1.IN THIS EXERCISE, YOU WILL LEARN ABOUT THE UNIT TESTING STORY IN VISUAL STUDIO 2015. ........................ 5
2.LOG IN AS JULIA (VSALM\JULIA). ALL USER PASSWORDS ARE P2SSW0RD. ............................................................................... 5 3.LAUNCH VISUAL STUDIO 2015 FROM THE TASKBAR AND OPEN TEAM EXPLORER. YOU SHOULD NOW BE CONNECTED TO THE FABRIKAMFIBER TEAM PROJECT. IF YOU ARE NOT AUTOMATICALLY CONNECTED TO THE FABRIKAMFIBER PROJECT, SELECT THE CONNECT TO TEAM PROJECTS BUTTON () TO DO SO. ...................................................................................................................................................... 5 4.IN TEAM EXPLORER – HOME, DOUBLE-CLICK ON THE FIRST FABRIKAMFIBER.CALLCENTER.SLN SOLUTION. ........................................... 5
4.A. ..................................................................................................................................................................... 5
4.B.FIGURE ......................................................................................................................................................... 5
4.C.LOADING FABRIKAM FIBER SOLUTION ........................................................................................................... 5
5.PRESS CTRL + SHIFT + B TO BUILD THE SOLUTION. .............................................................................................................. 5 6.OPEN THE TEST EXPLORER WINDOW FROM TEST | WINDOWS | TEST EXPLORER. NOTE THAT DISCOVERED TESTS ARE INITIALLY SET TO THE NOT RUN STATE. ............................................................................................................................................................ 5
6.A. ..................................................................................................................................................................... 5
6.B.FIGURE ......................................................................................................................................................... 5
6.C.TEST EXPLORER WINDOW .............................................................................................................................. 5
7.SELECT THE RUN ALL LINK TO EXECUTE ALL DISCOVERED UNIT TESTS. ........................................................................................ 5
7.A. ..................................................................................................................................................................... 5
7.B.FIGURE ......................................................................................................................................................... 5
7.C.RUNNING ALL UNIT TESTS .............................................................................................................................. 5
7.E.NOTE: VISUAL STUDIO ALSO PROVIDES CONTINUOUS TEST RUNNER FUNCTIONALITY THAT CAN BE ENABLED WITH THE TEST | TEST SETTINGS | RUN TESTS AFTER BUILD OPTION. WITH THIS OPTION SELECTED, TESTS WILL ALWAYS RUN AFTER BUILDS. ............................................................................................................................... 5
8.EXPAND THE PASSED TESTS GROUP (IF NECESSARY) AND DOUBLE-CLICK ON THE TEST WITH THE NAME CREATEINSERTSCUSTOMERANDSAVES TO OPEN THE SOURCE CODE. ............................................................................................................................................. 5
8.A. ..................................................................................................................................................................... 5
8.B.FIGURE ......................................................................................................................................................... 5
8.C.NAVIGATING TO A TEST METHOD THAT USES MSTEST .................................................................................... 5
9.IN THE CUSTOMERSCONTROLLERTEST.CS FILE THAT OPENS, NOTE THAT THE TEST METHOD USES THE EXPECTED TESTMETHOD ATTRIBUTE USED BY MSTEST TO MARK UNIT TESTS. ...................................................................................................................................... 6
9.A. ..................................................................................................................................................................... 6
9.B.FIGURE ......................................................................................................................................................... 6
9.C.MSTEST UNIT TEST CODE ............................................................................................................................... 6
10.IN THE SEARCH BOX AT THE TOP OF THE TEST EXPLORER WINDOW, ENTER “INDEX” AND NOTE THE AVAILABLE SEARCH FILTERS. ............... 6
10.A. ................................................................................................................................................................... 6
10.B.FIGURE ....................................................................................................................................................... 6
10.C.SEARCHING WITHIN TEST EXPLORER ............................................................................................................ 6
11.IN THE SEARCH RESULTS, DOUBLE-CLICK ON THE ONLY TEST LISTED WITH THE NAME INDEXRETURNSNONNULLVIEW TO OPEN THE SOURCE CODE. ......................................................................................................................................................................... 6
11.A. ................................................................................................................................................................... 6
11.B.FIGURE ....................................................................................................................................................... 6
11.C.NAVIGATING TO A TEST METHOD THAT USES XUNIT ..................................................................................... 6
12.IN THE HOMECONTROLLERTEST.CS FILE THAT OPENS, YOU CAN SEE THAT THE INDEXRETURNSNONNULLVIEW TEST ACTUALLY USES THE XUNIT TESTING FRAMEWORK. BOTH THE XUNIT FRAMEWORK AND THE TEST RUNNER THAT INTEGRATES WITH TEST EXPLORER CAN BE ADDED TO A PROJECT VIA NUGET. ................................................................................................................................................ 6
12.A. ................................................................................................................................................................... 6
12.B.FIGURE ....................................................................................................................................................... 6
13.SELECT X BUTTON TO CLEAR THE CURRENT SEARCH BOX IN TEST EXPLORER. .............................................................................. 6
13.A. ................................................................................................................................................................... 6
13.B.FIGURE ....................................................................................................................................................... 6
13.C.LOCATION OF X BUTTON TO CLEAR SEARCH BOX .......................................................................................... 7
14.NOTE THAT RESULTS ARE GROUPED BY TEST OUTCOME BY DEFAULT, WITH EXECUTION TIMES LISTED. ................................................ 7
14.A. ................................................................................................................................................................... 7
14.B.FIGURE ....................................................................................................................................................... 7
14.C.TEST RUN RESULTS ....................................................................................................................................... 7
15.TEST EXECUTION PERFORMANCE HAS BEEN GIVEN SOME ADDITIONAL ATTENTION IN RECENT VERSIONS. TAKE NOTE OF THE EXECUTION TIMES AND THEN SELECT THE RUN… BUTTON FOLLOWED BY THE REPEAT LAST RUN OPTION. NOTE THE DIFFERENCE IN EXECUTION TIME. ............... 7
15.A. ................................................................................................................................................................... 7
15.B.FIGURE ....................................................................................................................................................... 7
15.C.REPEATING THE LAST TEST RUN ................................................................................................................... 7
15.D.NOTE: THE EXECUTION TIMES THAT YOU SEE WILL BE DIFFERENT FROM THOSE SHOWN IN THE SCREENSHOTS. THE FIRST TIME YOU RUN UNIT TESTS THEY WILL EXECUTE MORE SLOWLY THAN ON SUBSEQUENTRUNS AS THE TESTS AND TESTING ENGINE ARE BEING LOADED FOR THE FIRST TIME. ........................................... 7
15.F. .................................................................................................................................................................... 7
15.G.FIGURE ....................................................................................................................................................... 7
15.H.SECOND TEST RUN WITHOUT CHANGES TO TESTS IS FASTER ........................................................................ 7
16.SINGLE-CLICK THE FAILED TEST NAMED “SCHEDULEACTIONCORRECTLYUPDATESREPOSITORIES” TO VIEW A SUMMARY OF TEST RESULTS. ..... 7
16.A. ................................................................................................................................................................... 7
16.B.FIGURE ....................................................................................................................................................... 7
16.C.SELECTING THE FAILED TEST ......................................................................................................................... 7
16.E.NOTE: YOU CAN ALSO RIGHT-CLICK ON TEST RESULTS AND USE THE COPY COMMAND TO COPY DETAILS TO THE CLIPBOARD. THIS WOULD BE USEFUL WHEN SENDING AN EMAIL, FOR EXAMPLE. ......................................... 7
17.THE SUMMARY VIEW OF THE FAILED TEST RUN SHOWS THAT AN ARGUMENTNULLEXCEPTION EXCEPTION OCCURRED DURING THE TEST RUN AND EVEN PROVIDES THE STACK TRACK AT THE TIME OF THE EXCEPTION. NOTE THAT WE CAN FOLLOW LINKS TO GO DIRECTLY TO THE SOURCE CODE FOR THE TEST OR TO POINTS WITHIN THE STACK TRACE. SELECT THE SOURCE LINK TO GO TO THE SOURCE CODE FOR THE TEST METHOD. . . 7
17.A. ................................................................................................................................................................... 8
17.B.FIGURE ....................................................................................................................................................... 8
17.C.NAVIGATE TO THE SOURCE OF THE TEST METHOD ........................................................................................ 8
17.E. ................................................................................................................................................................... 8
17.F.FIGURE ........................................................................................................................................................ 8
17.G.INITIAL VIEW AFTER NAVIGATING TO SOURCE OF THE TEST METHOD ........................................................... 8
18.FIND THE LINE OF CODE THAT IS COMMENTED OUT AND UN-COMMENT IT. ASSUME THAT THIS IS THE ROOT CAUSE FOR THE FAILED TEST. . . . 8
18.A. ................................................................................................................................................................... 8
18.B.FIGURE ....................................................................................................................................................... 8
18.C.FIXING THE ROOT PROBLEM OF THE FAILED TEST ......................................................................................... 8
19.PRESS CTRL+S TO SAVE YOUR CHANGES. ......................................................................................................................... 8 20.RIGHT-CLICK ON THE FAILED TEST IN TEST EXPLORER AND SELECT RUN SELECTED TESTS TO MAKE SURE THAT THE PROBLEM IS FIXED. ........ 8
20.A. ................................................................................................................................................................... 8
20.B.FIGURE ....................................................................................................................................................... 8
20.C.RUNNING THE FAILED TEST AGAIN TO VALIDATE FIX ..................................................................................... 8
21.SO FAR, WE HAVE SEEN HOW TO RUN ALL DISCOVERED TESTS, SEARCH FOR SPECIFIC TESTS, AND GROUP BY TEST OUTCOME. NOW LET’S TAKE AT A FEW OTHER WAYS THAT YOU CAN ORGANIZE AND NAVIGATE UNIT TESTS TO MAKE UNIT TESTING EASIER. FOR SOME MOTIVATIONAL CONTEXT, REMEMBER HOW LONG YOU NEEDED TO WAIT FOR THE CODED UI TEST TO COMPLETE, AND THEN IMAGINE THAT THERE ARE MANY MORE OF THOSE. IT WOULD BE NICE IF THE TESTS WERE ORGANIZED SUCH THAT YOU COULD EASILY AVOID KICKING OFF THOSE TESTS WITHOUT HAVING TO BE CAREFUL OF THE TESTS THAT YOU SELECT. ........................................................................................................... 8 22.TO START WITH, THERE ARE A NUMBER OF USEFUL GROUPING OPTIONS. RIGHT-CLICK SOMEWHERE WITHIN THE TEST EXPLORER WINDOW AND SELECT GROUP BY | CLASS. ....................................................................................................................................... 8
22.A. ................................................................................................................................................................... 8
22.B.FIGURE ....................................................................................................................................................... 8
22.C.GROUPING TESTS BY CLASS .......................................................................................................................... 8
23.ASSUMING THAT TESTS ARE WELL ORGANIZED WITHIN DESCRIPTIVE CLASS NAMES, THIS CAN MAKE IT MUCH EASIER TO SELECT AND RUN TESTS. FOR EXAMPLE, YOU COULD SELECT JUST THE TESTS FROM THE CUSTOMERSCONTROLLERTEST CLASS AND RUN THEM. ........................ 8
23.A. ................................................................................................................................................................... 9
23.B.FIGURE ....................................................................................................................................................... 9
23.C.GROUPING TESTS BY CLASS .......................................................................................................................... 9
24.RIGHT-CLICK SOMEWHERE WITHIN THE TEST EXPLORER WINDOW AND SELECT GROUP BY | PROJECT. .............................................. 9
24.A. ................................................................................................................................................................... 9
24.B.FIGURE ....................................................................................................................................................... 9
24.C.GROUPING TESTS BY PROJECT ...................................................................................................................... 9
25.GROUPING TESTS BY PROJECT WOULD OBVIOUSLY BE USEFUL FOR NAVIGATING AND RUNNING TESTS AT THE PROJECT LEVEL. .................... 9
25.A. ................................................................................................................................................................... 9
25.B.FIGURE ....................................................................................................................................................... 9
25.C.GROUPING TESTS BY PROJECT ...................................................................................................................... 9
26.YOU CAN ALSO USE TRAITS IN YOUR TEST CODE TO PROVIDE CUSTOM GROUPING. LET’S SAY THAT WE WANT TO GROUP ALL CODED UI TESTS TOGETHER. OPEN CODEDUITEST1.CS FROM THE FABRIKAMFIBER.WEB.UITESTS PROJECT. ................................................................ 9
26.A. ................................................................................................................................................................... 9
26.B.FIGURE ....................................................................................................................................................... 9
26.C.LOADING TEST CODE .................................................................................................................................... 9
27.UN-COMMENT THE CODEDUITEST ATTRIBUTE FROM THE TOP OF THE CODEDUITEST1 CLASS DEFINITION. ........................................ 9
27.A. ................................................................................................................................................................... 9
27.B.FIGURE ....................................................................................................................................................... 9
27.C.CODEDUITEST ATTRIBUTE ............................................................................................................................ 9
28.ADD THE TESTCATEGORY ATTRIBUTE TO THE CODEDUITESTMETHOD1 METHOD WITH A CATEGORY OF “UI”. .................................... 9
28.A. ................................................................................................................................................................... 9
28.B.FIGURE ..................................................................................................................................................... 10
28.C.ADDING CUSTOM TRAIT TO TEST METHOD ................................................................................................. 10
29.BUILD THE SOLUTION BY PRESSING CTRL+SHIFT+B. .......................................................................................................... 10 30.RIGHT-CLICK WITHIN TEST EXPLORER AND SELECT GROUP BY | TRAITS. ................................................................................. 10 31.WITH THE CODED UI TESTS CATEGORIZED APPROPRIATELY, IT IS NOW EASY TO SELECT AND RUN JUST THE UI TESTS IF DESIRED. .............. 10
31.A. ................................................................................................................................................................. 10
31.B.FIGURE ..................................................................................................................................................... 10
31.C.TEST EXPLORER WINDOW SHOWING TESTS GROUPED BY TRAITS ............................................................... 10
32.YOU CAN ALSO CREATE SETS OF TESTS CALLED PLAYLISTS. THIS PROVIDES CUSTOM GROUPING WITHOUT THE NEED TO MODIFY YOUR UNIT TESTING CODE. RIGHT-CLICK ON THE CODEDUITESTMETHOD1 TEST AND SELECT ADD TO PLAYLIST | NEW PLAYLIST. ............................. 10
32.A. ................................................................................................................................................................. 10
32.B.FIGURE ..................................................................................................................................................... 10
32.C.CREATING A NEW TEST PLAYLIST ................................................................................................................ 10
33.ENTER “UI TESTS” FOR THE PLAYLIST FILE NAME AND THEN SELECT SAVE. .............................................................................. 10
33.A. ................................................................................................................................................................. 10
33.B.FIGURE ..................................................................................................................................................... 10
33.C.SAVING A NEW TEST PLAYLIST .................................................................................................................... 10
34.SELECT THE PLAYLIST DROP-DOWN AND THEN SELECT THE “UI TESTS” PLAYLIST. ....................................................................... 10
34.A. ................................................................................................................................................................. 10
34.B.FIGURE ..................................................................................................................................................... 10
34.C.SELECTING A TEST PLAYLIST ........................................................................................................................ 10
35.WITH JUST THE “UI TESTS” PLAYLIST SELECTED, YOU WILL ONLY SEE THOSE SPECIFIC TESTS SHOWN IN TEST EXPLORER, MAKING IT MUCH EASIER TO FOCUS ON A PARTICULAR SELECTION OF TESTS. ........................................................................................................ 10
35.A. ................................................................................................................................................................. 11
35.B.FIGURE ..................................................................................................................................................... 11
35.C.TEST PLAYLIST SELECTED ............................................................................................................................. 11
36.TEST PLAYLISTS ARE SIMPLE XML FILES THAT DEFINE THE INDIVIDUAL TESTS TO INCLUDE. FOR EXAMPLE, HERE IS WHAT THE “UI TESTS” XML LOOKS LIKE (LOADED IN VISUAL STUDIO EDITOR). .......................................................................................................... 11
36.A. ................................................................................................................................................................. 11
36.B.FIGURE ..................................................................................................................................................... 11
36.C.CONTENTS OF A TEST PLAYLIST ................................................................................................................... 11
37.PLAYLISTS CAN BE SHARED WITH TEAM MEMBERS VIA A TEAM WEBSITE, EMAILED, AND EVEN ADDED TO SOURCE CONTROL IF DESIRED. TO LOAD A PLAYLIST FILE, YOU WOULD SELECT THE PLAYLIST DROP-DOWN AND THEN SELECT THE OPEN PLAYLIST FILE OPTION. YOU DO NOT NEED TO DO THIS FOR THIS LAB. ............................................................................................................................................... 11
37.A. ................................................................................................................................................................. 11
37.B.FIGURE ..................................................................................................................................................... 11
37.C.OPENING A PLAYLIST .................................................................................................................................. 11
EXERCISE 2: CODE COVERAGE ............................................................................................................................ 11
1.IN THIS EXERCISE, YOU WILL LEARN ABOUT CODE COVERAGE FEATURES THAT MAKE IT EASIER TO USE AND INTEGRATE INTO THE DEVELOPMENT CYCLE. ...................................................................................................... 11
2.YOU SHOULD ALREADY HAVE THE TEST EXPLORER WINDOW OPEN FROM THE PREVIOUS EXERCISE, BUT GO AHEAD AND OPEN IT IF NECESSARY (FROM TEST | WINDOWS | TEST EXPLORER). ..................................................................................................................... 11 3.RETURN TO THE DEFAULT PLAYLIST THAT INCLUDES ALL TESTS BY SELECTING THE PLAYLIST DROP-DOWN AND THEN SELECTING THE ALL TESTS OPTION. .................................................................................................................................................................... 11
3.A. ................................................................................................................................................................... 11
3.B.FIGURE ....................................................................................................................................................... 11
3.C.SELECTING THE ALL TESTS PLAYLIST .............................................................................................................. 11
4.TO ANALYZE CODE COVERAGE FOR ALL TESTS, SELECT THE RUN DROP-DOWN AND THEN SELECT THE ANALYZE CODE COVERAGE FOR ALL TESTS OPTION. THIS WILL INITIATE THE PROCESS OF BUILDING, TESTING, AND GATHERING CODE COVERAGE RESULTS. ........................................ 11
4.A. ................................................................................................................................................................... 11
4.B.FIGURE ....................................................................................................................................................... 11
4.C.ANALYZING CODE COVERAGE ....................................................................................................................... 11
5.YOU CAN VIEW THE RESULTS IN THE CODE COVERAGE RESULTS WINDOWS TO GET AN IDEA OF THE COVERED/NOT COVERED STATISTICS FOR ALL OF THE TESTS. IN THE SCREENSHOT BELOW, NOTE THAT COVERAGE IS MEASURED IN BLOCKS OF CODE BY DEFAULT, WHERE A BLOCK IS CODE WITH EXACTLY ONE ENTRY AND EXIT POINT. .......................................................................................................................... 12
5.A. ................................................................................................................................................................... 12
5.B.FIGURE ....................................................................................................................................................... 12
5.C.CODE COVERAGE RESULTS SUMMARY .......................................................................................................... 12
5.E.NOTE: IF YOU WOULD LIKE TO SEE THE RESULTS IN TERMS OF LINES, YOU CAN DO THAT BY RIGHT-CLICKING IN THE CODE COVERAGE RESULTS WINDOW AND SELECTING THE ADD/REMOVE COLUMNS OPTION. ................. 12
6.EXPAND THE ROOT NODE OF THE CODE COVERAGE RESULT TO VIEW THE COVERAGE BROKEN DOWN BY ASSEMBLY. BY DEFAULT, WE SEE ALL ASSEMBLIES THAT ARE LOADED DURING THE TEST RUN (AND FOR WHICH A .PDB FILE IS AVAILABLE). ..................................................... 12
6.A. ................................................................................................................................................................... 12
6.B.FIGURE ....................................................................................................................................................... 12
6.C.CODE COVERAGE RESULTS BY ASSEMBLY ...................................................................................................... 12
6.D.NOTE: YOU CAN CUSTOMIZE CONTROL OVER WHICH ASSEMBLIES ARE SELECTED FOR CODE COVERAGE ANALYSIS BY WRITING A .RUNSETTINGS FILE. FOR MORE INFORMATION, SEE THE MSDN ARTICLE CUSTOMIZING CODE COVERAGE ANALYSIS. .............................................................................................................................. 12
7.EXPAND THE FABRIKAMFIBER.WEB.DLL NODE TO VIEW THE COVERAGE BROKEN DOWN BY NAMESPACE. THIS ALLOWS US TO SEE THAT, ALTHOUGH WE DO HAVE SOME TEST COVERAGE OF THE CONTROLLER CLASSES, THE TEST TEAM HAS A LOT OF WORK TO DO IN ORDER TO PROVIDE COVERAGE TO THE OTHER NAMESPACES. ............................................................................................................................. 12
7.A. ................................................................................................................................................................... 12
7.B.FIGURE ....................................................................................................................................................... 12
7.C.CODE COVERAGE RESULTS FOR NAMESPACES WITHIN ASSEMBLY ................................................................. 12
8.EXPAND THE FABRIKAMFIBER.WEB.CONTROLLERS NAMESPACE NODE TO VIEW THE COVERAGE BROKEN DOWN BY CLASS. THIS SHOWS THAT THE HOMECONTROLLER CLASS IS COVERED WELL AND THAT THE EMPLOYEESCONTROLLER CURRENTLY HAS NO COVERAGE. ........................ 12
8.A. ................................................................................................................................................................... 12
8.B.FIGURE ....................................................................................................................................................... 12
8.C.CODE COVERAGE RESULTS FOR CLASSES WITHIN NAMESPACE ...................................................................... 12
9.FINALLY, LET’S DRILL DOWN INTO THE CLASS NODES TO SEE COVERAGE DOWN TO THE METHOD LEVEL BY EXPANDING THE CUSTOMERSCONTROLLER CLASS NODE. .............................................................................................................................. 12
9.A. ................................................................................................................................................................... 13
9.B.FIGURE ....................................................................................................................................................... 13
9.C.CODE COVERAGE RESULTS FOR METHODS .................................................................................................... 13
10.DOUBLE-CLICK ON THE CREATE(FABRIKAMFIBER.DAL.MODELS.CUSTOMER) CONSTRUCTOR TO NAVIGATE TO THE SOURCE CODE TO VISUALIZE THE BLOCK COVERAGE. ...................................................................................................................................... 13
10.A. ................................................................................................................................................................. 13
10.B.FIGURE ..................................................................................................................................................... 13
10.C.NAVIGATING TO SOURCE ........................................................................................................................... 13
11.IN THE EDITOR WINDOW FOR SOURCECONTROLLER.CS, YOU CAN SEE THAT THE CODE HIGHLIGHTED BLUE REPRESENTS THE BLOCK THAT WAS COVERED BY TESTS WHEREAS THE RED REPRESENTS THE BLOCK THAT WAS NOT COVERED. .................................................................. 13
11.A. ................................................................................................................................................................. 13
11.B.FIGURE ..................................................................................................................................................... 13
11.C.SOURCE VIEW SHOWING CODE COVERAGE COLORING ............................................................................... 13
12.IT IS ALSO POSSIBLE TO GET CODE COVERAGE FOR A SPECIFIC SELECTION OF TESTS. IN THE TEST EXPLORER WINDOW, SELECT THE CREATEINSERTSCUSTOMERANDSAVES TEST METHOD, RIGHT-CLICK, AND THEN SELECT ANALYZE CODE COVERAGE FOR SELECTED TESTS. ....... 13
12.A. ................................................................................................................................................................. 13
12.B.FIGURE ..................................................................................................................................................... 13
12.C.ANALYZE CODE COVERAGE FOR SPECIFIED TESTS ........................................................................................ 13
13.ONCE TEST EXECUTION COMPLETES FOR THE SELECTED TEST, EXPAND THE CODE COVERAGE NODE AND NOTE THAT ONLY THE ASSEMBLIES LOADED DURING TEST EXECUTION ARE SHOWN WITH STATISTICS. ................................................................................................ 13
13.A. ................................................................................................................................................................. 13
13.B.FIGURE ..................................................................................................................................................... 13
13.C.CODE COVERAGE RESULTS .......................................................................................................................... 13
14.IT IS ALSO EASY TO NAVIGATE BETWEEN DIFFERENT CODE COVERAGE RESULTS BY USING THE DROP-DOWN IN THE CODE COVERAGE RESULTS WINDOW. GO AHEAD AND SELECT THE FIRST CODE COVERAGE RESULTS FILE. ................................................................................. 13
14.A. ................................................................................................................................................................. 13
14.B.FIGURE ..................................................................................................................................................... 14
14.C.SELECTING SPECIFIC CODE COVERAGE RESULTS .......................................................................................... 14
15.LET’S SAY THAT WE WANT TO USE THESE CODE COVERAGE RESULTS IN A REPORT OR SIMPLY SHARE THEM EXTERNALLY. TO DO THAT, CLICK ON THE EXPORT RESULTS BUTTON. ........................................................................................................................................ 14
15.A. ................................................................................................................................................................. 14
15.B.FIGURE ..................................................................................................................................................... 14
15.C.EXPORTING CODE COVERAGE RESULTS ....................................................................................................... 14
16.IN THE SAVE COVERAGE DATA AS XML WINDOW, YOU COULD SAVE THE CODE COVERAGE DATA TO AN XML FILE, BUT FOR THE PURPOSES OF THIS LAB, GO AHEAD AND SIMPLY CANCEL OUT OF THE DIALOG AND CONTINUE ON TO THE NEXT EXERCISE. ....................................... 14
16.A. ................................................................................................................................................................. 14
16.B.FIGURE ..................................................................................................................................................... 14
16.C.SAVING CODE COVERAGE RESULTS TO FILE ................................................................................................. 14
EXERCISE 3: CODE CLONE ANALYSIS ................................................................................................................... 14
1.IN THIS EXERCISE, YOU WILL LEARN ABOUT THE CODE CLONE ANALYSIS FEATURE THAT LOOKS FOR SEMANTICALLY SIMILAR CODE USING A HEURISTIC SEARCH TECHNIQUE, RATHER THAN SIMPLY SEARCHING FOR EXACT MATCHES. .............................................................................................................................................. 14
2.SELECT ANALYZE | ANALYZE SOLUTION FOR CODE CLONES FROM THE MAIN MENU IN VISUAL STUDIO. ............................................ 14
2.A. ................................................................................................................................................................... 14
2.B.FIGURE ....................................................................................................................................................... 14
2.C.ANALYZE SOLUTION FOR CODE CLONES ........................................................................................................ 14
3.ONCE THE ANALYSIS IS COMPLETE, THE CODE CLONE ANALYSIS RESULTS WINDOW WILL SHOW CLONE CANDIDATES GROUPED BY MATCH STRENGTH. EXPAND THE STRONG MATCH GROUP TO EXPOSE THE TWO FILES THAT HAD A STRONG MATCH. ........................................... 14
3.A. ................................................................................................................................................................... 14
3.B.FIGURE ....................................................................................................................................................... 14
3.C.VIEWING THE STRONG FILE MATCHES .......................................................................................................... 14
4.EACH LINE SHOWS THE CLASS AND METHOD, SPECIFIC FILE, AND LINES THAT WERE DETERMINED TO BE STRONG MATCHES. IF YOU MOUSE OVER EACH MATCH, A SNIPPET OF CODE GIVES YOU QUICK GLANCE AT SOME OF THE MATCHING CODE. ................................................. 15
4.A. ................................................................................................................................................................... 15
4.B.FIGURE ....................................................................................................................................................... 15
4.C.MOUSE OVER RESULTS TO SEE DETAILS ........................................................................................................ 15
5.DOUBLE-CLICK ON EACH MATCH TO OPEN THEM IN CODE EDITOR WINDOWS, AND THEN RIGHT-CLICK ON THE TITLE TAB FOR ONE OF THEM AND SELECT NEW HORIZONTAL TAB GROUP FROM THE CONTEXT MENU. ..................................................................................... 15
5.A. ................................................................................................................................................................... 15
5.B.FIGURE ....................................................................................................................................................... 15
5.C.OPENING BOTH FILES FOR COMPARISON ..................................................................................................... 15
6.SCROLL THROUGH THE CODE TO LOCATE THE ASSIGNSCHEDULE METHOD FOR EACH FILE AND NOTE THAT IT IS IDENTICAL EXCEPT FOR THE LAST LINE THAT CALLS THE REDIRECTTOACTION METHOD. THIS INDICATES THAT THIS METHOD IS A GOOD CANDIDATE FOR REFACTORING. THIS KIND OF BROAD SEARCH IS PARTICULARLY USEFUL WHEN LOOKING FOR CODE THAT CAN BE REFACTORED FOR EASIER MAINTENANCE IN THE FUTURE. . 15
6.A. ................................................................................................................................................................... 15
6.B.FIGURE ....................................................................................................................................................... 15
6.C.COMPARING THE STRONG MATCH FROM THE CODE CLONE ANALYSIS .......................................................... 15
7.SELECT WINDOW | CLOSE ALL DOCUMENTS FROM THE MAIN MENU TO CLEAR UP SOME SCREEN REAL ESTATE SPACE. .......................... 15 8.YOU CAN ALSO NARROW THE FOCUS OF THE SEARCH FOR CODE CLONES IF DESIRED. IN SOLUTION EXPLORER, NAVIGATE TO CONTROLLERS FOLDER OF THE FABRIKAMFIBER.WEB PROJECT AND OPEN CUSTOMERSCONTROLLER.CS IN THE CODE EDITOR. ........................................ 15
8.A. ................................................................................................................................................................... 15
8.B.FIGURE ....................................................................................................................................................... 15
8.C.OPENING THE CUSTOMERSCONTROLLER.CS SOURCE FILE ............................................................................. 15
9.SCROLL DOWN TO THE CREATE METHOD THAT TAKES A CUSTOMER PARAMETER AND SELECT THE THREE LINES OF CODE WITHIN THE FIRST “IF” STATEMENT. ................................................................................................................................................................ 15
9.A. ................................................................................................................................................................... 15
9.B.FIGURE ....................................................................................................................................................... 15
9.C.SELECTING CODE FOR A CODE CLONE ANALYSIS ........................................................................................... 15
10.RIGHT-CLICK ON THE SELECTED LINES OF CODE AND THEN SELECT THE FIND MATCHING CLONES IN SOLUTION OPTION FROM THE CONTEXT MENU. ...................................................................................................................................................................... 16
10.A. ................................................................................................................................................................. 16
10.B.FIGURE ..................................................................................................................................................... 16
10.C.FINDING MATCHING CLONES FOR THE SELECTED CODE ............................................................................... 16
11.AFTER THE SEARCH IS COMPLETE, THE CODE CLONE SEARCH RESULTS WINDOW SHOWS SNIPPET MATCHES OF VARYING STRENGTH. ........ 16
11.A. ................................................................................................................................................................. 16
11.B.FIGURE ..................................................................................................................................................... 16
11.C.RESULTS OF CODE CLONE SEARCH .............................................................................................................. 16
12.EXPAND ALL OF THE CLONE GROUPS TO EXPOSE THE DISCOVERED MATCHES, INCLUDING THE ORIGINAL CLONE GROUP. YOU MAY WANT TO INCREASE THE SIZE OF THE CODE CLONE SEARCH RESULTS WINDOW SO THAT YOU CAN SEE ALL MATCHES AT A GLANCE. ........................... 16
12.A. ................................................................................................................................................................. 16
12.B.FIGURE ..................................................................................................................................................... 16
12.C.CODE CLONE RESULTS WITH GROUPS EXPANDED ....................................................................................... 16
13.HOLD THE MOUSE CURSOR OVER THE ORIGINAL CODE SNIPPET TO REMIND OURSELVES WHAT THE CODE CLONES ARE BEING COMPARED TO. ............................................................................................................................................................................... 16
13.A. ................................................................................................................................................................. 16
13.B.FIGURE ..................................................................................................................................................... 16
13.C.ORIGINAL CODE SNIPPET ........................................................................................................................... 16
14.HOLD THE MOUSE CURSOR OVER THE EXACT MATCH RESULT AND NOTE THAT THE EDIT METHOD USES EXACTLY THE SAME CODE AS THE CODE FROM THE CREATE METHOD. .................................................................................................................................... 16 15.HOLD THE MOUSE CURSOR OVER THE STRONG MATCH RESULT AND NOTE THAT THE ONLY DIFFERENCE IS THAT THE FIRST LINE IS A CALL TO A DELETE METHOD. ......................................................................................................................................................... 16
15.A. ................................................................................................................................................................. 16
15.B.FIGURE ..................................................................................................................................................... 16
15.C.CODE SNIPPET RESULT WITH STRONG MATCH TO ORIGINAL ....................................................................... 16
16.HOLD THE MOUSE CURSOR OVER THE FIRST MEDIUM MATCH RESULT AND NOTE THAT THE SNIPPET IS SIMILAR TO THE ORIGINAL, BUT NOW WE ARE WORKING WITH AN ENTIRELY DIFFERENT OBJECT (NOW EMPLOYEEREPOSITORY). .................................................................. 16
16.A. ................................................................................................................................................................. 16
16.B.FIGURE ..................................................................................................................................................... 17
16.C.CODE SNIPPET RESULT WITH STRONG MATCH TO ORIGINAL ....................................................................... 17
17.IN SUMMARY, THERE ARE THREE MAIN SCENARIOS WHERE THE IDENTIFICATION OF CODE CLONES MAY BE USEFUL TO DEVELOPERS: .......... 17 17.a.Identification of candidates for code refactoring ........................................................................................ 1717.b.Bug fixes or enhancements are made in code and the developer wants to see if there are other similar locations that should be updated ........................................................................................................................ 1717.c.As a learning tool for a new developer starting work on a new team project, e.g. the developer adds code to update a customer record and wants to see if there are practices used by the rest of the codebase such as using a Try… Catch block ...................................................................................................................................... 17
1.
Overview
1. In this lab, you will learn about how the unit testing engine in Visual Studio is extensible and opens the door for 3rd party unit testing adapters such as NUnit and xUnit.net. In addition, you will see some of the improvements made to code coverage support. You will also learn about how the code clone capability goes hand-in-hand with delivering high quality software by helping you identify blocks of semantically similar code which may be candidates for common bug fixes or refactoring.
Prerequisites
1. In order to complete this lab you will need the Visual Studio 2015 virtual machine provided by Microsoft. For more information on acquiring and using this virtual machine, please see this blog post.
About the Fabrikam Fiber Scenario
1. This set of hands-on-labs uses a fictional company, Fabrikam Fiber, as a backdrop to the scenarios you are learning about. Fabrikam Fiber provides cable television and related services to the United States. They are growing rapidly and have embraced Windows Azure to scale their customer-facing web site directly to end-users to allow them to self-service tickets and track technicians. They also use an on-premises ASP.NET MVC application for their customer service representatives to administer customer orders.
In this set of hands-on labs, you will take part in a number of scenarios that involve the development andtesting team at Fabrikam Fiber. The team, which consists of 8-10 people, has decided to use Visual Studioapplication lifecycle management tools to manage their source code, run their builds, test their web sites, and plan and track the project.
Exercises
1. This hands-on lab includes the following exercises:
2. Unit Testing
3. Code Coverage
4. Code Clone Analysis
1. Estimated time to complete this lab: 30 minutes.
Exercise 1: Unit Testing
1. In this exercise, you will learn about the unit testing story in Visual Studio 2015.
2. Log in as Julia (VSALM\Julia). All user passwords are P2ssw0rd.
3. Launch Visual Studio 2015 from the taskbar and open Team Explorer. You should now be connected to the FabrikamFiber team project. If you are not automatically connected to the
FabrikamFiber project, select the Connect to Team Projects button ( ) to do so.
4. In Team Explorer – Home, double-click on the first FabrikamFiber.CallCenter.sln solution.
4.a.4.b. Figure 14.c. Loading Fabrikam Fiber solution
4.d.
5. Press Ctrl + Shift + B to build the solution.
6. Open the Test Explorer window from Test | Windows | Test Explorer. Note that discovered tests are initially set to the Not Run state.
6.a.
6.b. Figure 26.c. Test Explorer window
6.d.
7. Select the Run All link to execute all discovered unit tests.
7.a.7.b. Figure 37.c. Running all unit tests
7.d.
7.e. Note: Visual Studio also provides Continuous Test Runner functionality that can be enabled with the Test | Test Settings | Run Tests After Build option. With this option selected, tests will always run after builds.
8. Expand the Passed Tests group (if necessary) and double-click on the test with the name CreateInsertsCustomerAndSaves to open the source code.
8.a.8.b. Figure 48.c. Navigating to a test method that uses MSTest
8.d.
9. In the CustomersControllerTest.cs file that opens, note that the test method uses the expected TestMethod attribute used by MSTest to mark unit tests.
9.a.9.b. Figure 59.c. MSTest unit test code
9.d.
Note: The test status indicator just above the method definition is a CodeLens indicator. It tells us that the last run was a success. You can learn more about the CodeLens feature in the “Collaboration Experiences for Development Teams using Team Foundation Server 2015” lab.
10. In the Search box at the top of the Test Explorer window, enter “index” and note the available search filters.
10.a.10.b. Figure 610.c. Searching within Test Explorer
10.d.
11. In the search results, double-click on the only test listed with the name IndexReturnsNonNullView to open the source code.
11.a.11.b. Figure 711.c. Navigating to a test method that uses XUnit
11.d.
12. In the HomeControllerTest.cs file that opens, you can see that the IndexReturnsNonNullView test actually uses the XUnit testing framework. Both the xUnit framework and the test runner that integrates with Test Explorer can be added to a project via NuGet.
12.a.12.b. Figure 8Example of a XUnit test in Visual Studio
12.c.
13. Select X button to clear the current search box in Test Explorer.
13.a.13.b. Figure 913.c. Location of X button to clear search box
13.d.
14. Note that results are grouped by test outcome by default, with execution times listed.
14.a.14.b. Figure 1014.c. Test run results
14.d.
15. Test execution performance has been given some additional attention in recent versions. Take note of the execution times and then select the Run… button followed by the Repeat Last Run option. Note the difference in execution time.
15.a.
15.b. Figure 1115.c. Repeating the last test run
15.d. Note: The execution times that you see will be different from those shown in the screenshots. The first time you run unit tests they will execute more slowly than on subsequent runs as the tests and testing engine are being loaded for the first time.
15.e.
15.f.15.g. Figure 1215.h. Second test run without changes to tests is faster
15.i.
16. Single-click the failed test named “ScheduleActionCorrectlyUpdatesRepositories” to view asummary of test results.
16.a.16.b. Figure 1316.c. Selecting the failed test
16.d.
16.e. Note: You can also right-click on test results and use the Copy command to copy details to the clipboard. This would be useful when sending an email, for example.
16.f.
17. The summary view of the failed test run shows that an ArgumentNullException exception occurred during the test run and even provides the stack track at the time of the exception. Note that we can follow links to go directly to the source code for the test or to points within thestack trace. Select the source link to go to the source code for the test method.
17.a.17.b. Figure 1417.c. Navigate to the source of the test method
17.d.
17.e.
17.f. Figure 1517.g. Initial view after navigating to source of the test method
17.h.
18. Find the line of code that is commented out and un-comment it. Assume that this is the root cause for the failed test.
18.a.
18.b. Figure 1618.c. Fixing the root problem of the failed test
18.d.
19. Press Ctrl+S to save your changes.
20. Right-click on the failed test in Test Explorer and select Run Selected Tests to make sure that the problem is fixed.
20.a.20.b. Figure 1720.c. Running the failed test again to validate fix
20.d.
21. So far, we have seen how to run all discovered tests, search for specific tests, and group by test outcome. Now let’s take at a few other ways that you can organize and navigate unit tests tomake unit testing easier. For some motivational context, remember how long you needed to wait for the coded UI test to complete, and then imagine that there are many more of those. It would be nice if the tests were organized such that you could easily avoid kicking off those tests without having to be careful of the tests that you select.
22. To start with, there are a number of useful grouping options. Right-click somewhere within the Test Explorer window and select Group By | Class.
22.a.
22.b. Figure 1822.c. Grouping tests by class
22.d.
23. Assuming that tests are well organized within descriptive class names, this can make it much easier to select and run tests. For example, you could select just the tests from the CustomersControllerTest class and run them.
23.a.23.b. Figure 1923.c. Grouping tests by class
23.d.
24. Right-click somewhere within the Test Explorer window and select Group By | Project.
24.a.24.b. Figure 2024.c. Grouping tests by project
24.d.
25. Grouping tests by project would obviously be useful for navigating and running tests at the project level.
25.a.25.b. Figure 2125.c. Grouping tests by project
25.d.
26. You can also use traits in your test code to provide custom grouping. Let’s say that we want to group all coded UI tests together. Open CodedUITest1.cs from the FabrikamFiber.Web.UITests project.
26.a.26.b. Figure 2226.c. Loading test code
26.d.
27. Un-comment the CodedUITest attribute from the top of the CodedUITest1 class definition.
27.a.27.b. Figure 2327.c. CodedUITest attribute
27.d.
28. Add the TestCategory attribute to the CodedUITestMethod1 method with a category of “UI”.
28.a.
28.b. Figure 2428.c. Adding custom trait to test method
28.d.
29. Build the solution by pressing Ctrl+Shift+B.
30. Right-click within Test Explorer and select Group By | Traits.
31. With the coded UI tests categorized appropriately, it is now easy to select and run just the UI tests if desired.
31.a.31.b. Figure 2531.c. Test Explorer window showing tests grouped by traits
31.d.
32. You can also create sets of tests called playlists. This provides custom grouping without the need to modify your unit testing code. Right-click on the CodedUITestMethod1 test and select Add to Playlist | New Playlist.
32.a.
32.b. Figure 2632.c. Creating a new test playlist
32.d.
33. Enter “UI Tests” for the playlist file name and then select Save.
33.a.
33.b. Figure 2733.c. Saving a new test playlist
33.d.
34. Select the Playlist drop-down and then select the “UI Tests” playlist.
34.a.34.b. Figure 2834.c. Selecting a test playlist
34.d.
35. With just the “UI Tests” playlist selected, you will only see those specific tests shown in Test Explorer, making it much easier to focus on a particular selection of tests.
35.a.35.b. Figure 2935.c. Test playlist selected
35.d.
36. Test playlists are simple XML files that define the individual tests to include. For example, here is what the “UI Tests” XML looks like (loaded in Visual Studio editor).
36.a.
36.b. Figure 3036.c. Contents of a test playlist
36.d.
37. Playlists can be shared with team members via a team website, emailed, and even added tosource control if desired. To load a playlist file, you would select the Playlist drop-down and thenselect the Open Playlist File option. You do not need to do this for this lab.
37.a.37.b. Figure 31
37.c. Opening a playlist
Exercise 2: Code Coverage
1. In this exercise, you will learn about code coverage features that make it easier to use and integrate into the development cycle.
2. You should already have the Test Explorer window open from the previous exercise, but go ahead and open it if necessary (from Test | Windows | Test Explorer).
3. Return to the default playlist that includes all tests by selecting the Playlist drop-down and then selecting the All Tests option.
3.a.3.b. Figure 323.c. Selecting the All Tests playlist
3.d.
4. To analyze code coverage for all tests, select the Run drop-down and then select the Analyze Code Coverage for All Tests option. This will initiate the process of building, testing, andgathering code coverage results.
4.a.
4.b. Figure 334.c. Analyzing code coverage
4.d.
5. You can view the results in the Code Coverage Results windows to get an idea of the Covered/Not Covered statistics for all of the tests. In the screenshot below, note that coverage ismeasured in blocks of code by default, where a block is code with exactly one entry and exit point.
5.a.
5.b. Figure 345.c. Code coverage results summary
5.d.
5.e. Note: If you would like to see the results in terms of lines, you can do that by right-clicking in the Code Coverage Results window and selecting the Add/Remove Columns option.
6. Expand the root node of the code coverage result to view the coverage broken down by assembly. By default, we see all assemblies that are loaded during the test run (and for which a .pdb file is available).
6.a.
6.b. Figure 356.c. Code coverage results by assembly
6.d. Note: You can customize control over which assemblies are selected for code coverage analysis by writing a .runsettings file. For more information, see the MSDN article Customizing Code Coverage Analysis.
7. Expand the fabrikamfiber.web.dll node to view the coverage broken down by namespace. This allows us to see that, although we do have some test coverage of the controller classes, the test team has a lot of work to do in order to provide coverage to the other namespaces.
7.a.
7.b. Figure 367.c. Code coverage results for namespaces within assembly
7.d.
8. Expand the FabrikamFiber.Web.Controllers namespace node to view the coverage broken down by class. This shows that the HomeController class is covered well and that the EmployeesController currently has no coverage.
8.a.
8.b. Figure 378.c. Code coverage results for classes within namespace
8.d.
9. Finally, let’s drill down into the class nodes to see coverage down to the method level by expanding the CustomersController class node.
9.a.
9.b. Figure 389.c. Code coverage results for methods
9.d.
10. Double-click on the Create(FabrikamFiber.DAL.Models.Customer) constructor to navigate to the source code to visualize the block coverage.
10.a.10.b. Figure 3910.c. Navigating to source
10.d.
11. In the editor window for SourceController.cs, you can see that the code highlighted blue represents the block that was covered by tests whereas the red represents the block that was not covered.
11.a.11.b. Figure 4011.c. Source view showing code coverage coloring
11.d.
12. It is also possible to get code coverage for a specific selection of tests. In the Test Explorer window, select the CreateInsertsCustomerAndSaves test method, right-click, and then select Analyze Code Coverage for Selected Tests.
12.a.12.b. Figure 4112.c. Analyze code coverage for specified tests
12.d.
13. Once test execution completes for the selected test, expand the code coverage node and note that only the assemblies loaded during test execution are shown with statistics.
13.a.
13.b. Figure 4213.c. Code coverage results
13.d.
14. It is also easy to navigate between different code coverage results by using the drop-down in the Code Coverage Results window. Go ahead and select the first code coverage results file.
14.a.14.b. Figure 4314.c. Selecting specific code coverage results
14.d.
15. Let’s say that we want to use these code coverage results in a report or simply share them externally. To do that, click on the Export Results button.
15.a.15.b. Figure 4415.c. Exporting code coverage results
15.d.
16. In the Save Coverage Data as XML window, you could save the code coverage data to an XML file, but for the purposes of this lab, go ahead and simply Cancel out of the dialog and continue on to the next exercise.
16.a.
16.b. Figure 4516.c. Saving code coverage results to file
Exercise 3: Code Clone Analysis
1. In this exercise, you will learn about the Code Clone analysis feature that looks for semantically similar code using a heuristic search technique, rather than simply searching for exact matches.
2. Select Analyze | Analyze Solution for Code Clones from the main menu in Visual Studio.
2.a.2.b. Figure 462.c. Analyze solution for code clones
2.d.
3. Once the analysis is complete, the Code Clone Analysis Results window will show clone candidates grouped by match strength. Expand the Strong Match group to expose the two files that had a strong match.
3.a.3.b. Figure 473.c. Viewing the strong file matches
3.d.
4. Each line shows the class and method, specific file, and lines that were determined to be strong matches. If you mouse over each match, a snippet of code gives you quick glance at some of the matching code.
4.a.
4.b. Figure 484.c. Mouse over results to see details
4.d.
5. Double-click on each match to open them in code editor windows, and then right-click on the title tab for one of them and select New Horizontal Tab Group from the context menu.
5.a.5.b. Figure 495.c. Opening both files for comparison
5.d.
6. Scroll through the code to locate the AssignSchedule method for each file and note that it is identical except for the last line that calls the RedirectToAction method. This indicates that this method is a good candidate for refactoring. This kind of broad search is particularly useful when looking for code that can be refactored for easier maintenance in the future.
6.a.
6.b. Figure 506.c. Comparing the strong match from the code clone analysis
6.d.
7. Select Window | Close All Documents from the main menu to clear up some screen real estate space.
8. You can also narrow the focus of the search for code clones if desired. In Solution Explorer, navigate to Controllers folder of the FabrikamFiber.Web project and open CustomersController.cs in the code editor.
8.a.8.b. Figure 518.c. Opening the CustomersController.cs source file
8.d.
9. Scroll down to the Create method that takes a Customer parameter and select the three lines of code within the first “if” statement.
9.a.9.b. Figure 529.c. Selecting code for a code clone analysis
9.d.
10. Right-click on the selected lines of code and then select the Find Matching Clones in Solution option from the context menu.
10.a.10.b. Figure 5310.c. Finding matching clones for the selected code
10.d.
11. After the search is complete, the Code Clone Search Results window shows snippet matches of varying strength.
11.a.11.b. Figure 5411.c. Results of code clone search
11.d.
12. Expand all of the clone groups to expose the discovered matches, including the original clone group. You may want to increase the size of the Code Clone Search Results window so thatyou can see all matches at a glance.
12.a.12.b. Figure 5512.c. Code clone results with groups expanded
12.d.
13. Hold the mouse cursor over the original code snippet to remind ourselves what the code clones are being compared to.
13.a.13.b. Figure 5613.c. Original code snippet
14. Hold the mouse cursor over the Exact Match result and note that the Edit method uses exactly the same code as the code from the Create method.
15. Hold the mouse cursor over the Strong Match result and note that the only difference is that the first line is a call to a Delete method.
15.a.15.b. Figure 5715.c. Code snippet result with strong match to original
16. Hold the mouse cursor over the first Medium Match result and note that the snippet is similar to the original, but now we are working with an entirely different object (now employeeRepository).
16.a.16.b. Figure 5816.c. Code snippet result with strong match to original
17. In summary, there are three main scenarios where the identification of code clones may be useful to developers:
17.a. Identification of candidates for code refactoring
17.b. Bug fixes or enhancements are made in code and the developer wants to see if there are other similar locations that should be updated
17.c. As a learning tool for a new developer starting work on a new team project, e.g. the developer adds code to update a customer record and wants to see if there are practices used by the rest of the codebase such as using a Try… Catch block