advanced testing and continuous integration...all tests run in single app launch tests launch app...
TRANSCRIPT
© 2016 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.
Developer Tools #WWDC16
Session 409
Advanced Testing andContinuous Integration
Zoltan Foley-Fisher Xcode EngineerEric Dudiak Xcode Engineer
Agenda
Agenda
Testing Concepts
Agenda
Testing Concepts
Xcode• Crash Log Gathering
Agenda
Testing Concepts
Xcode• Crash Log Gathering
Xcode Server• Advanced Triggers• Issue Tracking and Blame• Configurable Integration User
Agenda
Testing Concepts
Xcode• Crash Log Gathering
Xcode Server• Advanced Triggers• Issue Tracking and Blame• Configurable Integration User
xcodebuild
Overview
Overview
Testing frameworkXCTest
Testing frameworkXCTest
Sources
TestClassA.m
TestClassB.swift
TestClassC.swift
Testing frameworkXCTest
Compile
Build Products
UI.xctest
Unit.xctest
Sources
TestClassA.m
TestClassB.swift
TestClassC.swift
Test harnessXcode
Test harnessXcode
Author tests
Test harnessXcode
Author testsExecute tests
Test harnessXcode
Author testsExecute testsReview reports
Continuous IntegrationXcode Server
Continuous IntegrationXcode Server
Schedules bot runs
Continuous IntegrationXcode Server
Schedules bot runsGenerates reports
Continuous IntegrationXcode Server
Schedules bot runsGenerates reportsSends notifications
Command line toolxcodebuild
Command line toolxcodebuild
Used by Xcode Server
Command line toolxcodebuild
Used by Xcode ServerBuild and execute tests
Command line toolxcodebuild
Used by Xcode ServerBuild and execute testsFor custom CI systems
Previous Years’ Sessions
Testing with Xcode 6 WWDC 2014
Continuous Integration with Xcode 6 WWDC 2014
UI Testing in Xcode WWDC 2015
CI and Code Coverage in Xcode WWDC 2015
Testing Concepts
Behind the scenesHow Testing Works
MyTests.xctest MyApp.app
Fundamental conceptsTesting Timeline
Compilation Hosting Observation
How testing startsTest Hosting
Unit Tests UI Tests
How testing startsTest Hosting
MyApp.app
Unit Tests UI Tests
How testing startsTest Hosting
MyApp.app
Unit Tests UI Tests
How testing startsTest Hosting
MyApp.app
Unit Tests UI Tests
Host Application
How testing startsTest Hosting
MyApp.app UI Test Runner MyApp.app
Unit Tests UI Tests
Host Application
How testing startsTest Hosting
MyApp.app UI Test Runner MyApp.app
Unit Tests UI Tests
Host Application
How testing startsTest Hosting
MyApp.app UI Test Runner MyApp.app
Unit Tests UI Tests
Host Application Target Application
ComparisonTest Hosting
Unit Tests UI Tests
Direct access to host app data and APIs Uses Accessibility to access target app
All tests run in single app launch Tests launch app for every test case
ComparisonTest Hosting
Unit Tests UI Tests
Direct access to host app data and APIs Uses Accessibility to access target app
All tests run in single app launch Tests launch app for every test case
Fundamental conceptsTesting Timeline
Compilation Hosting
Fundamental conceptsTesting Timeline
Compilation Hosting Observation
Test case structureTest Observation
Test case structureTest Observation
Test Case 1 Test Case 2
Test case structureTest Observation
Test Suite A
Test case structureTest Observation
Test Suite A Test Suite B
Test case structureTest Observation
Test Suite A Test Suite B
Test Bundle
Test case structureTest Observation
Setup or tear down work
Test Suite A Test Suite B
Test Bundle
Test case structureTest Observation
Setup or tear down workCustom logging
Test Suite A Test Suite B
Test Bundle
XCTestObservation ProtocolTest Observation
Test Suite A Test Suite B
Test Bundle
optional public func testBundleWillStart(_ testBundle: AnyObject!)
BUNDLE WILL START
XCTestObservation ProtocolTest Observation
Test Suite A Test Suite B
Test Bundle
optional public func testSuiteWillStart(_ testSuite: AnyObject!)
SUITE WILL START
XCTestObservation ProtocolTest Observation
Test Suite A Test Suite B
Test Bundle
optional public func testCaseWillStart(_ testCase: AnyObject!)
CASE WILL START
XCTestObservation ProtocolTest Observation
Test Suite A Test Suite B
Test Bundle
optional public func testCaseDidFinish(_ testCase: AnyObject!)
CASE DID FINISH
XCTestObservation ProtocolTest Observation
Test Suite A Test Suite B
Test Bundle
optional public func testCase(_ testCase: AnyObject!, didFailWithDescription
description: AnyObject!, inFile filePath: AnyObject!, atLine lineNumber: AnyObject!)
CASE DID FAIL
XCTestObservation ProtocolTest Observation
Test Suite A Test Suite B
Test Bundle
optional public func testSuiteDidFinish(_ testSuite: AnyObject!)
SUITE DID FINISH
XCTestObservation ProtocolTest Observation
Test Suite A Test Suite B
Test Bundle
optional public func testBundleDidFinish(_ testBundle: AnyObject!)
BUNDLE DID FINISH
// Test Observation // Example LoggingObserver
class LoggingObserver: NSObject, XCTestObservation {
}
// Test Observation // Example LoggingObserver
class LoggingObserver: NSObject, XCTestObservation { override init() { super.init() let sharedCenter = XCTestObservationCenter.shared() sharedCenter.addTestObserver(self) }
}
// Test Observation // Example LoggingObserver
class LoggingObserver: NSObject, XCTestObservation { override init() { super.init() let sharedCenter = XCTestObservationCenter.shared() sharedCenter.addTestObserver(self) } internal func testBundleWillStart(_ testBundle: Bundle) { print("Lights down, show is about to start") }
}
// Test Observation // Example LoggingObserver
class LoggingObserver: NSObject, XCTestObservation { override init() { super.init() let sharedCenter = XCTestObservationCenter.shared() sharedCenter.addTestObserver(self) } internal func testBundleWillStart(_ testBundle: Bundle) { print("Lights down, show is about to start") } internal func testCase(_ testCase: AnyObject!, didFailWithDescription description: AnyObject!, inFile filePath: AnyObject!, atLine lineNumber: AnyObject!) print("That sounds off key…") }
}
// Test Observation // Example LoggingObserver
class LoggingObserver: NSObject, XCTestObservation { override init() { super.init() let sharedCenter = XCTestObservationCenter.shared() sharedCenter.addTestObserver(self) } internal func testBundleWillStart(_ testBundle: Bundle) { print("Lights down, show is about to start") } internal func testCase(_ testCase: AnyObject!, didFailWithDescription description: AnyObject!, inFile filePath: AnyObject!, atLine lineNumber: AnyObject!) print("That sounds off key…") } internal func testBundleDidFinish(_ testBundle: Bundle) { print("Lights up, don’t forget your coat!") } }
Use NSPrincipalClassTest Observation
Use NSPrincipalClassTest Observation
Test Bundle Info.plist
Use NSPrincipalClassTest Observation
Test Bundle Info.plistTest specific instantiation
SummaryTesting Concepts
Compilation Hosting Observation
Crash Log Gathering
Crashes during testingCrash Log Gathering
Crashes during testingCrash Log Gathering
Crashes during testingCrash Log Gathering
Test host or target application may crash
Crashes during testingCrash Log Gathering
Test host or target application may crashXcode completes test suite
Crashes during testingCrash Log Gathering
Test host or target application may crashXcode completes test suiteBut how to resolve the crash?
Improved test reportsCrash Log Gathering NEW
Improved test reportsCrash Log Gathering NEW
Improved test reportsCrash Log Gathering
For Unit and UI tests
NEW
Improved test reportsCrash Log Gathering
For Unit and UI testsLocal and Xcode Server crash logs
NEW
Improved test reportsCrash Log Gathering
For Unit and UI testsLocal and Xcode Server crash logsLogs included in test report
NEW
Improved test reportsCrash Log Gathering
For Unit and UI testsLocal and Xcode Server crash logsLogs included in test reportView crashes in Xcode’s Debug Navigator
NEW
DemoCrash Log Gathering
Improved test reportsCrash Log Gathering
Crash logs gathered in test reports
Improved test reportsCrash Log Gathering
Crash logs gathered in test reportsView crashes in Xcode’s Debug Navigator
Improved test reportsCrash Log Gathering
Continuous Integration in XcodeXcode Server
Eric DudiakXcode Engineer
OverviewXcode Server
OverviewXcode Server
Custom environment variables
OverviewXcode Server
Custom environment variablesAdvanced trigger editing
OverviewXcode Server
Custom environment variablesAdvanced trigger editingIssue tracking and blame
OverviewXcode Server
Custom environment variablesAdvanced trigger editingIssue tracking and blameUpgrade integrations
OverviewXcode Server
Custom environment variablesAdvanced trigger editingIssue tracking and blameUpgrade integrationsConfigurable integration user
New in Xcode 7.3Custom Environment Variables
Integration scriptsAdvanced Triggers NEW
Email notificationsAdvanced Triggers NEW
Email notificationsAdvanced Triggers NEW
Email notificationsAdvanced Triggers NEW
Email notificationsAdvanced Triggers NEW
Email notificationsAdvanced Triggers NEW
Email notificationsAdvanced Triggers NEW
We all have issuesIssue Tracking and Blame
We all have issuesIssue Tracking and Blame
Nobody is perfect
We all have issuesIssue Tracking and Blame
Nobody is perfect• Tests will fail
We all have issuesIssue Tracking and Blame
Nobody is perfect• Tests will fail• Errors will come up
We all have issuesIssue Tracking and Blame
Nobody is perfect• Tests will fail• Errors will come up• Xcode will blame you
We all have issuesIssue Tracking and Blame
Nobody is perfect• Tests will fail• Errors will come up• Xcode will blame you
Even if you are perfect
We all have issuesIssue Tracking and Blame
Nobody is perfect• Tests will fail• Errors will come up• Xcode will blame you
Even if you are perfect• SDKs change
We all have issuesIssue Tracking and Blame
Nobody is perfect• Tests will fail• Errors will come up• Xcode will blame you
Even if you are perfect• SDKs change• Language improvements
We all have issuesIssue Tracking and Blame
Nobody is perfect• Tests will fail• Errors will come up• Xcode will blame you
Even if you are perfect• SDKs change• Language improvements• Smarter tools
Identifying issue ownersIssue Tracking and Blame NEW
Identifying issue ownersIssue Tracking and Blame
Lets you know when it’s time to act
NEW
Identifying issue ownersIssue Tracking and Blame
Lets you know when it’s time to actYou broke it
NEW
Identifying issue ownersIssue Tracking and Blame
Lets you know when it’s time to actYou broke it• You introduced an issue
NEW
Identifying issue ownersIssue Tracking and Blame
Lets you know when it’s time to actYou broke it• You introduced an issue• Issue is on or near line you modified
NEW
Identifying issue ownersIssue Tracking and Blame
Lets you know when it’s time to actYou broke it• You introduced an issue• Issue is on or near line you modified• You are the only committer
NEW
Identifying issue ownersIssue Tracking and Blame
Lets you know when it’s time to actYou broke it• You introduced an issue• Issue is on or near line you modified• You are the only committer
You can help fix it
NEW
Identifying issue ownersIssue Tracking and Blame
Lets you know when it’s time to actYou broke it• You introduced an issue• Issue is on or near line you modified• You are the only committer
You can help fix it• You probably know how to fix an issue
NEW
Identifying issue ownersIssue Tracking and Blame
Lets you know when it’s time to actYou broke it• You introduced an issue• Issue is on or near line you modified• You are the only committer
You can help fix it• You probably know how to fix an issue• You seem to know about the area involved
NEW
Identifying issue ownersIssue Tracking and Blame
Lets you know when it’s time to actYou broke it• You introduced an issue• Issue is on or near line you modified• You are the only committer
You can help fix it• You probably know how to fix an issue• You seem to know about the area involved• Fuzzy matching
NEW
Bot configuration changesIssue Tracking and Blame NEW
Bot configuration changesIssue Tracking and Blame
Tracks changes to your bot configuration
NEW
Bot configuration changesIssue Tracking and Blame
Tracks changes to your bot configurationAttributes issues to changes where possible
NEW
Bot configuration changesIssue Tracking and Blame
Tracks changes to your bot configurationAttributes issues to changes where possibleIncluded in email reports
NEW
When it’s not your faultUpgrade Integrations NEW
When it’s not your faultUpgrade Integrations
Re-integrates your project
NEW
When it’s not your faultUpgrade Integrations
Re-integrates your projectSame revision as the last integration
NEW
When it’s not your faultUpgrade Integrations
Re-integrates your projectSame revision as the last integrationAny new issues are due to the upgrade
NEW
When it’s not your faultUpgrade Integrations
Re-integrates your projectSame revision as the last integrationAny new issues are due to the upgradePrevents blaming you for broken builds
NEW
Configurable Integration User
Configurable Integration User
Improved visibility into your integrations
Configurable Integration User
Improved visibility into your integrationsAllows customization
Configurable Integration User
Improved visibility into your integrationsAllows customizationYou own and manage how your integrations run
Configurable Integration User
Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the system
Configurable Integration User
Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the systemRuns as a menu extra
Custom Integration User
Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the systemRuns as a menu extra
Custom Integration User
Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the systemRuns as a menu extra
••••••••••••••••••••••••••••••
Custom Integration User
Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the systemRuns as a menu extra
Custom Integration User
Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the systemRuns as a menu extra
Custom Integration User
Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the systemRuns as a menu extra
Custom Integration User
Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the systemRuns as a menu extra
Custom Integration User
Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the systemRuns as a menu extra
Custom Integration User
Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the systemRuns as a menu extra
Custom Integration User
Improved visibility into your integrationsAllows customizationYou own and manage how your integrations runCan be any user on the systemRuns as a menu extra
DemoXcode Server
Configurable Integration User
Configurable Integration User
Improved visibility
Configurable Integration User
Improved visibilityCustomized simulators
Configurable Integration User
Improved visibilityCustomized simulatorsNormal macOS user
Configurable Integration User
Improved visibilityCustomized simulatorsNormal macOS userRuns as a menu extra
Best practicesConfigurable Integration User
Best practicesConfigurable Integration User
Dedicate a new user
Best practicesConfigurable Integration User
Dedicate a new userAvoid using administrator accounts
Best practicesConfigurable Integration User
Dedicate a new userAvoid using administrator accountsStay logged in with Fast User Switching
Best practicesConfigurable Integration User
Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lock
Best practicesConfigurable Integration User
Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs
Best practicesConfigurable Integration User
Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs• Simulators
Best practicesConfigurable Integration User
Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs• Simulators• Networking (VPN and proxies)
Best practicesConfigurable Integration User
Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs• Simulators• Networking (VPN and proxies)• User data and settings
Best practicesConfigurable Integration User
Dedicate a new userAvoid using administrator accountsStay logged in with Fast User SwitchingDisable screen lockCustomize for your needs• Simulators• Networking (VPN and proxies)• User data and settings• Advanced provisioning
Building block for Continuous Integration Systemsxcodebuild
Test actionxcodebuild
xcodebuild test -workspace <path>
-scheme <name>
-destination <specifier>
Test actionxcodebuild
Builds sources
xcodebuild test -workspace <path>
-scheme <name>
-destination <specifier>
Test actionxcodebuild
Builds sourcesInstalls app on destination if needed
xcodebuild test -workspace <path>
-scheme <name>
-destination <specifier>
Test actionxcodebuild
Builds sourcesInstalls app on destination if neededRuns tests
xcodebuild test -workspace <path>
-scheme <name>
-destination <specifier>
Test actionxcodebuild
Builds sourcesInstalls app on destination if neededRuns testsReports results in console
xcodebuild test -workspace <path>
-scheme <name>
-destination <specifier>
Customizing testsTest Options NEW
xcodebuild test -workspace <path>
-scheme <name>
-destination <specifier>
-only-testing:TestBundleA/TestSuiteA/TestCaseA
-only-testing:TestBundleB/TestSuiteB
-only-testing:TestBundleC
Customizing testsTest Options NEW
xcodebuild test -workspace <path>
-scheme <name>
-destination <specifier>
-only-testing:TestBundleA/TestSuiteA/TestCaseA
-only-testing:TestBundleB/TestSuiteB
-only-testing:TestBundleC
xcodebuild test -scheme <name>
-skip-testing:TestBundleD/TestSuiteD/TestCaseD
Separating building from testingAdditional Test Actions NEW
Test
Separating building from testingAdditional Test Actions NEW
Build for Testing Test without Building
Only buildingBuilding for Testing NEW
xcodebuild build-for-testing -workspace <path>
-scheme <name>
-destination <specifier>
Builds sources for testing
Only buildingBuilding for Testing NEW
xcodebuild build-for-testing -workspace <path>
-scheme <name>
-destination <specifier>
Builds sources for testingOutputs products in Derived Data
Only buildingBuilding for Testing NEW
xcodebuild build-for-testing -workspace <path>
-scheme <name>
-destination <specifier>
Builds sources for testingOutputs products in Derived DataGenerates an xctestrun file
Only buildingBuilding for Testing NEW
xcodebuild build-for-testing -workspace <path>
-scheme <name>
-destination <specifier>
Only testingTest Without Building NEW
xcodebuild test-without-building -workspace <path>
-scheme <name>
-destination <specifier>
Only testingTest Without Building
Finds binary products in Derived Data
NEW
xcodebuild test-without-building -workspace <path>
-scheme <name>
-destination <specifier>
Only testingTest Without Building
Finds binary products in Derived DataInstalls if necessary
NEW
xcodebuild test-without-building -workspace <path>
-scheme <name>
-destination <specifier>
Only testingTest Without Building
Finds binary products in Derived DataInstalls if necessaryRuns tests and report results as before
NEW
xcodebuild test-without-building -workspace <path>
-scheme <name>
-destination <specifier>
Only testingTest Without Building NEW
xcodebuild test-without-building -xctestrun <path>
-destination <specifier>
Only testingTest Without Building
Ingests the xctestrun file
NEW
xcodebuild test-without-building -xctestrun <path>
-destination <specifier>
Only testingTest Without Building
Ingests the xctestrun fileFinds binary products relative to xctestrun file
NEW
xcodebuild test-without-building -xctestrun <path>
-destination <specifier>
Only testingTest Without Building
Ingests the xctestrun fileFinds binary products relative to xctestrun fileRuns tests, reports results as before
NEW
xcodebuild test-without-building -xctestrun <path>
-destination <specifier>
Distributed testingxcodebuild
Build Machine Test Machine #2
Test Machine #1
Distributed testingxcodebuild
Build Machine Test Machine #2
Test Machine #1
Distributed testingxcodebuild
Build Machine Test Machine #2
Test Machine #1
xcodebuild build-for-testing
Distributed testingxcodebuild
Build Machine Test Machine #2
Test Machine #1
xcodebuild build-for-testing
xcodebuild test-without-building
Distributed testingxcodebuild
Build Machine Test Machine #2
Test Machine #1
xcodebuild build-for-testing
xcodebuild test-without-building
xcodebuild test-without-building
Distributed testingxcodebuild
Build Machine Test Machine #2
Test Machine #1
xcodebuild build-for-testing
Test manifestxctestrun NEW
Test manifestxctestrun NEW
Test manifestxctestrun
Tests to run or skip
NEW
Test manifestxctestrun
Tests to run or skipEnvironment variables
NEW
Test manifestxctestrun
Tests to run or skipEnvironment variablesCommand-line arguments
NEW
Test manifestxctestrun
Tests to run or skipEnvironment variablesCommand-line argumentsDocumentation in man pages
NEW
Test manifestxctestrun
Tests to run or skipEnvironment variablesCommand-line argumentsDocumentation in man pagesman xcodebuild.xctestrun
NEW
Summary
Summary
Testing Concepts
Xcode• Crash Log Gathering
Xcode Server• Advanced Triggers• Issue Tracking and Blame• Configurable Integration User
xcodebuild
More Information
https://developer.apple.com/wwdc16/409
Labs
Xcode Open Hours Developer Tools Lab C Thursday 9:00AM
Xcode Open Hours Developer Tools Lab B Friday 9:00AM