timelapse: interactive record/replay for the web

65
Timelapse Interactive record/replay for the web Brian Burg Andrew J. Ko Michael D. Ernst Computer Science and Engineering University of Washington

Upload: brrian

Post on 10-Jun-2015

1.852 views

Category:

Technology


4 download

DESCRIPTION

These are the slides for a series of work-in-progress talks given about Timelapse in June 2012. Please send comments and questions to @brrian on twitter, or find my email on the Timelapse project page: http://cs.washington.edu/homes/burg/timelapse/

TRANSCRIPT

Page 1: Timelapse: interactive record/replay for the web

Timelapse Interactive record/replay for the web

Brian Burg Andrew J. Ko Michael D. Ernst

Computer Science and EngineeringUniversity of Washington

Page 2: Timelapse: interactive record/replay for the web

Reproducing program behavior is essential in software development

Demonstration:

interacting with a program to achieve a behavior

Investigation:

interacting with developer tools to understand behavior

The phases of demonstrating and investigating are inseparable.

Page 3: Timelapse: interactive record/replay for the web

Task coupling can be hazardous

Example: debugging a color picker widget

Page 4: Timelapse: interactive record/replay for the web

Barriers to reproducing behavior

Demonstration

• Tedious, time-consuming, and manual

• Difficult to explain repro. steps (e.g., in a bug report)

Investigation

• Repeated demonstration to investigate same behavior

• The behavior being investigated is not held constant

• Interleaving the two phases incurs task switching penalties

Page 5: Timelapse: interactive record/replay for the web

Simplifying by decoupling

The means of investigation could change independently without re-demonstration.

Decoupling behavior demonstration from investigation simplifies reproduction and enables new practices.

Behavior could be demonstrated, and later, investigated, by different people.

Behavior could be reproduced precisely, automatically, quickly, and easily.

http://dryicons.com, http://www.getfirebug.com/

Page 6: Timelapse: interactive record/replay for the web

Solution: deterministic record/replay

User-space librariesJockey (C), Mugshot (JavaScript)

Application-specificVideo game engines

Operating systemsdOS, Determinator

Virtual machinesReVirt, LEAP, VMWare Workstation

HardwareDMP, RCDC, Calvin

Old idea: achieve identical execution by controlling nondeterminism.

Recent idea: modern browsers are virtual machines.

New idea: adapt virtual machine record/replay techniques to web browsers

Page 7: Timelapse: interactive record/replay for the web

Timelapse: record/replay for the web

Contributions

• Adaptation of VM record/replay techniques to the web

• Infrastructure and tool for interactive record/replay

• Design concepts for visualizing, interacting with recordings

Timelapse is an infrastructure and developer tool to record, replay, and investigate web application behavior.

Page 8: Timelapse: interactive record/replay for the web

Demonstrating and Investigating(with Timelapse)

Example: color picker, revisited

1. Begin recording2. Demonstrate behavior3. End recording4. Investigate as needed

Page 9: Timelapse: interactive record/replay for the web

Recreating a specific execution

To eliminate nondeterminism,

• Identify what is/isn’t deterministic

• Decide what sources to capture, and how

• Figure out how to replay the recorded nondeterminism

Timelapse adapts definitions and techniques from VM research.

Executions differ because of sources of nondeterminism.

Page 10: Timelapse: interactive record/replay for the web

Programs and Inputs

main.js

An execution is determined by a program and its inputs.

main.js

function foo(a,b) { return a + b; }

function bar(c) { return c * Math.random(); }

foo(argv[1], argv[2]);

bar(argv[1]); bar(argv[3]);

main.js

Page 11: Timelapse: interactive record/replay for the web

Programs and Inputs

main.js

An execution is determined by a program and its inputs.

main.js

function foo(a,b) { return a + b; }

function bar(c) { return c * Math.random(); }

foo(argv[1], argv[2]);

bar(argv[1]); bar(argv[3]);

main.js

Page 12: Timelapse: interactive record/replay for the web

Programs and Inputs

Sources of nondeterminism that affect the course of execution are implicit inputs to the program.

function foo(a,b) { return a + b; }

function bar(c) { return c * Math.random(); }

foo(argv[1], argv[2]);

bar(argv[1]); bar(argv[3]);

An execution is determined by a program and its inputs.

main.js

Page 13: Timelapse: interactive record/replay for the web

Interpreters and Inputs

>> foo(a,b) { return a + b; }

>> bar(c) { return c *

Math.random(); }

defs.js

>> load(“defs.js”);

>> foo(1,2);

>> foo(3,4);

>> bar(5);

>> bar(5);

REPL session

1. load defs.js

2. CALL foo 1, 2

3. CALL foo 3, 4

4. CALL bar, 5

5. RET Math.random, 0.453863

6. CALL bar, 5

7. RET Math.random, 0.986364

interpreter inputs

The session could be reproduced exactly by reusing inputs.

In an interpreter session, the program is the interpreter itself.

Page 14: Timelapse: interactive record/replay for the web

Web browsers are interpreters

Web browsers interpret JavaScript, CSS, HTML, and user input.

Web Interpreter OutputInput

Page 15: Timelapse: interactive record/replay for the web

Web browsers are interpreters

Executions can be reproduced by capturing and reusing inputs.

Web Interpreter OutputInput

Inputs Log

Page 16: Timelapse: interactive record/replay for the web

Record/replay goals

Precise deterministic record/replay• Must capture and reuse explicit and implicit inputs• Able to detect divergence of replay from recording

Self-contained, sharable recordings• Must not depend on any machine-specific context• Replay does not require external communication

Integration with existing architecture and tools• Record/replay must have low performance overhead• Controllable and compatible with existing developer tools

Page 17: Timelapse: interactive record/replay for the web

Record/replay technique: proxying

During normal execution, Date.now() returns the current time.

>> foo(a,b) { return a + b; }

>bar(c) { return “now:”+

Date.now(); }

/* file: Source/wtf/DateMath.h */

inline double jsCurrentTime(){return floor(WTF::currentTimeMS());}

Page 18: Timelapse: interactive record/replay for the web

Record/replay technique: proxying

During recording, the return value of Date.now() is saved.

>> foo(a,b) { return a + b; }

>bar(c) { return “now:”+

Date.now(); }

/* file: Source/wtf/DateMath.h */

inline double jsCurrentTime(){return floor(WTF::currentTimeMS());}

Inputs log

Page 19: Timelapse: interactive record/replay for the web

Record/replay technique: proxying

On replay, the logged return value of Date.now() is used.

>> foo(a,b) { return a + b; }

>bar(c) { return “now:”+

Date.now(); }

/* file: Source/wtf/DateMath.h */

inline double jsCurrentTime(){return floor(WTF::currentTimeMS());}

Inputs log

Page 20: Timelapse: interactive record/replay for the web

Shim: the thing in the middle

Shims are used to implement deterministic record/replay.

The hard part of implementing record/replay is designing and placing shims.

Page 21: Timelapse: interactive record/replay for the web

Background: browser architectures

Web interpreters are embeddable and cross-platform.

User input

Callbacks, visual output

Date.now, setTimeou

t

timer fire

Embedders Web Interpreter(WebKit, Gecko)

Platforms

Page 22: Timelapse: interactive record/replay for the web

Shim placement in the browser stack

Shims are placed at the API boundaries of the web interpreter.

Embedders Web Interpreter(WebKit, Gecko)

Platforms

Page 23: Timelapse: interactive record/replay for the web

Comparison: shim placement

For record/replay, VM only needs shim between OS/hardware

Page 24: Timelapse: interactive record/replay for the web

Why not place shims elsewhere?

At the OS or VM level (VMWare Replay)

Low overhead and exact replay

Can’t integrate with developer tools

At the JS library level (Mugshot: Mickens et al., 2010)

Cross-platform

Slower, lower fidelity, can’t integrate with tools

At the level of DOM events only (WaRR: Adrica et al., 2011)

Most DOM events are deterministic and/or have no effect

Skipping pre-dispatch browser code causes strange behavior

Page 25: Timelapse: interactive record/replay for the web

Interpreter inputs by source

User: mouse, keyboard, scroll, resizeNetwork: images, scripts, HTML, AJAXCommands: navigate, open, close page

Internal nondeterminism:Animations, transitions, multimedia,async. actions (load, error, popstate)

Functions: Date.now, Math.randomCaching: resources, cookiesTimers: timer interrupt schedule

Page 26: Timelapse: interactive record/replay for the web

“Pull” vs. “Push” inputsPUSH

TOmouse, keyboard, scroll, navigate, network events

resource data,client API hooks, input handling

PULL FROM

Animation, transition, load, error, media events

timer callbacks timestamp, random, cookies, persistent state

Page 27: Timelapse: interactive record/replay for the web

interruptsRTDSC, etc

Comparison: “Pull” vs. “Push”

Page 28: Timelapse: interactive record/replay for the web

Mechanics of execution replay

Injecting “push” inputs causes dispatch of DOM event(s).

1. mousedown

2. mouseup

3. click

4. focus

5. submit

DOM Event DIspatch

function onClicked(..) {

... = Date.now();

}

JavaScript execution

Page 29: Timelapse: interactive record/replay for the web

Comparison: replay timing

Problem: when to inject “push” inputs to cause same execution

VMs and Timelapse share notions of counting and injecting.

• Count branches, instructions

• Inject hardware interrupts

VM record/replay

• Count DOM event dispatches

• Inject “push” inputs

Browser record/replay

Page 30: Timelapse: interactive record/replay for the web

Challenge: finding and taming nondeterminism

Taming nondeterminism is easier with principled design.

Networked games like Starcraft 2 rely on record/replay facilities.

Video games explicitly design to a record/replay-like architecture.

Taming nondeterminism with shims relies on well-defined APIs.

Taming nondeterminism post-hoc requires refactoring and hacks.

Page 31: Timelapse: interactive record/replay for the web

Challenge: dealing with real web platforms

http://hawanja.deviantart.com/

• >1 MLOC (mainly C++/ JavaScript)

• Hundreds of committers, and 500-1000 commits per week

• 6 independent build systems

• Supports many OS, CPU, GUI toolkits, graphics engines, toggleable features, embedders

Web interpreters are complex, many-limbed, and fast-changing.

Page 32: Timelapse: interactive record/replay for the web

Lesson: consider many, implement one

http://commons.wikimedia.org

It’s fine to choose one reasonable configuration. But, don’t ignore essential complexity.

Platform-specific examples:• input method editor handling

• rendering and animation

• resource caching, cookies

Embedder-specific examples:• JavaScript engine

• Embedding API usage

For research, only address essential architectural complexity.

Page 33: Timelapse: interactive record/replay for the web

Interacting with a recording

The Omniscient Debugger for JavaControlling video on YouTube

Page 34: Timelapse: interactive record/replay for the web

Interacting with Whyline [Ko & Myers]

Whyline for Java: program output Whyline for Java: program output

Page 35: Timelapse: interactive record/replay for the web

Interacting with Timelapse

Timelapse Panel inside Web Inspector Application being replayed

Page 36: Timelapse: interactive record/replay for the web

Timeline visualization

InfoVis mantra: overview, zoom and filter, details-on-demand.

Filter inputs by

type

Program inputs over

time, by type

Time ruler

Page 37: Timelapse: interactive record/replay for the web

Timeline visualization

InfoVis mantra: overview, zoom and filter, details-on-demand.

Filter inputs by time span

Page 38: Timelapse: interactive record/replay for the web

Table of program inputs

Input types by

color

Inputs by id and time

Table is adjustable

Global record

controls

Panel-specific controls

InfoVis mantra: overview, zoom and filter, details-on-demand.

Page 39: Timelapse: interactive record/replay for the web

Recording, replaying, and modality

• Recording can be started/stopped from any Inspector panel

• Replay is controlled from the Timelapse panel only

• Modality displayed next to record/replay controls

Page 40: Timelapse: interactive record/replay for the web

Slider control and modality• Timeline and table are linked by sliders and filters.

• Dragging slider seeks replay to the dropped position.

Sliders show same position Dragging induces drop previews

Page 41: Timelapse: interactive record/replay for the web

How do developers use Timelapse?

Actual use cases? Good affordances? Workflow integration?

Formative user study

• Subjects: 9+3 professional web developers

• Tasks: create a test case, fix a bug

• Time: 45 min/task, + tutorial & exit interview

• How: within-subjects design, Camtasia, Timelapse

Page 42: Timelapse: interactive record/replay for the web

Important use cases

Play, watch and pause

Visual binary search

Stepping through inputs

Execution anchoring

Refining repro. steps

Page 43: Timelapse: interactive record/replay for the web

Important use cases

Play, watch and pause

Visual binary search

Stepping through inputs

Execution anchoring

Refining repro. steps

Page 44: Timelapse: interactive record/replay for the web

Important use cases

Play, watch and pause

Visual binary search

Stepping through inputs

Execution anchoring

Refining repro. steps

Page 45: Timelapse: interactive record/replay for the web

Important use cases

Play, watch and pause

Visual binary search

Stepping through inputs

Execution anchoring

Refining repro. steps

Page 46: Timelapse: interactive record/replay for the web

Important use cases

Play, watch and pause

Visual binary search

Stepping through inputs

Execution anchoring

Refining repro. steps

Page 47: Timelapse: interactive record/replay for the web

Important use cases

Play, watch and pause

Visual binary search

Stepping through inputs

Execution anchoring

Refining repro. steps

Page 48: Timelapse: interactive record/replay for the web

Study Results

No significant effect on task completion time. Why?

• Timelapse didn’t affect types of information sought

• “Program inputs” not related to existing concepts

• New strategies only possible with tight integration

Interactive record/replay mainly* accelerates existing strategies.

Page 49: Timelapse: interactive record/replay for the web

Future Work

“Canned” executions can bring research ideas into practice.

Dynamic analysison-demand race detection, code coverage, optimizations

Program synthesiscreate automated tests from recorded interactions

Corpus analysistrain models, test type systems, or evaluate analyses

Page 50: Timelapse: interactive record/replay for the web

Future Work

“Canned” executions can bring research ideas into practice.

Dynamic analysison-demand race detection, code coverage, optimizations

Program synthesiscreate automated tests from recorded interactions

Corpus analysistrain models, test type systems, or evaluate analyses

Page 51: Timelapse: interactive record/replay for the web

On-demand dynamic analysis

Dynamic analyses generate data at runtime with instrumentation.

Instrumented execution

Generated data

Dynamic analysis

Analysis Output

Page 52: Timelapse: interactive record/replay for the web

On-demand dynamic analysis

Record/replay enables on-demand dynamic analysis tools.

Timelapserecording

Normal execution

RECORD

Instrumented execution

REPLAY

Developer tools

Data and analysis

Page 53: Timelapse: interactive record/replay for the web

Conclusion

Timelapse decouples demonstration and investigation using ideas from virtual machine record/replay work.

The Timelapse tool supports interactive inspection.

Timelapse’s infrastructure is useful for other purposes.

The Timelapse research prototype is an open-source fork of WebKit, and is freely available from the project website.

https://bitbucket.org/burg/timelapse/

Page 54: Timelapse: interactive record/replay for the web

CALL FOR BACKUP

Page 55: Timelapse: interactive record/replay for the web

Network: a real proxy, not a shim

This is a hack.

In the short-term, easier than attempting to recreate platform-dependent network streams/handles

Long-term solution:

A resource loader implementation with built-in record/replay primitives, or a higher-level network API

An HTTP proxy records and replays network traffic.

Page 56: Timelapse: interactive record/replay for the web

Replay fidelity and completeness

Web interpreters expose a large and ever-changing API.

Timelapse doesn’t tame all sources of nondeterminism.

Excepting untamed sources, the DOM tree and JavaScript heap are identical for all recorded and replayed executions.

Divergence is automatically detected by comparing DOM events.

Divergence detection supports piecewise implementation.

Page 57: Timelapse: interactive record/replay for the web

Performance characteristics

Record and replay slowdowns are negligible because shims are placed along existing architectural boundaries.

Timelapse is ideal for CPU- and user-bound programs.

Network-bound applications are slower, due to prototype shortcuts (disabling caching and pipelining).

Replay/seek speed could be improved dramatically by checkpointing and clipping unnecessary layout/renders.

Performance hasn’t been a priority, but is adequate anyway.

Page 58: Timelapse: interactive record/replay for the web

Embedding and platform APIs

Embedders

EMBE

DD

ING

API

PLAT

FORM

API

Web Interpreter(WebKit, Gecko)

Abstraction layers separate web interpreters from platforms/embedders.

Platforms

Page 59: Timelapse: interactive record/replay for the web

Embedding and platform APIs

Embedders

EMBE

DD

ING

API

PLAT

FORM

API

Web Interpreter(WebKit, Gecko)

Abstraction layers separate web interpreters from platforms/embedders.

Platforms

Page 60: Timelapse: interactive record/replay for the web

Embedding and platform APIs

Embedders Web Interpreter(WebKit, Gecko)

Platforms

EMBE

DD

ING

API

PLAT

FORM

API

Shims sit between the web interpreter and abstraction layers.

Page 61: Timelapse: interactive record/replay for the web

Embedding and platform APIs

Embedders Web Interpreter(WebKit, Gecko)

Platforms

EMBE

DD

ING

API

PLAT

FORM

API

Shims sit between the web interpreter and abstraction layers.

Page 62: Timelapse: interactive record/replay for the web

Shim behavior when recording

[user mouse press]-> (IPC communication)-> WebKit::WebPage::mouseEvent()-> WebCore::InputProxy::handleMousePress()---> WebCore::DeterminismController::capture()-> WebCore::EventHandler::handleMousePress()-> WebCore::Node::dispatchMouseEvent()

EMBE

DD

ING

API

PLAT

FORM

API

Start recording!

Page 63: Timelapse: interactive record/replay for the web

Shim behavior when recording

[user mouse press]-> (IPC communication)-> WebKit::WebPage::mouseEvent()-> WebCore::InputProxy::handleMousePress()---> WebCore::DeterminismController::capture()-> WebCore::EventHandler::handleMousePress()-> WebCore::Node::dispatchMouseEvent()

EMBE

DD

ING

API

PLAT

FORM

API

Page 64: Timelapse: interactive record/replay for the web

EMBE

DD

ING

API

PLAT

FORM

API

Shim behavior when replaying

DeterminismController::dispatchNextAction()-> MousePressAction::dispatch()-> InputProxy::handleMousePress(true)-> EventHandler::handleMousePress()-> Node::dispatchMouseEvent()-> WebKit::WebPage::mouseEvent()-> InputProxy::handleMousePress(false)

Start replaying!

Page 65: Timelapse: interactive record/replay for the web

EMBE

DD

ING

API

PLAT

FORM

API

Shim behavior when replaying

DeterminismController::dispatchNextAction()-> MousePressAction::dispatch()-> InputProxy::handleMousePress(true)-> EventHandler::handleMousePress()-> Node::dispatchMouseEvent()-> WebKit::WebPage::mouseEvent()-> InputProxy::handleMousePress(false)