timelapse interactive record/replay for the...

37
Timelapse Interactive record/replay for web apps Brian Burg Andrew J. Ko Michael D. Ernst Computer Science and Engineering University of Washington Richard J. Bailey 1

Upload: others

Post on 14-Aug-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Timelapse Interactive record/replay for web apps

Brian Burg Andrew J. Ko Michael D. Ernst

Computer Science and Engineering University of Washington

Richard J. Bailey

1

Page 2: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

“It’s hard to debug failures in the field”

2

Distributed across time and space

Hardware and software variation

Users are not software testers

Page 3: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Users can’t report failures accurately

Works for me..

“a piece doesn’t rotate properly!!”

end-user encounters bug in production

code

files bug report with ad-hoc information

developer unable to reproduce the bug

3

Page 4: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Users can’t report failures accurately

“The most severe problems were errors in steps to reproduce and incomplete information.”

“What makes a good bug report”. Zimmerman et al. TSE Vol. 36, No 5 2010

“a piece doesn’t rotate properly!!”

4

Works for me..

Page 5: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Users can’t report failures accurately

Bug reporters and developers want better tool support for reproducing buggy behavior.

“What makes a good bug report”. Zimmerman et al. TSE Vol. 36, No 5 2010

“a piece doesn’t rotate properly!!”

5

Works for me..

Page 6: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Existing tools are imprecise and hard to use

6

Selenium/ WebDriver

CoScripter Leshed et al, CHI 2008

Sikuli Script Yeh et al, UIST 2009

Capture and simulate user input. Designed for test and task automation.

macro replay (CoScripter, Selenium, Sikuli)

Page 7: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Existing tools are imprecise and hard to use

macro replay (CoScripter, Selenium, Sikuli)

7

Capture and simulate user input. Designed for test and task automation.

deterministic replay (Mugshot, WaRR)

Nondeterministic. Requires extra setup ahead of time. Can’t use with a debugger.

Save and reuse nondeterministic inputs to exactly recreate a specific execution. Play/pause buttons only. Slows down execution. Can’t use with a debugger.

Page 8: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Timelapse: a precise, fast, integrated replay tool

This talk:

• An interface for capturing and replaying program behavior

• Techniques for cheap, precise record/replay in web browsers

• How developers use record/replay during debugging tasks

8

Page 9: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

How to capture program behavior

9

Page 10: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

How to navigate a recording

10

Page 11: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Using replay while debugging

11

Page 12: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Browsers interpret input, render output

Output Browser Input (User, Network, Timers)

12

Web Interpreter (WebKit, Gecko)

Page 13: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Timelapse captures a browser’s inputs

Output

13

Inputs Log

Web Interpreter (WebKit, Gecko)

Browser Input (User, Network, Timers)

Page 14: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Timelapse replays a browser’s inputs

Output

Inputs Log

14

Web Interpreter (WebKit, Gecko)

Page 15: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Browsers have layered architectures

User input, commands

policy decisions

Date.now, win.colorDepth

event loop callbacks

Web Interpreter (WebKit, Gecko)

Platforms

15

Embedders (Firefox, Safari,

Chrome)

Page 16: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Timelapse intercepts input at layer boundaries

16

Embedders (Firefox, Safari,

Chrome)

Web Interpreter (WebKit, Gecko)

Platforms

Page 17: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

• Hardware interrupts

• Nondeterministic instructions

• Instruction counts

VM record/replay

• Async callbacks

• Nondeterministic APIs

• DOM event counts

Browser record/replay

17

Inspired by VM record/replay

Page 18: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Memoizing nondeterministic APIs

During normal execution, Date.now() returns the current time.

oo(a,b) { return a + b; } (c) { return “now:”+ Date.now(); }

/* file: Source/wtf/DateMath.h */ inline double jsCurrentTime() { return floor(WTF::currentTimeMS()); }

18 Web Interpreter Platform API

Page 19: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Inputs Log

Memoizing nondeterministic APIs

During recording, the return value of Date.now() is saved.

oo(a,b) { return a + b; } (c) { return “now:”+ Date.now(); }

/* file: Source/wtf/DateMath.h */ inline double jsCurrentTime() { return floor(WTF::currentTimeMS()); }

19 Web Interpreter Platform API

Page 20: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Memoizing nondeterministic APIs

On replay, the logged return value of Date.now() is used.

oo(a,b) { return a + b; } (c) { return “now:”+ Date.now(); }

/* file: Source/wtf/DateMath.h */ inline double jsCurrentTime() { return floor(WTF::currentTimeMS()); }

Inputs Log Web Interpreter Platform API

Page 21: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Making callbacks deterministic

enqueue()

while (true) { var event = queue.pop(); this.dispatchToListeners(event); }

Event Loop

timerFired()

Callback executes

Callback registered

Problem: accurately capturing and

simulating event loop dispatches.

Page 22: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

timer 42, 34 DOM events

Making callbacks deterministic

enqueue()

timerFired()

Callback executes

Callback registered

while (true) { var event = queue.pop(); this.dispatch(event); }

Inputs Log

Event Loop

Page 23: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Making callbacks deterministic

enqueue()

Callback registered

while (true) { var event = queue.pop(); this.dispatchToListeners(event); }

Event Loop

timerFired()

Callback executes Inputs Log

34 DOM events!

...

...

...

Page 24: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Runtime overheads are acceptable

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Run times (multiple of baseline)

Baseline

Recording

1× Replay

Seeking

24

Page 25: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Recordings are small and compressible

25

0

100

200

300

400

500

600

700

800JS

Linu

xJS

Ray

trace

rS

pace

Inva

ders

Moz

illa.

org

Cod

eMirr

orC

olor

pick

erD

uckD

uckG

o

Size (KB)

In-memory

Serialized

Compressed

Site recording duration

(s)

resources on page

(KB)

log growth

(KB/sec)

JSLinux 10.5 4500 0.8

JS Raytracer 6.3 5.9 1.6

Space Invaders 25.8 712 2.2

Mozilla.org 22.3 2800 1.3

CodeMirror 16.6 168 1.0

Colorpicker 15.3 577 1.7

DuckDuckGo 14.1 1900 2.1

Page 26: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Page resources dominate recording size

26

0500

100015002000250030003500400045005000

JSLi

nux

JS R

aytra

cer

Spa

ce In

vade

rsM

ozill

a.or

gC

odeM

irror

Col

orpi

cker

Duc

kDuc

kGo

Size (KB)

In-memory

Serialized

Compressed

Site recording duration

(s)

resources on page

(KB)

log growth

(KB/sec)

JSLinux 10.5 4500 0.8

JS Raytracer 6.3 5.9 1.6

Space Invaders 25.8 712 2.2

Mozilla.org 22.3 2800 1.3

CodeMirror 16.6 168 1.0

Colorpicker 15.3 577 1.7

DuckDuckGo 14.1 1900 2.1

Page 27: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

How would developers use it?

27

Study Design 20+ developers with industry experience within-subjects, 2 tasks per person, 45 minutes per task, 4 treatments

Performance

RQ: changes to frequency/duration?

Reproduction

RQ: complete tasks more quickly? more successfully? Who, why?

Page 28: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

How did developers use it?

28

Study Design 20+ developers with industry experience within-subjects, 2 tasks per person, 45 minutes per task, 4 treatments

Performance

Shorter and more frequent repro actions; Time spent unchanged (max. 25%; avg.15%) Successful developers quickly integrated replay into their existing workflows. Unsuccessful developers who used opportunistic strategies were distracted.

Reproduction

Page 29: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Current & Future Work

Visualizations Interaction histories aid navigation, but not program understanding.

Passive capturing

Precision and low overhead don’t matter if you forget to start capturing.

Post-hoc analysis

Developers can gather more runtime data without reproducing behavior: Post-hoc logging, Post-hoc Whyline, Post-hoc SeeSS, Testcase extraction

Page 30: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Conclusion

Visualizations Interaction histories supported–but didn’t reduce–reproduction of program state.

Infrastructure Replay infrastructure enables new research, tools and workflows.

Record/Replay Virtual machine replay techniques work well when applied to web applications.

github.com/burg/timelapse

Page 31: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Replay fidelity and completeness

Web interpreters expose a large and ever-changing API.

Timelapse doesn’t tame all sources of nondeterminism (yet).

Excepting untamed sources, the DOM tree and JavaScript heap are identical for all recorded and replayed executions.

Possible divergence is automatically detected when: • DOM event counts differ on capture and replay • Memoized inputs are overused or unused • Network request details differ unexpectedly • Known-bad APIs are used by client code

Divergence detection supports piecewise implementation.

31

Page 32: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Interpreter inputs by source User: mouse, keyboard, scroll, resize Network: images, scripts, HTML, AJAX Commands: page navigation

Internal nondeterminism: Animations, transitions, multimedia, async script and parser yields

Functions: Date.now, Math.random, etc Caching: resources, cookies Timers: timer schedule

32

Page 33: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Shim: the thing in the middle

Shims are used to implement deterministic record/replay.

The hard part of implementing record/replay is designing and placing shims. 33

Page 34: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Embedding and platform APIs

Embedders

EM

BE

DD

ING

A

PI

PLA

TFO

RM

AP

I

Web Interpreter (WebKit, Gecko)

Abstraction layers separate web interpreters from platforms/embedders.

Platforms

34

Page 35: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Embedding and platform APIs

Embedders

EM

BE

DD

ING

A

PI

PLA

TFO

RM

AP

I

Web Interpreter (WebKit, Gecko)

Abstraction layers separate web interpreters from platforms/embedders.

Platforms

35

Page 36: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Embedding and platform APIs

Embedders

Web Interpreter (WebKit, Gecko) Platforms

EM

BE

DD

ING

A

PI

PLA

TFO

RM

AP

I

Shims sit between the web interpreter and abstraction layers.

36

Page 37: Timelapse Interactive record/replay for the webmernst/pubs/record-replay-uist2013-slides.pdfTimelapse: a precise, fast, integrated replay tool . This talk: • An interface for capturing

Embedding and platform APIs

Embedders

Web Interpreter (WebKit, Gecko) Platforms

EM

BE

DD

ING

A

PI

PLA

TFO

RM

AP

I

Shims sit between the web interpreter and abstraction layers.

37