visual debugger for jupyter notebooks: myth or reality? · 2019-07-10 · visual debugger for...

Visual Debugger for Jupyter Notebooks:

Myth or Reality?Elizaveta Shashkova

EuroPython 2019

About Me

•Software Developer at JetBrains, PyCharm IDE

•Debugger and Data Science tools

• @lisa_shashkova

�2

Visual Debugger

�3

Jupyter Notebooks

•Popular scientific tool

• File is a sequence of cells

�4

Jupyter Notebooks Debug

�5

•Logging with print statements

•Command-line debugger ipdb

Jupyter Notebooks Debug

�6

Myth or Reality?

Myth or Reality?

Contents

•Python files debugging

• Jupyter breakpoints

•Debugger communication

• Jupyter visual debugger

�9

Tracing Function

�10

def tracefunc(frame, event, arg): print(frame.f_lineno, event) return tracefunc

sys.settrace(tracefunc)

1 2 3 4 5 6

Tracing Function

�11

def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets)

sys.settrace(tracefunc) greet_neighbors()

1 2 3 4 5 6 7 8 9

Tracing Function

�12

1 2 3 4 5 6 7 8 9

1 call def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets)

sys.settrace(tracefunc) greet_neighbors()

Tracing Function

�13

1 2 3 4 5 6 7 8 9

1 call 2 line

def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets)

sys.settrace(tracefunc) greet_neighbors()

Tracing Function

�14

1 2 3 4 5 6 7 8 9

1 call 2 line 3 line 4 line Hi Mars!

def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets)

sys.settrace(tracefunc) greet_neighbors()

Tracing Function

�15

1 2 3 4 5 6 7 8 9

1 call 2 line 3 line 4 line Hi Mars! 3 line 4 line Hi Venus!

def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets)

sys.settrace(tracefunc) greet_neighbors()

Tracing Function

�16

1 2 3 4 5 6 7 8 9

def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets)

sys.settrace(tracefunc) greet_neighbors()

1 call 2 line 3 line 4 line Hi Mars! 3 line 4 line Hi Venus! 5 line 5 return

Breakpoint

•frame.f_lineno - current line number

•frame.f_code.co_filename - current file name

�17

Breakpoint

•frame.f_lineno - current line number

•frame.f_code.co_filename - current file name

•Equals to breakpoint’s file and line -> suspend program!

�18

Contents

•Python files debugging

• Jupyter breakpoints

•Debugger communication

• Jupyter visual debugger

�19

Contents

•Python files debugging

• Jupyter breakpoints

•Debugger communication

• Jupyter visual debugger

�20

Cells Execution

�21

IPython kernel

Front-end

IDE

Cells Execution

�22

code IPython kernel

Front-end

IDE

IPython kernel

Cells Execution

�23

code executionFront-end

IDE

Cells Execution

�24

result

IPython kernel

Front-end

IDE

Cells Execution

•Kernel generates a unique name for each cell

• <ipython-input-5-11faed10a894>

•File name of a generated code object

�25

IPython kernel

code execution

Jupyter Breakpoints

•Python files: (filename, line number) -> unique location

�26

Jupyter Breakpoints

•Python files: (filename, line number) -> unique location

• Jupyter Notebooks?

�27

Jupyter Breakpoints

•Python files: (filename, line number) -> unique location

• Jupyter Notebooks:

•generated cell name

• line inside code object

�28

Source Mapping

�29

IPython kernelIDE

cell source codeMyNotebook.ipynb

generated <code object>

Source Mapping

�30

IPython kernelIDE

cell source codeMyNotebook.ipynb

cell idgenerated

<code object>

IPython kernelIDE

Source Mapping

�31

?

cell source codeMyNotebook.ipynb

cell idgenerated

<code object>

Source Mapping

�32

•Tracking cells execution in the IDE

Source Mapping

�33

•Tracking cells execution in the IDE

•Silent cell execution in IPython kernel

Debug Cell Execution

�34

<cell source code>

Debug Cell Execution

�35

<cell source code>

patch name generation

- silent mode

Debug Cell Execution

�36

<cell source code>

patch name generation

cell id

- silent mode

Jupyter Tracing Function

•frame.f_code.co_filename - generated name

�37

Jupyter Tracing Function

•frame.f_code.co_filename - generated name

•Map: generated name -> cell id

�38

Jupyter Tracing Function

•frame.f_code.co_filename - generated name

•Map: generated name -> cell id

• Send message to the IDE

�39

Jupyter Tracing Function

•frame.f_code.co_filename - generated name

•Map: generated name -> cell id

• Send message to the IDE

�40

Contents

•Python files debugging

• Jupyter breakpoints

•Debugger communication

• Jupyter visual debugger

�41

Contents

•Python files debugging

• Jupyter breakpoints

•Debugger communication

• Jupyter visual debugger

�42

Debug Communication

�43

“Add breakpoint in a cell 3, line 2”

IPython kernel

Front-end

IDE

Debug Communication

�44

“Add breakpoint in a cell 3, line 2”

IPython kernel

Front-end

IDE

Debug Communication

�45

•Additional connection

•Reuse Jupyter channels

Jupyter Messaging

�46

Front-end

IPython kernelKernel proxy

Jupyter Messaging

�47

Front-end

IPython kernelKernel proxy

Debug Communication

�48

•Additional connection

•Reuse Jupyter channels

Debug Communication

�49

•Additional connection

•Reuse Jupyter channels

Jupyter Messaging

�50

Front-endKernel proxy

IPython kernelShell

IOPub

stdin

Jupyter Architecture

�51

•Event loop in a main thread for execution events

•Event loop for output events

Jupyter Architecture

�52

code

debug

x Blocked

IPython kernel

Front-end

IDE

Debug Communication

�53

•Additional connection

•Reuse Jupyter channels

Debug Communication

�54

•Additional connection

•Reuse Jupyter channels

But ipdb works!

But ipdb Works!

�55

But ipdb Works!

�56

•Based on input()

•Reuses user input channel

Debug Communication

�57

•Additional connection

•Reuse Jupyter channels

Contents

•Python files debugging

• Jupyter breakpoints

•Debugger communication

• Jupyter visual debugger

�58

Contents

•Python files debugging

• Jupyter breakpoints

•Debugger communication

• Jupyter visual debugger

�59

Jupyter Visual Debugger

• Jupyter tracing function

�60

Jupyter Visual Debugger

• Jupyter tracing function

•Mapping between editor and generated code

�61

Jupyter Visual Debugger

• Jupyter tracing function

•Mapping between editor and generated code

•Debugger connection

�62

Live Demo

�63

Live Demo

�64

•PyCharm doesn’t convert Jupyter Notebooks to Python files!

•On disk it’s still the same JSON file with .ipynb extension

Jupyter Visual Debugger

• Jupyter tracing function

•Mapping between editor and generated code

•Debugger connection

�65

Jupyter Visual Debugger

•Implement in your favourite IDE

�66

Jupyter Visual Debugger

•Implement in your favourite IDE

•Try it in PyCharm Pro!

�67

Jupyter Visual Debugger

•Implement in your favourite IDE

•Try it in PyCharm Pro!

•Questions?

�68

@lisa_shashkova