visual debugger for jupyter notebooks: myth or reality? · 2019-07-10 · visual debugger for...

Visual Debugger for Jupyter Notebooks:

Myth or Reality?Elizaveta Shashkova

EuroPython 2019

About Me

•Software Developer at JetBrains, PyCharm IDE

•Debugger and Data Science tools

• @lisa_shashkova

�2

Visual Debugger

�3

Jupyter Notebooks

•Popular scientific tool

• File is a sequence of cells

�4

Jupyter Notebooks Debug

�5

•Logging with print statements

•Command-line debugger ipdb

Jupyter Notebooks Debug

�6

Myth or Reality?

Contents

•Python files debugging

• Jupyter breakpoints

•Debugger communication

• Jupyter visual debugger

�9

Tracing Function

�10

def tracefunc(frame, event, arg): print(frame.f_lineno, event) return tracefunc

sys.settrace(tracefunc)

1 2 3 4 5 6

Tracing Function

�11

def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets)

sys.settrace(tracefunc) greet_neighbors()

1 2 3 4 5 6 7 8 9

Tracing Function

�12

1 2 3 4 5 6 7 8 9

1 call def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets)


Tracing Function

�13

1 2 3 4 5 6 7 8 9

1 call 2 line



Tracing Function

�14

1 2 3 4 5 6 7 8 9

1 call 2 line 3 line 4 line Hi Mars!



Tracing Function

�15

1 2 3 4 5 6 7 8 9

1 call 2 line 3 line 4 line Hi Mars! 3 line 4 line Hi Venus!



Tracing Function

�16

1 2 3 4 5 6 7 8 9



1 call 2 line 3 line 4 line Hi Mars! 3 line 4 line Hi Venus! 5 line 5 return

Breakpoint

•frame.f_lineno - current line number

•frame.f_code.co_filename - current file name

�17

Breakpoint

•frame.f_lineno - current line number

•frame.f_code.co_filename - current file name

•Equals to breakpoint’s file and line -> suspend program!

�18

Contents





�19

Contents





�20

Cells Execution

�21

IPython kernel

Front-end

IDE

Cells Execution

�22

code IPython kernel

Front-end

IDE

IPython kernel

Cells Execution

�23

code executionFront-end

IDE

Cells Execution

�24

result

IPython kernel

Front-end

IDE

Cells Execution

•Kernel generates a unique name for each cell

• <ipython-input-5-11faed10a894>

•File name of a generated code object

�25

IPython kernel

code execution

Jupyter Breakpoints

•Python files: (filename, line number) -> unique location

�26

Jupyter Breakpoints


• Jupyter Notebooks?

�27

Jupyter Breakpoints


• Jupyter Notebooks:

•generated cell name

• line inside code object

�28

Source Mapping

�29

IPython kernelIDE

cell source codeMyNotebook.ipynb

generated <code object>

Source Mapping

�30

IPython kernelIDE


cell idgenerated

<code object>

IPython kernelIDE

Source Mapping

�31

?


cell idgenerated

<code object>

Source Mapping

�32

•Tracking cells execution in the IDE

Source Mapping

�33

•Tracking cells execution in the IDE

•Silent cell execution in IPython kernel

Debug Cell Execution

�34

<cell source code>


�35

<cell source code>

patch name generation

- silent mode


�36

<cell source code>

patch name generation

cell id

- silent mode

Jupyter Tracing Function

•frame.f_code.co_filename - generated name

�37



•Map: generated name -> cell id

�38




• Send message to the IDE

�39




• Send message to the IDE

�40

Contents





�41

Contents





�42

Debug Communication

�43

“Add breakpoint in a cell 3, line 2”

IPython kernel

Front-end

IDE

Debug Communication

�44

“Add breakpoint in a cell 3, line 2”

IPython kernel

Front-end

IDE

Debug Communication

�45

•Additional connection

•Reuse Jupyter channels

Jupyter Messaging

�46

Front-end

IPython kernelKernel proxy

Jupyter Messaging

�47

Front-end

IPython kernelKernel proxy

Debug Communication

�48



Debug Communication

�49



Jupyter Messaging

�50

Front-endKernel proxy

IPython kernelShell

IOPub

stdin

Jupyter Architecture

�51

•Event loop in a main thread for execution events

•Event loop for output events

Jupyter Architecture

�52

code

debug

x Blocked

IPython kernel

Front-end

IDE

Debug Communication

�53



Debug Communication

�54



But ipdb works!

But ipdb Works!

�55

But ipdb Works!

�56

•Based on input()

•Reuses user input channel

Debug Communication

�57



Contents





�58

Contents





�59

Jupyter Visual Debugger

• Jupyter tracing function

�60



•Mapping between editor and generated code

�61




•Debugger connection

�62

Live Demo

�63

Live Demo

�64

•PyCharm doesn’t convert Jupyter Notebooks to Python files!

•On disk it’s still the same JSON file with .ipynb extension




•Debugger connection

�65


•Implement in your favourite IDE

�66



•Try it in PyCharm Pro!

�67



•Try it in PyCharm Pro!

•Questions?

�68

@lisa_shashkova

visual debugger for jupyter notebooks: myth or reality? · 2019-07-10 · visual debugger for...

Documents