learning to plan chemical syntheses...3n-mcts algorithm neural nets objective selection expand and...
TRANSCRIPT
![Page 1: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/1.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Learning To Plan Chemical Syntheses
Yunhao (Jack) Ji Yizhan (Ethan) JiangShuja (Shuja) Khalid
Lipai (Jim) Xu
Yunhao (Jack) Ji
![Page 2: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/2.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Introduction● Retrosynthesis
Yunhao (Jack) Ji
1
8, 9 2, 6
7, 8 3, 6
4, 5, 6
A Search Tree Representation
![Page 3: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/3.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Motivation and Related Work● Manual constructing a valid tree can be hard
Yunhao (Jack) Ji
![Page 4: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/4.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Motivation and Related Work● Computer-assisted synthesis planning (CASP) can automatically extract the
transformations ● The generated tree has short depth but large branching factors and hard to
define heuristics.
An illustration of an example search tree to a synthesis planning
Yunhao (Jack) Ji
![Page 5: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/5.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Neural Networks
Learn Chemical Reaction Rules 12.4 million reactions from Reaxys database as dataset
Yizhan (Ethan) Jiang
![Page 6: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/6.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Neural Networks for Action Selection
Actions in AlphaGo Actions in Chemical Synthesis
Yizhan (Ethan) Jiang
![Page 7: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/7.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Neural Networks for Action Selection (1/2) ● Expansion Policy Neural Network
○ Find K most possible molecular transformations
Yizhan (Ethan) Jiang
![Page 8: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/8.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Neural Networks for Action Selection (2/2) ● In-Scope Filter Neural Network
○ Filter out infeasible transformations
Yizhan (Ethan) Jiang
![Page 9: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/9.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Neural Network for Rollout ● Rollout Policy Neural Network
○ Select 10 most possible transformations○ Only three layers for creating fast rollout policy
Yizhan (Ethan) Jiang
![Page 10: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/10.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Synthesis Planning with 3N-MCTS
Selection
Expansion
Evaluate
Backup
Shuja Khalid
3N - MCTS
The 3 Neural Networks covered on previous slides Next!
![Page 11: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/11.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Synthesis Planning with 3N-MCTS
Target Molecule
? ?
Selection
Expansion
Evaluate
Backup
: Visit count of state-action pair: Prior probability of visiting state-action pair
c : Exploration constant: Scalar value of state-action pair
Shuja Khalid
![Page 12: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/12.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Synthesis Planning with 3N-MCTS
ATarget Molecule
B C
Selection
Expansion
Evaluate
Backup
Shuja Khalid
![Page 13: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/13.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Synthesis Planning with 3N-MCTS
ATarget Molecule
B C
- Check if state is terminal - Terminal → evaluate with the reward function- Non-terminal → begin rollout/evaluation step
- Recursively sample actions from rollout policy until termination condition is met
Selection
Expansion
Evaluate
Backup
Shuja Khalid
![Page 14: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/14.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Synthesis Planning with 3N-MCTS
A
B C
: Visit count of state-action pair
: Custom objective function : Reward ∈ [-1,0,1]
Selection
Expansion
Evaluate
Backup
Shuja Khalid
![Page 15: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/15.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Synthesis Planning with MCTS
Shuja Khalid
AlphaGo Zero 3N-MCTS
Algorithm
Selection Expand andevaluate
Backup
Selection Expansion
RolloutUpdate
(p, v) = f(s) q = froll(s) , t = fexp(s) , p = fscope(s, r)
p: probability of selecting each move from a list of action probabilitiesv: scalar evaluation that estimates the probability of the current player winning from position s
q: scalar evaluation of node r: reactions between moleculest: possible transformationsp: probability of the molecules reacting
Select the set of actions (from a fixed set of actions) that
will lead to victory! Take that Lee Sedol!
Selecting the set of transformations (from a fixed set of
transformations) that will help us find new drugs to cure
diseases! Take that cancer!
Neural Nets
Goal
![Page 16: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/16.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Results & Discussion- Comparison with related methods
- Preference of chemical experts
- Limitations
Lipai (Jim) Xu
![Page 17: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/17.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Comparison with related methods
Lipai (Jim) Xu
Method
Time lim
3N-MCTS nBFS hBFS
5 sec 80% 40% 0%
60 sec 92% 71% 4%
1200 sec ~93% ~80% ~75%
![Page 18: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/18.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Preference of Chemical Experts
Lipai (Jim) Xu
![Page 19: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/19.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Limitations
● Not enough train data for some tasks● Stereochemistry ● Not totally admitted by the industry
Lipai (Jim) Xu
![Page 20: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/20.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Thank You
Lipai (Jim) Xu
![Page 21: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/21.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
ReferencesBackground image: http://turnoff.us/geek/binary-tree (with changes)
Alpha Go content: http://discovery.ucl.ac.uk/10045895/1/agz_unformatted_nature.pdf
Learning to Plan Chemical Synthesis content: https://arxiv.org/pdf/1708.04202.pdf
![Page 22: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/22.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Presenter - Your Name!
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
![Page 23: Learning To Plan Chemical Syntheses...3N-MCTS Algorithm Neural Nets Objective Selection Expand and evaluate Backup Selection Expansion Update Rollout (p, v) = f(s) q = f roll (s) ,](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe8548b1e034c0040192124/html5/thumbnails/23.jpg)
Presenter -
CSC 2547Learning To Search
Introduction Neural Networks MCTS Results
Synthesis Planning with MCTS
Shuja Khalid
Alpha Go Zero 3N-MCTS
Algorithm
Neural Nets
Objective
Selection Expand andevaluate
Backup
Selection Expansion
RolloutUpdate
(p, v) = f(s) q = froll(s) , t = fexp(s) , p = fscope(s, r) p: probability of selecting each move from a list of action probabilitiesv: scalar evaluation that estimates the probability of the current player winning from position s
q: scalar evaluation of node ; r: reactions between molecules t: possible transformations ; p: probability of the molecules reacting
where, U(s,a) ∝ P(s,a)/(1+N(s,a)) Q(s,a): action-value ; N(s,a): count visit ; P(s,a): prior probability
Maximise an upper confidence bound on Q(s,a) + U(s,a) Maximise the Q function which includes an adjustable objective W(bi)where, N(s,a): count visit ; zi: reward received during rollout