Upload File

> v ] v p ^ µ v } ^ µ v u } o w } v v ] } v ] d u } o o ] ( ] ] }...

  • Home
  • Documents
  • > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u
172
Deep Learning Sequence to Sequence models: Connectionist Temporal Classification 1

Upload: others

Post on 02-Dec-2020

4 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Deep LearningSequence to Sequence models:

Connectionist Temporal Classification

1

Page 2: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Sequence-to-sequence modelling• Problem:

– A sequence goes in– A different sequence comes out

• E.g.– Speech recognition: Speech goes in, a word sequence comes out

• Alternately output may be phoneme or character sequence

– Machine translation: Word sequence goes in, word sequence comes out

– Dialog : User statement goes in, system response comes out– Question answering : Question comes in, answer goes out

• In general – No synchrony between and .

2

Page 3: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Sequence to sequence

• Sequence goes in, sequence comes out• No notion of “time synchrony” between input and output

– May even not even maintain order of symbols• E.g. “I ate an apple” “Ich habe einen apfel gegessen”

– Or even seem related to the input• E.g. “My screen is blank” “Please check if your computer is plugged in.”

3

Seq2seq

Seq2seqI ate an apple Ich habe einen apfel gegessen

I ate an apple

v

Page 4: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Sequence to sequence

• Sequence goes in, sequence comes out• No notion of “time synchrony” between input and output

– May even not even maintain order of symbols• E.g. “I ate an apple” “Ich habe einen apfel gegessen”

– Or even seem related to the input• E.g. “My screen is blank” “Can you check if your computer is plugged in?”

4

Seq2seq

Seq2seqI ate an apple Ich habe einen apfel gegessen

I ate an apple

v

Page 5: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Case 1: Order-aligned but not time synchronous

• The input and output sequences happen in the same order– Although they may not be time synchronous, they can be

“aligned” against one another– E.g. Speech recognition

• The input speech can be aligned to the phoneme sequence output

Time

X(t)

Y(t)

t=0

h-1

5

Page 6: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Problems

• How do we perform inference on such a model– How to output time-asynchronous sequences

• How do we train such models

6

Page 7: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Problems

• How do we perform inference on such a model– How to output time-asynchronous sequences

• How do we train such models

7

Page 8: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

The inference problem

• Objective: Given a sequence of inputs, asynchronously output a sequence of symbols– “Decoding”

/B/ /F/ /IY/

8

/IY/

Page 9: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Recap: Inference

• How do we know when to output symbols– In fact, the network produces outputs at every

time– Which of these are the real outputs?

/B/

9

/F/ /IY//IY/

Page 10: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

The actual output of the network

• At each time the network outputs a probability for each output symbol given all inputs until that time– E.g.

/AH/

/B/

/D/

/EH/

/IY/

/F/

/G/

10

Page 11: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Overall objective

• Find most likely symbol sequence given inputs

11

/AH/

/B/

/D/

/EH/

/IY/

/F/

/G/

Page 12: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Finding the best output

• Option 1: Simply select the most probable symbol at each time

12

/AH/

/B/

/D/

/EH/

/IY/

/F/

/G/

Page 13: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Finding the best output

• Option 1: Simply select the most probable symbol at each time– Merge adjacent repeated symbols, and place the actual emission

of the symbol in the final instant 13

/AH/

/B/

/D/

/EH/

/IY/

/F/

/G/ /G/

/F/

/IY/

/D/

Page 14: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Simple pseudocode

• Assuming is already computed using the underlying RNN

n = 1best(1)= argmaxi(y(1,i))for t = 1:T

best(t)= argmaxi(y(t,i))if (best(t) != best(t-1))

out(n) = best(t-1)time(n) = t-1n = n+1

14

Page 15: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

The actual output of the network

• Option 1: Simply select the most probable symbol at each time– Merge adjacent repeated symbols, and place the actual emission

of the symbol in the final instant 15

/AH/

/B/

/D/

/EH/

/IY/

/F/

/G/ /G/

/F/

/IY/

/D/

Cannot distinguish between an extended symbol andrepetitions of the symbol

/F/

Page 16: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Greedy Decoding: Recap

• This is in fact a suboptimal decode that actually finds the most likely time-synchronous output sequence– Which is not necessarily the most likely order-synchronous sequence– We will return to this topic later 16

/AH/

/B/

/D/

/EH/

/IY/

/F/

/G/

Page 17: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

The sequence-to-sequence problem

• How do we know when to output symbols– In fact, the network produces outputs at every time– Which of these are the real outputs

• How do we train these models?

/B/

17

/F/ /IY//IY/

Page 18: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Recap: Training with alignment

• Training data: input sequence + output sequence– Output sequence length <= input sequence length

• Given the alignment of the output to the input– The phoneme /B/ ends at X2, /AH/ at X6, /T/ at X9

/B/ /AH/ /T/

18

Page 19: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Recap: Characterizing an alignment

19

/B/ /AH/ /T/

• Given only the order-synchronous sequence and its time stamps–

– E.g.

• Repeat symbols to convert it to a time-synchronous sequence–

– E.g.

Page 20: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Recap: Characterizing an alignment

• Given only the order-synchronous sequence and its time stamps–

– E.g.

• Repeat symbols to convert it to a time-synchronous sequence–

– E.g.

20

/B/ /AH/ /T//B//B//B/ /AH//AH//AH/ /T/

Page 21: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Recap: Characterizing an alignment

• Given only the order-synchronous sequence and its time stamps– 𝑆 𝑇 , 𝑆 𝑇 , … , 𝑆 𝑇

– E.g. 𝑆 =/𝐵/ 3 , 𝑆 =/𝐵/ 7 , 𝑆 =/𝑇/ 9 ,

• Repeat symbols to convert it to a time-synchronous sequence– 𝑠 = 𝑆 , 𝑠 = 𝑆 , … , 𝑆 = 𝑆 , 𝑠 = 𝑆 , … , 𝑠 = 𝑆 , 𝑠 = 𝑆 , … , 𝑠 = 𝑆

– E.g. 𝑠 , 𝑠 , … , 𝑠 =/𝐵//𝐵//𝐵//𝐵//𝐴𝐻//𝐴𝐻//𝐴𝐻//𝐴𝐻//𝐴𝐻//𝑇//𝑇/

• For our purpose an alignment of to an input of length N has the form– 𝒔𝟎, 𝒔𝟏, … , 𝒔𝑵 𝟏 = 𝑺𝟎, 𝑺𝟎, … , 𝑺𝟎, 𝑺𝟏, 𝑺𝟏, … , 𝑺𝟏, 𝑺𝟐, … , 𝑺𝑲 𝟏 (of length 𝑵)

• Any sequence of this kind of length that contracts (by eliminating repetitions) to is a candidate alignment of

21

/B/ /AH/ /T//B//B//B/ /AH//AH//AH/ /T/

Page 22: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• Given the order-aligned output sequence with timing

/B/

Div Div Div

/F/ /IY/

22

/IY/

Div

Recap: Training with alignment

Page 23: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• Given the order aligned output sequence with timing– Convert it to a time-synchronous alignment by repeating symbols

• Compute the divergence from the time-aligned sequence

/B/

Div Div Div

/F/ /IY/

DivDivDivDivDivDivDiv

23

/IY/

Page 24: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• The gradient w.r.t the -th output vector

– Zeros except at the component corresponding to the target aligned to that time

/B/

Div Div Div

/F/ /IY/

DivDivDivDivDivDivDiv

24

/IY/

Page 25: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Problem: Alignment not provided

• Only the sequence of output symbols is provided for the training data– But no indication of which one occurs where

• How do we compute the divergence?– And how do we compute its gradient w.r.t.

/B/ /IY/ /IY/

? ? ? ? ? ? ? ? ? ?

25

/F/

Page 26: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Recap: Training without alignment

• We know how to train if the alignment is provided

• Problem: Alignment is not provided

• Solution:1. Guess the alignment2. Consider all possible alignments

26

Page 27: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Solution 1: Guess the alignment

• Initialize: Assign an initial alignment– Either randomly, based on some heuristic, or any other rationale

• Iterate:– Train the network using the current alignment– Reestimate the alignment for each training instance

• Using the Viterbi algorithm

? ? ? ? ? ? ? ? ? ?

27

/B/ /B/ /IY/ /IY/ /IY/ /F/ /F/ /F/ /F/ /IY/

Page 28: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

28

Arrange the constructed table so that from top to bottom it has the exact sequence of symbols required

Recap: Estimating the alignment: Step 1/B/

/IY/

/IY/

/F/

Page 29: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Recap: Viterbi algorithm

• Initialization:

• for

for

• 𝐵𝑃 𝑡, 𝑙 =𝑙 − 1 ∶ 𝑖𝑓 𝐵𝑠𝑐𝑟 𝑡 − 1, 𝑙 − 1 > 𝐵𝑠𝑐𝑟 𝑡 − 1, 𝑙 𝑙 − 1;

𝑙 ∶ 𝑒𝑙𝑠𝑒

• 𝐵𝑠𝑐𝑟(𝑡, 𝑙) = 𝐵𝑠𝑐𝑟(𝐵𝑃(𝑡, 𝑙)) × 𝑦

29

/IY/

/B/

/F/

/IY/

Page 30: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Recap: Viterbi algorithm

•

• for

30

/B/ /B/ /IY/ /F/ /F/ /IY/ /IY/ /IY/ /IY/

/IY/

/B/

/F/

/IY/

Page 31: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

VITERBI#N is the number of symbols in the target output#S(i) is the ith symbol in target output#T = length of input

#First create output tableFor i = 1:N

s(1:T,i) = y(1:T, S(i))

#Now run the Viterbi algorithm# First, at t = 1BP(1,1) = -1Bscr(1,1) = s(1,1)Bscr(1,2:N) = -inftyfor t = 2:T

BP(t,1) = 1;Bscr(t,1) = Bscr(t-1,1)*s(t,1)for i = 1:min(t,N)

BP(t,i) = Bscr(t-1,i) > Bscr(t-1,i-1) ? i : i-1Bscr(t,i) = Bscr(t-1,BP(t,i))*s(t,i)

# BacktraceAlignedSymbol(T) = Nfor t = T downto 2

AlignedSymbol(t-1) = BP(t,AlignedSymbol(t))

31Using 1..N and 1..T indexing, instead of 0..N-1, 0..T-1, for convenience of notation

Page 32: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

VITERBI#N is the number of symbols in the target output#S(i) is the ith symbol in target output#T = length of input

# First, at t = 1BP(1,1) = -1Bscr(1,1) = y(1,S(1))Bscr(1,2:N) = -inftyfor t = 2:T

BP(t,1) = 1;Bscr(t,1) = Bscr(t-1,1)*y(t,S(1))for i = 2:min(t,N)

BP(t,i) = Bscr(t-1,i) > Bscr(t-1,i-1) ? i : i-1Bscr(t,i) = Bscr(t-1,BP(t,i))*y(t,S(i))

# BacktraceAlignedSymbol(T) = Nfor t = T downto 2

AlignedSymbol(t-1) = BP(t,AlignedSymbol(t))

32Using 1..N and 1..T indexing, instead of 0..N-1, 0..T-1, for convenience of notation

Without explicit construction of output table

Page 33: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Recap: Iterative Estimate and Training

? ? ? ? ? ? ? ? ? ?

33

/B/ /B/ /IY/ /F/ /F/ /IY/ /IY/ /IY/ /IY/ /IY/

Decode to obtainalignments

Train model withgiven alignments

Initializealignments

The “decode” and “train” steps may be combined into a single “decode, find alignment,compute derivatives” step for SGD and mini-batch updates

Page 34: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Iterative update: Problem

• Approach heavily dependent on initial alignment

• Prone to poor local optima

• Alternate solution: Do not commit to an alignment during any pass..

34

Page 35: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Recap: Training without alignment

• We know how to train if the alignment is provided

• Problem: Alignment is not provided

• Solution:1. Guess the alignment2. Consider all possible alignments

35

Page 36: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• We commit to the single “best” estimated alignment– The most likely alignment

– This can be way off, particularly in early iterations, or if the model is poorly initialized

• Alternate view: there is a probability distribution over alignments– Selecting a single alignment is the same as drawing a single sample from this

distribution– Selecting the most likely alignment is the same as deterministically always drawing

the most probable value from the distribution 36

The reason for suboptimality

/IY/

/B/

/F/

/IY/

Page 37: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• We commit to the single “best” estimated alignment– The most likely alignment

– This can be way off, particularly in early iterations, or if the model is poorly initialized

• Alternate view: there is a probability distribution over alignments of the target Symbol sequence (to the input)– Selecting a single alignment is the same as drawing a single sample from it– Selecting the most likely alignment is the same as deterministically always drawing the most probable

value from the distribution

37

The reason for suboptimality

/IY/

/B/

/F/

/IY/

Page 38: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• Instead of only selecting the most likely alignment, use the statistical expectation over all possible alignments

– Use the entire distribution of alignments– This will mitigate the issue of suboptimal selection of alignment

38

Averaging over all alignments

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 39: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• Using the linearity of expectation

– This reduces to finding the expected divergence at each input

∈ … 39

The expectation over all alignments

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 40: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• Using the linearity of expectation

– This reduces to finding the expected divergence at each input

∈ … 40

The expectation over all alignments

t 0 1 2 3 4 5 6 7 8

The probability of aligning the specific symbol s at time t,given that unaligned sequence and given the input sequence We need to be able to compute this

/IY/

/B/

/F/

/IY/

Page 41: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• is the total probability of all valid paths in the graph for target sequence that go through the symbol

(the th symbol in the sequence ) at time

• We will compute this using the “forward-backward” algorithm

41

A posteriori probabilities of symbols

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 42: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• can be decomposed as

• Where is a symbol that can follow in a sequence– Here it is either or

42

A posteriori probabilities of symbols

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 43: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• can be decomposed as

• Where is a symbol that can follow in a sequence– Here it is either or (red blocks in figure)– The equation literally says that after the blue block, either of the two

red arrows may be followed

43

A posteriori probabilities of symbols

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 44: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• can be decomposed as

• Where is a symbol that can follow in a sequence– Here it is either or (red blocks in figure)– The equation literally says that after the blue block, either of the

two red arrows may be followed44

A posteriori probabilities of symbols

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 45: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• can be decomposed as

• Using Bayes Rule

• The probability of the subgraph in the blue outline, times the conditional probability of the red-encircled subgraph, given the blue subgraph

45

A posteriori probabilities of symbols

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 46: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• can be decomposed as

• Using Bayes Rule

• For a recurrent network without feedback from the output we can make the conditional independence assumption:

46

A posteriori probabilities of symbols

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Assuming past output symbols do not directly feed back into the net

Page 47: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Conditional independence

• Dependency graph: Input sequence governs hidden variables

• Hidden variables govern output predictions , , individually• , , are conditionally independent given

• Since is deterministically derived from , , , are also conditionally independent given – This wouldn’t be true if the relation between and were not deterministic or

if is unknown, or if the s at any time went back into the net as inputs47

Page 48: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• We will call the first term the forward probability • We will call the second term the backward probability

48

A posteriori symbol probability

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 49: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• We will call the first term the forward probability • We will call the second term the backward probability

49

A posteriori symbol probability

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 50: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• The is the total probability of the subgraph shown– The total probability of all paths leading to the

alignment of to time 50

Computing : Forward algorithm

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 51: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

( )

: ∈ ( )

51

Computing : Forward algorithm

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

• Where is any symbol that is permitted to come before an and may include • is its row index, and can take values and in this example

Page 52: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

𝛼 𝑡, 𝑟 = 𝑃 𝑆 . . 𝑆 , 𝑠 = 𝑆 |𝐗

𝛼 3, 𝐼𝑌 = 𝛼 2, 𝐵 𝑦 + 𝛼 2, 𝐼𝑌 𝑦

𝛼 𝑡, 𝑟 = 𝛼(𝑡 − 1, 𝑞) 𝑌( )

: ∈ ( )

52

Computing : Forward algorithm

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

• Where is any symbol that is permitted to come before an and may include • is its row index, and can take values and in this example

Page 53: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• The is the total probability of the subgraph shown

53

Forward algorithm

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 54: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

54

Forward algorithm

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 55: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Forward algorithm

• Initialization:

• for

for

55

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 56: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Forward algorithm

• Initialization:

• for

for

• 𝛼(𝑡, 𝑙) = (𝛼 𝑡 − 1, 𝑙 + 𝛼 𝑡 − 1, 𝑙 − 1 )𝑦

56

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 57: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Forward algorithm

• Initialization:

• for

for

• 𝛼(𝑡, 𝑙) = (𝛼 𝑡 − 1, 𝑙 + 𝛼 𝑡 − 1, 𝑙 − 1 )𝑦

57

tt 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 58: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Forward algorithm

• Initialization:

• for

for

• 𝛼(𝑡, 𝑙) = (𝛼 𝑡 − 1, 𝑙 + 𝛼 𝑡 − 1, 𝑙 − 1 )𝑦

58

tt 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 59: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Forward algorithm

• Initialization:

• for

for

• 𝛼(𝑡, 𝑙) = (𝛼 𝑡 − 1, 𝑙 + 𝛼 𝑡 − 1, 𝑙 − 1 )𝑦

59

tt 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 60: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Forward algorithm

• Initialization:

• for

for

• 𝛼(𝑡, 𝑙) = (𝛼 𝑡 − 1, 𝑙 + 𝛼 𝑡 − 1, 𝑙 − 1 )𝑦

60

tt 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 61: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

In practice..

• The recursion

will generally underflow

• Instead we can do it in the log domain

– This can be computed entirely without underflow

61

Page 62: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Forward algorithm: Alternate statement

• The algorithm can also be stated as follows which separates the graph probability from the observation probability. This is needed to compute derivatives

• Initialization:

• for 𝛼(𝑡, 0) = 𝛼(𝑡 − 1,0)

for 𝑙 = 1 … 𝐾 − 1

• 𝛼(𝑡, 𝑙) = 𝛼 𝑡 − 1, 𝑙 + 𝛼 𝑡 − 1, 𝑙 − 1

62

tt 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 63: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• The probability of the entire symbol sequence is the alpha at the bottom right node

63

The final forward probability

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 64: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

SIMPLE FORWARD ALGORITHM#N is the number of symbols in the target output#S(i) is the ith symbol in target output#y(t,i) is the output of the network for the ith symbol at time t#T = length of input

#First create output tableFor i = 1:N

s(1:T,i) = y(1:T, S(i))

#The forward recursion# First, at t = 1alpha(1,1) = s(1,1)alpha(1,2:N) = 0for t = 2:T

alpha(t,1) = alpha(t-1,1)*s(t,1)for i = 2:N

alpha(t,i) = alpha(t-1,i-1) + alpha(t-1,i)alpha(t,i) *= s(t,i)

64

Can actually be done without explicitly composing the output table

Using 1..N and 1..T indexing, instead of 0..N-1, 0..T-1, for convenience of notation

Page 65: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

SIMPLE FORWARD ALGORITHM

#N is the number of symbols in the target output#S(i) is the ith symbol in target output#y(t,i) is the network output for the ith symbol at time t#T = length of input

#The forward recursion# First, at t = 1alpha(1,1) = y(1,S(1))alpha(1,2:N) = 0for t = 2:T

alpha(t,1) = alpha(t-1,1)*y(t,S(1))for i = 2:N

alpha(t,i) = alpha(t-1,i-1) + alpha(t-1,i)alpha(t,i) *= y(t,S(i))

65

Without explicitly composing the output table

Using 1..N and 1..T indexing, instead of 0..N-1, 0..T-1, for convenience of notation

Page 66: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• We will call the first term the forward probability • We will call the second term the backward probability

66

A posteriori symbol probability

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

We have seen how to compute this

Page 67: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• We will call the first term the forward probability • We will call the second term the backward probability

67

A posteriori symbol probability

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

We have seen how to compute this

Page 68: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• We will call the first term the forward probability • We will call the second term the backward probability

68

A posteriori symbol probability

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Lets look at this

Page 69: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• is the probability of the exposed subgraph, not including the orange shaded box

69

Bacward probability

tt 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 70: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

70

Backward probability

tt 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 71: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Backward probability

tt 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 72: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Backward probability

tt 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 73: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

73

Backward algorithm

tt 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 74: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

74

Backward algorithm

tt 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

• The is the total probability of the subgraph shown

• The terms at any time are defined recursively in terms of the terms at the next time

Page 75: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Backward algorithm

• Initialization:

• for

for

• ( ) ( )

75

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 76: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Backward algorithm

• Initialization:

• for

for

• ( ) ( )

76

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 77: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Backward algorithm

• Initialization:

• for

for

• ( ) ( )

77

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 78: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Backward algorithm

• Initialization:

• for

for

• ( ) ( )

78

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 79: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Backward algorithm

• Initialization:

• for

for

• ( ) ( )

79

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 80: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

SIMPLE BACKWARD ALGORITHM#N is the number of symbols in the target output#S(i) is the ith symbol in target output#y(t,i) is the output of the network for the ith symbol at time t#T = length of input

#First create output tableFor i = 1:N

s(1:T,i) = y(1:T, S(i))

#The backward recursion# First, at t = Tbeta(T,N) = 1beta(T,1:N-1) = 0for t = T-1 downto 1

beta(t,N) = beta(t+1,N)*s(t+1,N)for i = N-1 downto 1

beta(t,i) = beta(t+1,i)*s(t+1,i) + beta(t+1,i+1))*s(t+1,i+1)

80

Can actually be done without explicitly composing the output table

Using 1..N and 1..T indexing, instead of 0..N-1, 0..T-1, for convenience of notation

Page 81: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

BACKWARD ALGORITHM#N is the number of symbols in the target output#S(i) is the ith symbol in target output#y(t,i) is the output of the network for the ith symbol at time t#T = length of input

#The backward recursion# First, at t = Tbeta(T,N) = 1beta(T,1:N-1) = 0for t = T-1 downto 1

beta(t,N) = beta(t+1,N)*y(t+1,S(N))for i = N-1 downto 1

beta(t,i) = beta(t+1,i)*y(t+1,S(i)) + beta(t+1,i+1))*y(t+1,S(i+1))

81

Without explicitly composing the output table

Using 1..N and 1..T indexing, instead of 0..N-1, 0..T-1, for convenience of notation

Page 82: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• Some implementations of the backward algorithm will use the above formula

• Note that here the probability of the observation at t is also factored into beta

• It will have to be unfactored later (we’ll see how)82

Alternate Backward algorithm

tt 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 83: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• We will call the first term the forward probability

• We will call the second term the backward probability

83

The joint probability

We now can compute this

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 84: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• We will call the first term the forward probability • We will call the second term the backward probability

84

The joint probability

Backward algoForward algo

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 85: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• The posterior is given by

The posterior probability

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 86: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• Let the posterior be represented by

The posterior probability

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 87: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

COMPUTING POSTERIORS#N is the number of symbols in the target output#S(i) is the ith symbol in target output#y(t,i) is the output of the network for the ith symbol at time t#T = length of input

#Assuming the forward are completed firstalpha = forward(y, S) # forward probabilities computedbeta = backward(y, S) # backward probabilities computed

#Now compute the posteriorsfor t = 1:T

sumgamma(t) = 0for i = 1:N

gamma(t,i) = alpha(t,i) * beta(t,i)sumgamma(t) += gamma(t,i)

endfor i=1:N

gamma(t,i) = gamma(t,i) / sumgamma(t)

87Using 1..N and 1..T indexing, instead of 0..N-1, 0..T-1, for convenience of notation

Page 88: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• The posterior is given by

• We can also write this using the modified beta formula as (you will see this in papers)

( )

( )

The posterior probability

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 89: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

∈ …

( )

• The derivative of the divergence w.r.t the output Yt of the net at any time:

– Components will be non-zero only for symbols that occur in the training instance89

The expected divergence

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 90: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

∈ …

( )

• The derivative of the divergence w.r.t the output of the net at any time:

– Components will be non-zero only for symbols that occur in the training instance90

The expected divergence

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 91: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

∈ …

( )

• The derivative of the divergence w.r.t the output of the net at any time:

– Components will be non-zero only for symbols that occur in the training instance91

The expected divergence

Must compute these terms from here

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 92: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

∈ …

( )

• The derivative of the divergence w.r.t the output of the net at any time:

– Components will be non-zero only for symbols that occur in the training instance92

The expected divergence

Must compute these terms from here

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 93: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

∈ …

( )

• The derivative of the divergence w.r.t the output of the net at any time:

– Components will be non-zero only for symbols that occur in the training instance93

The expected divergence

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

The derivatives at both these locations must be summed to get

Page 94: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

∈ …

( )

• The derivative of the divergence w.r.t the output of the net at any time:

– Components will be non-zero only for symbols that occur in the training instance94

The expected divergence

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

The derivatives at both these locations must be summed to get

Page 95: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

∈ …

( )

• The derivative of the divergence w.r.t the output of the net at any time:

– Components will be non-zero only for symbols that occur in the training instance95

The expected divergence

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

The derivatives at both these locations must be summed to get

Page 96: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

∈ …

( )

• The derivative of the divergence w.r.t the output of the net at any time:

– Components will be non-zero only for symbols that occur in the training instancee96

The expected divergence

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

The derivatives at both these locations must be summed to get

The approximation is exact if we think of this as a maximum-likelihood estimate

Page 97: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

( )

• The derivative of the divergence w.r.t any particular output of the network must sum over all instances of that symbol in the target sequence

– E.g. the derivative w.r.t 𝑦 will sum over both rows representing /IY/ in the above figure 97

Derivative of the expected divergence

The derivatives at both these locations must be summed to get

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 98: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

COMPUTING DERIVATIVES

#N is the number of symbols in the target output#S(i) is the ith symbol in target output#y(t,i) is the output of the network for the ith symbol at time t#T = length of input

#Assuming the forward are completed firstalpha = forward(y, S) # forward probabilities computedbeta = backward(y, S) # backward probabilities computed

# Compute posteriors from alpha and betagamma = computeposteriors(alpha, beta)

#Compute derivativesfor t = 1:T

dy(t,1:L) = 0 # Initialize all derivatives at time t to 0for i = 1:N

dy(t,S(i)) -= gamma(t,i) / y(t,S(i))

98Using 1..N and 1..T indexing, instead of 0..N-1, 0..T-1, for convenience of notation

Page 99: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Overall training procedure for Seq2Seq case 1

• Problem: Given input and output sequences without alignment, train models

99

/B/ /IY/ /IY/

? ? ? ? ? ? ? ? ? ?/F/

Page 100: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Overall training procedure for Seq2Seq case 1

• Step 1: Setup the network– Typically many-layered LSTM

• Step 2: Initialize all parameters of the network

100

Page 101: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Overall Training: Forward pass

101

• Foreach training instance• Step 3: Forward pass. Pass the training instance through

the network and obtain all symbol probabilities at each time

Page 102: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

/B/

/IY/

/IY/

/F/

102

Overall training: Backward pass

• Foreach training instance• Step 3: Forward pass. Pass the training instance through

the network and obtain all symbol probabilities at each time

• Step 4: Construct the graph representing the specific symbol sequence in the instance. This may require havingmultiple rows of nodes with the same symbol scores

Page 103: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• Foreach training instance:– Step 5: Perform the forward backward algorithm

to compute and at each time, for each row of nodes in the graph. Compute .

– Step 6: Compute derivative of divergence for each

103

Overall training: Backward pass

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 104: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Overall training: Backward pass

• Foreach instance– Step 6: Compute derivative of divergence for each

• Step 7: Backpropagate and aggregate derivatives

over minibatch and update parameters

104

Page 105: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Story so far: CTC models• Sequence-to-sequence networks which irregularly output symbols can be

“decoded” by Viterbi decoding– Which assumes that a symbol is output at each time and merges adjacent

symbols

• They require alignment of the output to the symbol sequence for training– This alignment is generally not given

• Training can be performed by iteratively estimating the alignment by Viterbi-decoding and time-synchronous training

• Alternately, it can be performed by optimizing the expected error over all possible alignments– Posterior probabilities for the expectation can be computed using the forward

backward algorithm

105

Page 106: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

A key decoding problem

• Consider a problem where the output symbols are characters

• We have a decode: R R R E E E E D

• Is this the compressed symbol sequence RED or REED?

106

Page 107: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

We’ve seen this before

107

/AH/

/B/

/D/

/EH/

/IY/

/F/

/G/ /G/

/F/

/IY/

/D/

Cannot distinguish between an extended symbol andrepetitions of the symbol

/F/

• /G/ /F/ /F/ /IY/ /D/ or /G/ /F/ /IY/ /D/ ?

Page 108: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

A key decoding problem• We have a decode: R R R E E E E E D• Is this the symbol sequence RED or REED?

• Solution: Introduce an explicit extra symbol which serves to separate discrete versions of a symbol– A “blank” (represented by “-”)– RRR---EE---DDD = RED– RR-E--EED = REED– RR-R---EE---D-DD = RREDD– R-R-R---E-EDD-DDDD-D = RRREEDDD

• The next symbol at the end of a sequence of blanks is always a new character• When a symbol repeats, there must be at least one blank between the repetitions

• The symbol set recognized by the network must now include the extra blank symbol– Which too must be trained

108

Page 109: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

A key decoding problem• We have a decode: R R R E E E E E D• Is this the symbol sequence RED or REED?

• Solution: Introduce an explicit extra symbol which serves to separate discrete versions of a symbol– A “blank” (represented by “-”)– RRR---EE---DDD = RED– RR-E--EED = REED– RR-R---EE---D-DD = RREDD– R-R-R---E-EDD-DDDD-D = RRREEDDD

• The next symbol at the end of a sequence of blanks is always a new character• When a symbol repeats, there must be at least one blank between the repetitions

• The symbol set recognized by the network must now include the extra blank symbol– Which too must be trained

109

Page 110: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

The modified forward output

110

• Note the extra “blank” at the output

–

Page 111: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

–

The modified forward output

111

• Note the extra “blank” at the output

/B/ /IY/ /F/ /IY/

Page 112: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

–

The modified forward output

112

• Note the extra “blank” at the output

/B/ /IY/ /F/ /IY/

Page 113: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

–

The modified forward output

113

• Note the extra “blank” at the output

/B/ /IY/ /F/ /F/ /IY/

Page 114: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

114

Composing the graph for training

• The original method without blanks

• Changing the example to /B/ /IY/ /IY/ /F/ from /B/ /IY/ /F/ /IY/ for illustration

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/IY/

/F/

Page 115: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

/IY/

/B/

/IY/

115

Composing the graph for training

• With blanks• Note: a row of blanks between any two symbols• Also blanks at the very beginning and the very end

–

–

–

–

–

/F/

Page 116: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

/IY/

/B/

/F/

/IY/

116

Composing the graph for training

• Add edges such that all paths from initial node(s) to final node(s) unambiguously represent the target symbol sequence

–

–

–

–

–

Page 117: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

/IY/

/B/

/F/

/IY/

–

–

–

–

–

117

Composing the graph for training

• The first and last column are allowed to also end at initial and final blanks

Page 118: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

/IY/

/B/

/F/

/IY/

118

Composing the graph for training

• The first and last column are allowed to also end at initial and final blanks

• Skips are permitted across a blank, but only if the symbols on either side are different• Because a blank is mandatory between repetitions of a symbol but not

required between distinct symbols

–

–

–

–

–

Page 119: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Composing the graph

#N is the number of symbols in the target output#S(i) is the ith symbol in target output

#Compose an extended symbol sequence Sext from S, that has the blanks#in the appropriate place#Also keep track of whether an extended symbol Sext(j) is allowed to connect#directly to Sext(j-2) (instead of only to Sext(j-1)) or not

function [Sext,skipconnect] = extendedsequencewithblanks(S)j = 1for i = 1:N

Sext(j) = ‘b’ # blankskipconnect(j) = 0j = j+1

Sext(j) = S(i)if (i > 1 && S(i) != S(i-1))

skipconnect(j) = 1else

skipconnect(j) = 0j = j+1

endSext(j) = ‘b’skipconnect(j) = 0

return Sext, skipconnect

119Using 1..N and 1..T indexing, instead of 0..N-1, 0..T-1, for convenience of notation

Page 120: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

MODIFIED VITERBI ALIGNMENT WITH BLANKS

[Sext, skipconnect] = extendedsequencewithblanks(S)N = length(Sext) # length of extended sequence

# Viterbi starts hereBP(1,1) = -1Bscr(1,1) = y(1,Sext(1)) # BlankBscr(1,2) = y(1,Sext(2))Bscr(1,2:N) = -inftyfor t = 2:T

BP(t,1) = BP(t-1,1);Bscr(t,1) = Bscr(t-1,1)*y(t,Sext(1))for i = 1:N

if skipconnect(i)BP(t,i) = argmax_i(Bscr(t-1,i), Bscr(t-1,i-1), Bscr(t-1,i-2)

elseBP(t,i) = argmax_i(Bscr(t-1,i), Bscr(t-1,i-1))

Bscr(t,i) = Bscr(t-1,BP(t,i))*y(t,Sext(i))

# BacktraceAlignedSymbol(T) = Bscr(T,N) > Bscr(T,N-1) ? N, N-1;for t = T downto 1

AlignedSymbol(t-1) = BP(t,AlignedSymbol(t))

120Using 1..N and 1..T indexing, instead of 0..N-1, 0..T-1, for convenience of notation

Without explicit construction of output table

Example of using blanks for alignment: Viterbi alignment with blanks

Page 121: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Modified Forward Algorithm

• Initialization:–

121

/IY/

/B/

/F/

/IY/

–

–

–

–

–

t

Page 122: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Modified Forward Algorithm

• Iteration:( )

• If 𝑆 𝑟 = " − " or 𝑆 𝑟 = 𝑆 𝑟 − 2( )

• Otherwise 122

/IY/

/B/

/F/

/IY/

–

–

–

–

–

t𝛼 𝑡, 𝑟 = 𝛼(𝑡 − 1, 𝑞) 𝑌

( )

: ∈ ( )

Page 123: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Modified Forward Algorithm

• Iteration:( )

• If 𝑆 𝑟 = " − " or 𝑆 𝑟 = 𝑆 𝑟 − 2( )

• Otherwise 123

/IY/

/B/

/F/

/IY/

–

–

–

–

–

t

Page 124: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

FORWARD ALGORITHM (with blanks)

[Sext, skipconnect] = extendedsequencewithblanks(S)N = length(Sext) # Length of extended sequence

#The forward recursion# First, at t = 1alpha(1,1) = y(1,Sext(1)) #This is the blankalpha(1,2) = y(1,Sext(2))alpha(1,3:N) = 0for t = 2:T

alpha(t,1) = alpha(t-1,1)*y(t,Sext(1))for i = 2:N

alpha(t,i) = alpha(t-1,i-1) + alpha(t-1,i))if (skipconnect(i))

alpha(t,i) += alpha(t-1,i-2)alpha(t,i) *= y(t,Sext(i))

124

Without explicitly composing the output table

Using 1..N and 1..T indexing, instead of 0..N-1, 0..T-1, for convenience of notation

Page 125: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Modified Backward Algorithm

• Initialization:

125

/IY/

/B/

/F/

/IY/

–

–

–

–

–

t

Page 126: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Modified Backward Algorithm

• Iteration:( ) ( )

• If 𝑆 𝑟 = " − " or 𝑆 𝑟 = 𝑆 𝑟 + 2( ) ( ) ( )

• Otherwise126

/IY/

/B/

/F/

/IY/

–

–

–

–

–

t𝛽 𝑡, 𝑟 = 𝛽 𝑡 + 1, 𝑞 𝑦

: ∈ ( )

Page 127: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

BACKWARD ALGORITHM WITH BLANKS

[Sext, skipconnect] = extendedsequencewithblanks(S)N = length(Sext) # Length of extended sequence

#The backward recursion# First, at t = Tbeta(T,N) = 1beta(T,N-1) = 1beta(T,1:N-2) = 0for t = T-1 downto 1

beta(t,N) = beta(t+1,N)*y(t+1,Sext(N))for i = N-1 downto 1

beta(t,i) = beta(t+1,i)*y(t+1,Sext(i)) + beta(t+1,i+1))*y(t+1,Sext(i+1))if (i<N-2 && skipconnect(i+2))

beta(t,i) += beta(t+1,i+2)*y(t+1,Sext(i+2))

127

Without explicitly composing the output table

Using 1..N and 1..T indexing, instead of 0..N-1, 0..T-1, for convenience of notation

Page 128: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

The rest of the computation

• Posteriors and derivatives are computed exactly as before

• But using the extended graphs with blanks

128

Page 129: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

COMPUTING POSTERIORS

[Sext, skipconnect] = extendedsequencewithblanks(S)N = length(Sext) # Length of extended sequence

#Assuming the forward are completed firstalpha = forward(y, Sext) # forward probabilities computedbeta = backward(y, Sext) # backward probabilities computed

#Now compute the posteriorsfor t = 1:T

sumgamma(t) = 0for i = 1:N

gamma(t,i) = alpha(t,i) * beta(t,i)sumgamma(t) += gamma(t,i)

endfor i=1:N

gamma(t,i) = gamma(t,i) / sumgamma(t)

129Using 1..N and 1..T indexing, instead of 0..N-1, 0..T-1, for convenience of notation

Page 130: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

COMPUTING DERIVATIVES

[Sext, skipconnect] = extendedsequencewithblanks(S)N = length(Sext) # Length of extended sequence

#Assuming the forward are completed firstalpha = forward(y, Sext) # forward probabilities computedbeta = backward(y, Sext) # backward probabilities computed

# Compute posteriors from alpha and betagamma = computeposteriors(alpha, beta)

#Compute derivativesfor t = 1:T

dy(t,1:L) = 0 #Initialize all derivatives at time t to 0for i = 1:N

dy(t,Sext(i)) -= gamma(t,i) / y(t,Sext(i))

130Using 1..N and 1..T indexing, instead of 0..N-1, 0..T-1, for convenience of notation

Page 131: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Overall training procedure for Seq2Seq with blanks

• Problem: Given input and output sequences without alignment, train models

131

/B/ /IY/ /IY/

? ? ? ? ? ? ? ? ? ?/F/

Page 132: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Overall training procedure

• Step 1: Setup the network– Typically many-layered LSTM

• Step 2: Initialize all parameters of the network– Include a “blank” symbol in vocabulary

132

Page 133: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Overall Training: Forward pass

133

• Foreach training instance• Step 3: Forward pass. Pass the training instance through

the network and obtain all symbol probabilities at each time, including blanks

–

Page 134: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

134

Overall training: Backward pass

• Foreach training instance• Step 3: Forward pass. Pass the training instance through

the network and obtain all symbol probabilities at each time

• Step 4: Construct the graph representing the specific symbol sequence in the instance. Use appropriate connections if blanks are included

Page 135: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• Foreach training instance:– Step 5: Perform the forward backward algorithm to compute

and at each time, for each row of nodes in the graph using the modified forward-backward equations. Compute a posteriori probabilities from them

– Step 6: Compute derivative of divergence for each 135

Overall training: Backward pass

Page 136: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Overall training: Backward pass

• Foreach instance– Step 6: Compute derivative of divergence for each

• Step 7: Backpropagate and aggregate derivatives

over minibatch and update parameters

136

Page 137: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

CTC: Connectionist Temporal Classification

• The overall framework we saw is referred to as CTC

• Applies to models that output order-aligned, but time-asynchronous outputs

137

Page 138: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Returning to an old problem: Decoding

• The greedy decode computes its output by finding the most likely symbol at each time and merging repetitions in the sequence

• This is in fact a suboptimal decode that actually finds the most likely time-synchronous output sequence– Which is not necessarily the most likely order-synchronous sequence 138

/AH/

/B/

/D/

/EH/

/IY/

/F/

/G/

Page 139: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Greedy decodes are suboptimal

• Consider the following candidate decodes– R R – E E D (RED, 0.7)– R R – – E D (RED, 0.68)– R R E E E D (RED, 0.69)– T T E E E D (TED, 0.71)– T T – E E D (TED, 0.3)– T T – – E D (TED, 0.29)

• A greedy decode picks the most likely output: TED• A decode that considers the sum of all alignments of

the same final output will select RED• Which is more reasonable?

139

Page 140: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Greedy decodes are suboptimal• Consider the following candidate decodes

– R R – E E D (RED, 0.7)– R R – – E D (RED, 0.68)– R R E E E D (RED, 0.69)– T T E E E D (TED, 0.71)– T T – E E D (TED, 0.3)– T T – – E D (TED, 0.29)

• A greedy decode picks the most likely output: TED• A decode that considers the sum of all alignments of the

same final output will select RED• Which is more reasonable?• And yet, remarkably, greedy decoding can be surprisingly

effective, when using decoding with blanks140

Page 141: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

What a CTC system outputs

• Ref: Graves• Symbol outputs peak at the ends of the sounds

– Typical output: - - R - - - E - - -D– Model output naturally eliminates alignment ambiguities

• But this is still suboptimal.. 141

Page 142: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Actual objective of decoding

• Want to find most likely order-aligned symbol sequence– R E D

– What greedy decode finds: most likely time synchronous symbol sequence• – /R/ /R/ – – /EH//EH//D/

• Which must be compressed

• Find the order-aligned symbol sequence , given an input , that is most likely

142

Page 143: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

• The probability of the entire symbol sequence is the alpha at the bottom right node

143

Recall: The forward probability

t 0 1 2 3 4 5 6 7 8

/IY/

/B/

/F/

/IY/

Page 144: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Actual decoding objective

• Find the most likely (asynchronous) symbol sequence

• Unfortunately, explicit computation of this will require evaluate of an exponential number of symbol sequences

• Solution: Organize all possible symbol sequences as a (semi)tree

144

Page 145: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Hypothesis semi-tree

• The semi tree of hypotheses (assuming only 3 symbols in the vocabulary)• Every symbol connects to every symbol other than itself

– It also connects to a blank, which connects to every symbol including itself• The simple structure repeats recursively• Each node represents a unique (partial) symbol sequence! 145

Highlighted boxes representpossible symbols for first frame

Page 146: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

The decoding graph for the tree

• Graph with more than 2 symbols will be similar but much more cluttered and complicated

146

Page 147: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

The decoding graph for the tree

• The figure to the left is the tree, drawn in a vertical line• The graph is just the tree unrolled over time

– For a vocabulary of V symbols, every node connects out to V other nodes at the next time

• Every node in the graph represents a unique symbol sequence

147

Page 148: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

The decoding graph for the tree

• The forward score at the final time represents the full forward score for a unique symbol sequence (including sequences terminating in blanks)

• Select the symbol sequence with the largest alpha at the final time– Some sequences may have two alphas, one for the sequence itself, one for the

sequence followed by a blank– Add the alphas before selecting the most likely 148

𝛼(𝑆 𝑆 )

𝛼(𝑆 𝑆 )

𝛼(𝑆 −)

𝛼(𝑆 𝑆 )

𝛼(𝑆 𝑆 )

𝛼(𝑆 −)

𝛼(𝑆 )

𝛼(𝑆 )

𝛼(−)

Page 149: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Recall: Forward Algorithm

•

149

/IY/

/B/

/F/

/IY/

–

–

–

–

–

t

Page 150: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

The decoding graph for the tree

• The forward score at the final time represents the full forward score for a unique symbol sequence (including sequences terminating in blanks)

• Select the symbol sequence with the largest alpha– Sequences may two alphas, one for the sequence itself, one for the sequence

followed by a blank– Add the alphas before selecting the most likely 150

𝛼(𝑆 𝑆 )

𝛼(𝑆 𝑆 )

𝛼(𝑆 −)

𝛼(𝑆 𝑆 )

𝛼(𝑆 𝑆 )

𝛼(𝑆 −)

𝛼(𝑆 )

𝛼(𝑆 )

𝛼(−)

Page 151: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

CTC decoding

• This is the “theoretically correct” CTC decoder• In practice, the graph gets exponentially large very quickly• To prevent this pruning strategies are employed to keep the graph (and

computation) manageable– This may cause suboptimal decodes, however– The fact that CTC scores peak at symbol terminations minimizes the damage

due to pruning 151

Page 152: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Beamsearch Pseudocode Notes

• Retaining separate lists of paths and pathscores for paths terminating in blanks, and those terminating in valid symbols– Since blanks are special– Do not explicitly represent blanks in the partial decode strings

• Pseudocode takes liberties (particularly w.r.t null strings)– I.e. you must be careful if you convert this to code

• Key– PathScore : array of scores for paths ending with symbols– BlankPathScore : array of scores for paths ending with blanks– SymbolSet : A list of symbols not including the blank

152

Page 153: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

BEAM SEARCH

Global PathScore = [], BlankPathScore = []

# First time instant: Initialize paths with each of the symbols,# including blank, using score at time t=1NewPathsWithTerminalBlank, NewPathsWithTerminalSymbol, NewBlankPathScore, NewPathScore =

InitializePaths(SymbolSet, y[:,0])

# Subsequent time steps for t = 1:T

# Prune the collection down to the BeamWidthPathsWithTerminalBlank, PathsWithTerminalSymbol, BlankPathScore, PathScore =

Prune(NewPathsWithTerminalBlank, NewPathsWithTerminalSymbol, NewBlankPathScore, NewPathScore, BeamWidth)

# First extend paths by a blankNewPathsWithTerminalBlank, NewBlankPathScore = ExtendWithBlank(PathsWithTerminalBlank,

PathsWithTerminalSymbol, y[:,t])

# Next extend paths by a symbol NewPathsWithTerminalSymbol, NewPathScore = ExtendWithSymbol(PathsWithTerminalBlank,

PathsWithTerminalSymbol, SymbolSet, y[:,t])

end

# Merge identical paths differing only by the final blankMergedPaths, FinalPathScore = MergeIdenticalPaths(NewPathsWithTerminalBlank, NewBlankPathScore

NewPathsWithTerminalSymbol, NewPathScore)

# Pick best pathBestPath = argmax(FinalPathScore) # Find the path with the best score

153

Page 154: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

BEAM SEARCH

Global PathScore = [], BlankPathScore = []

# First time instant: Initialize paths with each of the symbols,# including blank, using score at time t=1NewPathsWithTerminalBlank, NewPathsWithTerminalSymbol, NewBlankPathScore, NewPathScore =

InitializePaths(SymbolSet, y[:,0])

# Subsequent time steps for t = 1:T

# Prune the collection down to the BeamWidthPathsWithTerminalBlank, PathsWithTerminalSymbol, PathScore, BlankPathScore =

Prune(NewPathsWithTerminalBlank, NewPathsWithTerminalSymbol, NewBlankPathScore, NewPathScore, BeamWidth)

# First extend paths by a blankNewPathsWithTerminalBlank, NewBlankPathScore = ExtendWithBlank(PathsWithTerminalBlank,

PathsWithTerminalSymbol, y[:,t])

# Next extend paths by a symbol NewPathsWithTerminalSymbol, NewPathScore = ExtendWithSymbol(PathsWithTerminalBlank,

PathsWithTerminalSymbol, SymbolSet, y[:,t])

end# Merge identical paths differing only by the final blankMergedPaths, FinalPathScore = MergeIdenticalPaths(NewPathsWithTerminalBlank, NewBlankPathScore

NewPathsWithTerminalSymbol, NewPathScore)

# Pick best pathBestPath = argmax(FinalPathScore) # Find the path with the best score

154

Page 155: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

BEAM SEARCH

Global PathScore = [], BlankPathScore = []

# First time instant: Initialize paths with each of the symbols,# including blank, using score at time t=1NewPathsWithTerminalBlank, NewPathsWithTerminalSymbol, NewBlankPathScore, NewPathScore =

InitializePaths(SymbolSet, y[:,0])

# Subsequent time steps for t = 1:T

# Prune the collection down to the BeamWidthPathsWithTerminalBlank, PathsWithTerminalSymbol, BlankPathScore, PathScore =

Prune(NewPathsWithTerminalBlank, NewPathsWithTerminalSymbol, NewBlankPathScore, NewPathScore, BeamWidth)

# First extend paths by a blankNewPathsWithTerminalBlank, NewBlankPathScore = ExtendWithBlank(PathsWithTerminalBlank,

PathsWithTerminalSymbol, y[:,t])

# Next extend paths by a symbol NewPathsWithTerminalSymbol, NewPathScore = ExtendWithSymbol(PathsWithTerminalBlank,

PathsWithTerminalSymbol, SymbolSet, y[:,t])

end# Merge identical paths differing only by the final blankMergedPaths, FinalPathScore = MergeIdenticalPaths(NewPathsWithTerminalBlank, NewBlankPathScore

NewPathsWithTerminalSymbol, NewPathScore)

# Pick best pathBestPath = argmax(FinalPathScore) # Find the path with the best score

155

xx

Page 156: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

BEAM SEARCH

Global PathScore = [], BlankPathScore = []

# First time instant: Initialize paths with each of the symbols,# including blank, using score at time t=1NewPathsWithTerminalBlank, NewPathsWithTerminalSymbol, NewBlankPathScore, NewPathScore =

InitializePaths(SymbolSet, y[:,0])

# Subsequent time steps for t = 1:T

# Prune the collection down to the BeamWidthPathsWithTerminalBlank, PathsWithTerminalSymbol, BlankPathScore, PathScore =

Prune(NewPathsWithTerminalBlank, NewPathsWithTerminalSymbol, NewBlankPathScore, NewPathScore, BeamWidth)

# First extend paths by a blankNewPathsWithTerminalBlank, NewBlankPathScore = ExtendWithBlank(PathsWithTerminalBlank,

PathsWithTerminalSymbol, y[:,t])

# Next extend paths by a symbol NewPathsWithTerminalSymbol, NewPathScore = ExtendWithSymbol(PathsWithTerminalBlank,

PathsWithTerminalSymbol, SymbolSet, y[:,t])

end# Merge identical paths differing only by the final blankMergedPaths, FinalPathScore = MergeIdenticalPaths(NewPathsWithTerminalBlank, NewBlankPathScore

NewPathsWithTerminalSymbol, NewPathScore)

# Pick best pathBestPath = argmax(FinalPathScore) # Find the path with the best score

156

Page 157: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

BEAM SEARCH

Global PathScore = [], BlankPathScore = []

# First time instant: Initialize paths with each of the symbols,# including blank, using score at time t=1NewPathsWithTerminalBlank, NewPathsWithTerminalSymbol, NewBlankPathScore, NewPathScore =

InitializePaths(SymbolSet, y[:,0])

# Subsequent time steps for t = 1:T

# Prune the collection down to the BeamWidthPathsWithTerminalBlank, PathsWithTerminalSymbol, BlankPathScore, PathScore =

Prune(NewPathsWithTerminalBlank, NewPathsWithTerminalSymbol, NewBlankPathScore, NewPathScore, BeamWidth)

# First extend paths by a blankNewPathsWithTerminalBlank, NewBlankPathScore = ExtendWithBlank(PathsWithTerminalBlank,

PathsWithTerminalSymbol, y[:,t])

# Next extend paths by a symbol NewPathsWithTerminalSymbol, NewPathScore = ExtendWithSymbol(PathsWithTerminalBlank,

PathsWithTerminalSymbol, SymbolSet, y[:,t])

end# Merge identical paths differing only by the final blankMergedPaths, FinalPathScore = MergeIdenticalPaths(NewPathsWithTerminalBlank, NewBlankPathScore

NewPathsWithTerminalSymbol, NewPathScore)

# Pick best pathBestPath = argmax(FinalPathScore) # Find the path with the best score

157

Page 158: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

BEAM SEARCH InitializePaths: FIRST TIME INSTANT

function InitializePaths(SymbolSet, y)

InitialBlankPathScore = [], InitialPathScore = []# First push the blank into a path-ending-with-blank stack. No symbol has been invoked yetpath = nullInitialBlankPathScore[path] = y[blank] # Score of blank at t=1InitialPathsWithFinalBlank = {path}

# Push rest of the symbols into a path-ending-with-symbol stackInitialPathsWithFinalSymbol = {}for c in SymbolSet # This is the entire symbol set, without the blank

path = cInitialPathScore[path] = y[c] # Score of symbol c at t=1InitialPathsWithFinalSymbol += path # Set addition

end

return InitialPathsWithFinalBlank, InitialPathsWithFinalSymbol, InitialBlankPathScore, InitialPathScore

158

Page 159: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

BEAM SEARCH: Extending with blanks

Global PathScore, BlankPathScore

function ExtendWithBlank(PathsWithTerminalBlank, PathsWithTerminalSymbol, y)UpdatedPathsWithTerminalBlank = {}UpdatedBlankPathScore = []# First work on paths with terminal blanks #(This represents transitions along horizontal trellis edges for blanks)for path in PathsWithTerminalBlank:

# Repeating a blank doesn’t change the symbol sequenceUpdatedPathsWithTerminalBlank += path # Set additionUpdatedBlankPathScore[path] = BlankPathScore[path]*y[blank]

end

# Then extend paths with terminal symbols by blanksfor path in PathsWithTerminalSymbol:

# If there is already an equivalent string in UpdatesPathsWithTerminalBlank# simply add the score. If not create a new entryif path in UpdatedPathsWithTerminalBlank

UpdatedBlankPathScore[path] += Pathscore[path]* y[blank]else

UpdatedPathsWithTerminalBlank += path # Set additionUpdatedBlankPathScore[path] = PathScore[path] * y[blank]

endend

return UpdatedPathsWithTerminalBlank,UpdatedBlankPathScore

159

Page 160: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

BEAM SEARCH: Extending with symbols

Global PathScore, BlankPathScore

function ExtendWithSymbol(PathsWithTerminalBlank, PathsWithTerminalSymbol, SymbolSet, y)UpdatedPathsWithTerminalSymbol = {}UpdatedPathScore = []

# First extend the paths terminating in blanks. This will always create a new sequencefor path in PathsWithTerminalBlank:

for c in SymbolSet: # SymbolSet does not include blanksnewpath = path + c # ConcatenationUpdatedPathsWithTerminalSymbol += newpath # Set additionUpdatedPathScore[newpath] = BlankPathScore[path] * y(c)

endend

# Next work on paths with terminal symbolsfor path in PathsWithTerminalSymbol:

# Extend the path with every symbol other than blankfor c in SymbolSet: # SymbolSet does not include blanks

newpath = (c == path[end]) ? path : path + c # Horizontal transitions don’t extend the sequenceif newpath in UpdatedPathsWithTerminalSymbol: # Already in list, merge paths

UpdatedPathScore[newpath] += PathScore[path] * y[c] else # Create new path

UpdatedPathsWithTerminalSymbol += newpath # Set additionUpdatedPathScore[newpath] = PathScore[path] * y[c]

endend

end

return UpdatedPathsWithTerminalSymbol,UpdatedPathScore

160

Page 161: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

BEAM SEARCH: Pruning low-scoring entries

Global PathScore, BlankPathScore

function Prune(PathsWithTerminalBlank, PathsWithTerminalSymbol, BlankPathScore, PathScore, BeamWidth)PrunedBlankPathScore = []PrunedPathScore = []# First gather all the relevant scoresi = 1for p in PathsWithTerminalBlank

scorelist[i] = BlankPathScore[p]i++

endfor p in PathsWithTerminalSymbol

scorelist[i] = PathScore[p]i++

end

# Sort and find cutoff score that retains exactly BeamWidth pathssort(scorelist) # In decreasing ordercutoff = BeamWidth < length(scorelist) ? scorelist[BeamWidth] : scorelist[end]

PrunedPathsWithTerminalBlank = {}for p in PathsWithTerminalBlank

if BlankPathScore[p] >= cutoffPrunedPathsWithTerminalBlank += p # Set additionPrunedBlankPathScore[p] = BlankPathScore[p]

endend

PrunedPathsWithTerminalSymbol = {}for p in PathsWithTerminalSymbol

if PathScore[p] >= cutoffPrunedPathsWithTerminalSymbol += p # Set additionPrunedPathScore[p] = PathScore[p]

endend

return PrunedPathsWithTerminalBlank, PrunedPathsWithTerminalSymbol, PrunedBlankPathScore, PrunedPathScore

161

Page 162: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

BEAM SEARCH: Merging final paths

# Note : not using global variable here

function MergeIdenticalPaths(PathsWithTerminalBlank, BlankPathScore,PathsWithTerminalSymbol, PathScore)

# All paths with terminal symbols will remainMergedPaths = PathsWithTerminalSymbolFinalPathScore = PathScore

# Paths with terminal blanks will contribute scores to existing identical paths from # PathsWithTerminalSymbol if present, or be included in the final set, otherwisefor p in PathsWithTerminalBlank

if p in MergedPathsFinalPathScore[p] += BlankPathScore[p]

elseMergedPaths += p # Set additionFinalPathScore[p] = BlankPathScore[p]

endend

return MergedPaths, FinalPathScore

162

Page 163: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Story so far: CTC models• Sequence-to-sequence networks which irregularly produce output

symbols can be trained by– Iteratively aligning the target output to the input and time-synchronous

training– Optimizing the expected error over all possible alignments: CTC training

• Distinct repetition of symbols can be disambiguated from repetitions representing the extended output of a single symbol by the introduction of blanks

• Decoding the models can be performed by– Best-path decoding, i.e. Viterbi decoding– Optimal CTC decoding based on the application of the forward algorithm to a

tree-structured representation of all possible output strings

163

Page 164: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

CTC caveats• The “blank” structure (with concurrent modifications to the

forward-backward equations) is only one way to deal with the problem of repeating symbols

• Possible variants:– Symbols partitioned into two or more sequential subunits

• No blanks are required, since subunits must be visited in order

– Symbol-specific blanks• Doubles the “vocabulary”

– CTC can use bidirectional recurrent nets• And frequently does

– Other variants possible..

164

Page 165: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Most common CTC applications

• Speech recognition– Speech in, phoneme sequence out– Speech in, character sequence (spelling out)

• Handwriting recognition

165

Page 166: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Speech recognition using Recurrent Nets

• Recurrent neural networks (with LSTMs) can be used to perform speech recognition– Input: Sequences of audio feature vectors– Output: Phonetic label of each vector

Time

X(t)

t=0

166

Page 167: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Speech recognition using Recurrent Nets

• Alternative: Directly output phoneme, character or word sequence

Time

X(t)

t=0

167

Page 168: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Next up: Attention models

168

Page 169: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

CNN-LSTM-DNN for speech recognition

• Ensembles of RNN/LSTM, DNN, & ConvNets (CNN) :

• T. Sainath, O. Vinyals, A. Senior, H. Sak. “Convolutional, Long Short-Term Memory, Fully Connected Deep Neural Networks,” ICASSP 2015.

169

Page 170: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Translating Videos to Natural Language Using Deep Recurrent Neural Networks

Translating Videos to Natural Language Using Deep Recurrent Neural Networks Subhashini Venugopalan, Huijun Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, Kate SaenkoNorth American Chapter of the Association for Computational Linguistics, Denver, Colorado, June 2015.

170

Page 171: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

171

Page 172: > v ] v P ^ µ v } ^ µ v u } o W } v v ] } v ] d u } o o ] ( ] ] } vdeeplearning.cs.cmu.edu/F20/document/slides/lec15... · 2020. 11. 28. · ^ ] u o µ } } µ u ] v p ] o Ç } u

Not explained

• Can be combined with CNNs– Lower-layer CNNs to extract features for RNN

• Can be used in tracking– Incremental prediction

172


Actuonix L16 Datasheet E - Amazon S3L16+Datasheet.pdf · Á Á Á X µ } v ] Æ X } u X W µ Z } U À } o µ u µ } U v µ } u µ v v } o µ } v ] Æ X } u

v µ ] } v ( } ^ µ v u o } Ç u v Z µ & } u ] v ^ u µ Ç W o µ d Z ^ µ v u o ... · 2020. 4. 22. · P v v u ] o } Ç } µ Z } µ P Z ^ u µ Ç W o µ X d Z µ v v v } P ]

Do W ] µ o µ u ] v D } - EMERJ · telar U o u v ] ( À } u } À } v } µ ]-sitos indispensáveis do ] µ o µ u ] v u } , do fumus boni iuris (requisitos

ñ u µ µ D ] v & v µ v ] v } µ µ v u ! u v W...o ] ] v µ ^ Z } } ] v P À o u µ í ì ô ï ~D ] r: µ ] o o î ì í ô } o o } ] } v v ] µ ] } v X µ } u v W > v u v µ v

} u u ] } z } µ } u u µ v ] Ç U ^ ^ } µ ] , o ( µ o ^ } µ v P Ks/ r í õ W o P W · 2020. 8. 7. · Ç ] P v ] v P Z ] o P U / } u u ] u Ç o ( } Z } À U } µ U v µ o X

, µ u v ] ] v > Z ] ] W^ / u o u v ] } v ] v , µ u v ] ] v Z } v W …s3-us-west-2.amazonaws.com/.../PSEA-Presentation.pdfW } u } ] v P ( µ o µ } ( µ Z } ] Ç ì ó l í ì l

u ] v v v } µ v } µ µ X - Scouting Event

u µ o Ì ] v ( } u i ] Ì Á ] v Ç Z Ì Ç µ ] P v ] µ ... · s Ì v ] l v í & } u µ o Ì ] v ( } u i ] Ì Á ] v Ç Z Ì Ç µ ] P v ] µ ] ' } } u } ] v v v ] Ï } u } Á

v } · v u U µ ] ( ( v P µ u v Ç ~ } À o } ] v P tt } l ( } o o ( µ v ] } v U v } i µ o u u

d u À ] o i o Ç Ç v U o ] v v ] l v u ] v i o ] v v À Ç v ... · r À } v v µ t v µ u ] v µ o À µ µ v v v µ o ] v v ] l v u ] v v < Ç W

^D ] v ] u µ u Z µ ] } µ u v } u hE ] v v o µ ] u v } ( } µ u v …...^D ] v ] u µ u Z µ ] } µ u v _ } u hE ] v v o µ ] u v } ( } µ u v } Z ] v Z Z D^ ^ } } o This handout

v Z ] ( ' U ^dW X U D X^ X ~'^ V ~KZ / V ~^ ] v W } P u ... · W v P µ Z W u ] v h v } o v ^ µ Z µ W v Ç ] u v v d Z D µ µ v D ^ ] u v µ Z : µ l ^ ] u ~ ] µ v } ] o ] > }

LGS Presentation Ne · KDD Z / > lZ d /> µ ] v U µ ] u v o ] v P } u v ] U o ] À } l v ] v P U u ] v v v Wh >/ tKZ

Z ] < } o U ^ µ P ] } < Ç o : v U µ Ç ^ µ P ] } : µ o Ç î ... · ^ } µ W P íd } o ^ Z À v µ v Æ v ] µ r ' } À v u v Á ]

o < o À P o } P o P ] v µ v · 2019-12-18 · µ v P µ X h v P v l o µ u ] P u u v U u v ] P µ U Z À ] ( ( t v u µ v P

v U Z v } o } P Ç v ] v v } À ] } v v µ o ] } o ] Ç ] v W ... Technology and... · õ ò W µ o ] W } o ] Ç W } µ P µ : } µ v o U s } o µ u í U E µ u í U î ì í ò õ

^ ð ñ í W K ] v P ^ Ç u µ µ u v î ì î ì D } µ o ô ^ Ç v Z ...•/ u P ] v Ç } µ [ ] v P Z r ( ] u o u v ] } v } ( } u µ µ •d Z ] v ( ] ~ l Ç v µ ~ l Ç U À o

Think Green, - Katun€¦ · Think Green, Think Remanufactured < µ v } ( ( À o v } u Á ] À ] Ç } ( µ } u v U ] v o µ ] v P < µ v

} µ µ ] µ v Ç v u ] µ Y v u µ v Z ] P Y } } i [ À v ] Y À ...eglise.catholique04.fr/wp-content/uploads/2019/06/... · ° t r s { s u } µ u v u ] i } µ P µ o ] u v U v [

E À o W / v u ] } P ] µ o µ · Á Á Á X ] µ } o X } u d Æ } î W K u µ À ! u µ u µ ] v v } } µ } v o µ À ] v ] u U } u } ] v u } Á Á Á X ] µ } o X } u P µ o P µ

µ Z d µ v v µ Ç - WHO...WKd^ A } µ o } Z } ] Z Ç ] Ç v } u V ZW^ A } u o Æ P ] } v o ] v Ç v } u ' s^ Æ } µ K µ } u > ] v l

v µ u } v ^ o } Ç } u } ] ] } v s ] o } v z } µ v P t } l ] v · ^ } µ ^ µ ï ð ~/ r: } ]

0LQLPXP 5HTXLUHPHQWV 6HQWLQHO 6DIH :RUN 3ODQQLQJ … · / v µ Ç D ] v ] u µ u Z µ ] u v U ^ v ] v o U ^ ( t } l W o v v ] v P v W o v K ] } v ^ Z u u } µ o

s } o µ u î W v À ] } v u v o / u u v Z } · ^ ] o ^ v < v } µ ~> v À ] } v ' } µ

E'& ' µ ] v } µ u v ( } K^, t o l ] v P rt } l ] v P ^ µ ......D Æ ] u µ u ] v v o } u v Z } o o } ~ Á ] P Z v ( } } ( o o u o } Ç U µ ] u v U À Z ] o U

Kia V14 New Map Features S4...d Z o } Á ( µ Á µ ] v P ] ( ( v ] } X / ( Ç } µ µ v u À ] } v ] Z u } v Á U Z d Z o } Á ( µ Á µ ] v P ] ( ( v ] } X / ( Ç } µ µ v u À

v } µ v ] µ Z Ç } Æ Ç Z o } } µ ] v ( } } À ] r í õ ] t v i ] } v M - ISPSW · 2020. 8. 2. · K v Z ] µ } ( v u ] u v P u v U µ v o Ç ] v v ^ } µ Z ] v } µ v ] v i

P R ] u o u v o f 7 ] v P À s u À µ Ç µ µ z u < f o À µ Ì µ · P À l ] À ] ] } o µ µ u l ] ] v Ç ( v f Ì f v R f v µ o µ v v ^z v ] l ] À ] l o _ µ } v µ v f

u u v ] P o o Z ( · • v l W ] Ì µ u ] µ v l Z o µ v P ] ' o o Z ( À u P v u ] u v µ Z µ Z ] v v ' o o Z ( µ ó ï ô / ï ' o

^ u ] ] > µ v Z W u ] µ u µ Ç v

E Á , u Z ] > ] v U . U v Z P ] K µ } v · 2020-04-07 · E Á , u Z ] u o } Ç u v ^ µ ] Ç U } v } u ] v > } D l / v ( } u } v µ µ. E Á , u Z ] u o } Ç u v ^ µ ] Ç U }

µ µ u v î ì î ì K ] v P ^ Ç u } u } v v v ^ µ µ...W } P u l } } l } E } Z } P u ] } o o Ç ] À i µ Ç } v ] l Z v } ] v µ ] } v } µ v } ] v ] v v } ( } P u ] v P Æ µ

} v v ] } v µ v u } ] ( ] } u } o u v v µ u o ] u ] v } o ...€¦ · } v v ] } v µ v u } ] ( ] } u } o u v v µ u o ] u ] v } o ] } ] ] } v W î ì í ô ¶ . Ô Ï Ë Ð Ñ Õ

ï ±physiquecarnotsupiv.blog.free.fr/public/Feuilles_exercices/Mecanique… · ^ µ W ^/ í r Æ ] Z Ç ] µ } Z v P ] µ µ u } µ À u v [ µ v } ] v u ] o ï± µ §

PropertyPRO Supporting Memorandum - final (clean) - 17022020 · W } ÇWZK ^ µ } ] v P D u } v µ u µ o ] v W } Ç / v ] µ W } ÇWZK ^ µ } ] v P D u } v µ u