A modern application of Linear Programming:Optimal transport – Part IThe optimal transport problem as a Linear Program

1/22

Disclaimers

You will not be tested on that, and this will not help you to understandbetter Linear Programming.

The goal is to present you a modern direction of research where LinearProgramming plays a role.

2/22

Disclaimers

You will not be tested on that, and this will not help you to understandbetter Linear Programming.

The goal is to present you a modern direction of research where LinearProgramming plays a role.

2/22

Teaser

The content of this lecture will explain how this movie was generated.

3/22

1. The optimal transport problem

2. A special case: Birkhoff Von Neumann theorem

3. Link with geometry: interpolating “shapes”

4/22

1. The optimal transport problem

An economic introduction 1

Coal mines extract coal, while factories need coal to operate. We want tofind the most efficient transportation plan for the coal from mines tofactories.

Mine Number Production Factory number Requirement1 12 1 102 8 2 53 3 3 154 7

As usual, values are made up.

We make the important assumption that the total number of coal extracted(30) is equal to the total number of coal used by the factories (30), eventhough the number of mines and factories are different.

5/22

An economic introduction 1

Coal mines extract coal, while factories need coal to operate. We want tofind the most efficient transportation plan for the coal from mines tofactories.

Mine Number Production Factory number Requirement1 12 1 102 8 2 53 3 3 154 7

As usual, values are made up.

We make the important assumption that the total number of coal extracted(30) is equal to the total number of coal used by the factories (30), eventhough the number of mines and factories are different.

5/22

An economic introduction 1

Coal mines extract coal, while factories need coal to operate. We want tofind the most efficient transportation plan for the coal from mines tofactories.

Mine Number Production Factory number Requirement1 12 1 102 8 2 53 3 3 154 7

As usual, values are made up.

We make the important assumption that the total number of coal extracted(30) is equal to the total number of coal used by the factories (30), eventhough the number of mines and factories are different.

5/22

An economic introduction 2

We assume that the cost to transport one unit of coal from mine i to factoryj is equal to Cij with the following data:

PPPPPPPPPMineFactory 1 2 3

1 7 5 82 1 3 153 16 18 34 5 2 11

Importantly, we assume that the cost is linear in the amount of masstransported.

6/22

An economic introduction 2

We assume that the cost to transport one unit of coal from mine i to factoryj is equal to Cij with the following data:

PPPPPPPPPMineFactory 1 2 3

1 7 5 82 1 3 153 16 18 34 5 2 11

Importantly, we assume that the cost is linear in the amount of masstransported.

6/22

The Linear Program: Decision variables

Let’s call Pij the amount of coal transported from mine i to factory j. Theseare the decision variables and they are non negative. There are 4× 3 = 12 ofthem.

The total cost is

4∑i=1

3∑j=1

PijCij = 7P11 + 5P12 + 8P13 + P21 + 3P22 + 15P23

+ 16P31 + 18P32 + 3P33 + 5P41 + 2P42 + 11P43.

7/22

The Linear Program: Decision variables

Let’s call Pij the amount of coal transported from mine i to factory j. Theseare the decision variables and they are non negative. There are 4× 3 = 12 ofthem.

The total cost is

4∑i=1

3∑j=1

PijCij = 7P11 + 5P12 + 8P13 + P21 + 3P22 + 15P23

+ 16P31 + 18P32 + 3P33 + 5P41 + 2P42 + 11P43.

7/22

The Linear Program: Constraints

All the production of Mine 1 should be shipped:

12︸︷︷︸Coal produced by Mine 1

= P11 + P12 + P13︸ ︷︷ ︸Coal shipped from Mine 1

And 3 similar constraints for the 3 remaining mines.

Factory 1 must receive its coal:

10︸︷︷︸Coal used by Factory 1

= P11 + P21 + P31 + P41︸ ︷︷ ︸Coal shipped to Factory 1

And 2 similar constraints for the two other factories.

8/22

The Linear Program: Constraints

All the production of Mine 1 should be shipped:

12︸︷︷︸Coal produced by Mine 1

= P11 + P12 + P13︸ ︷︷ ︸Coal shipped from Mine 1

And 3 similar constraints for the 3 remaining mines.

Factory 1 must receive its coal:

10︸︷︷︸Coal used by Factory 1

= P11 + P21 + P31 + P41︸ ︷︷ ︸Coal shipped to Factory 1

And 2 similar constraints for the two other factories.

8/22

The Linear Program: final result

Minimizing:

7P11 + 5P12 + 8P13 + P21 + 3P22 + 15P23+ 16P31 + 18P32 + 3P33 + 5P41 + 2P42 + 11P43

Under the constraints

Mines

P11 +P12 +P13 = 12

P21 +P22 +P23 = 8

P31 +P32 +P33 = 3

P41 +P42 +P43 = 7

Factories

P11 +P21 +P31 +P41 = 10

P12 +P22 +P32 +P42 = 5

P13 +P23 +P33 +P43 = 15

And the non negativity constraints Pij > 0 for i ∈ {1, 2, 3, 4} and j ∈ {1, 2, 3}.

9/22

The optimal transport: general case 1

You have n initial locations indexed by i and m final locations indexed by j.You have a number ai produced at location i, and a number bj consumed atlocation j. The cost of transporting one unit of mass from i to j is Cij.

We assume balance of mass condition:n∑i=1

ai =m∑j=1

bj.

The decision variables are Pij the amount of mass transported from i to j.

10/22

The optimal transport: general case 1

You have n initial locations indexed by i and m final locations indexed by j.You have a number ai produced at location i, and a number bj consumed atlocation j. The cost of transporting one unit of mass from i to j is Cij.

We assume balance of mass condition:n∑i=1

ai =m∑j=1

bj.

The decision variables are Pij the amount of mass transported from i to j.

10/22

The optimal transport: general case 1

You have n initial locations indexed by i and m final locations indexed by j.You have a number ai produced at location i, and a number bj consumed atlocation j. The cost of transporting one unit of mass from i to j is Cij.

We assume balance of mass condition:n∑i=1

ai =m∑j=1

bj.

The decision variables are Pij the amount of mass transported from i to j.

10/22

The optimal transport problem: general case 2

Optimal transport problemWe want to find the cheapest way to transport, that is we want to minimize

n∑i=1

m∑j=1

PijCij

subject to ∑mj=1 Pij = ai ∀i ∈ {1, 2, . . . ,n}∑ni=1 Pij = bj ∀j ∈ {1, 2, . . . ,m}

and the non negativity constraints Pij > 0 for all i, j.

There are n×m decision variables and n+m equality constraints. This is aproblem in standard equality form.

Note that the constraints are about the sums along rows and columns of thematrix P.

11/22

The optimal transport problem: general case 2

Optimal transport problemWe want to find the cheapest way to transport, that is we want to minimize

n∑i=1

m∑j=1

PijCij

subject to ∑mj=1 Pij = ai ∀i ∈ {1, 2, . . . ,n}∑ni=1 Pij = bj ∀j ∈ {1, 2, . . . ,m}

and the non negativity constraints Pij > 0 for all i, j.

There are n×m decision variables and n+m equality constraints. This is aproblem in standard equality form.

Note that the constraints are about the sums along rows and columns of thematrix P.

11/22

The dual problem

Let ui be the dual variable associated to the constraint that Mine i ships allof its coal. Let vj be the dual variable associated to the constraint thatFactory j gets all of its coal.

The dual is the maximization ofn∑i=1

aiui +m∑j=1

bjvj

= [Profit of the company]

subject to

ui + vj 6 Cij︸ ︷︷ ︸The company is cheaper than C

∀(i, j) ∈ {1, 2, . . . ,n} × {1, 2, . . . ,m}.

This is the LP solved by a transport company which charges ui to load oneunit of mass at location i and charges vj to unload one unit of mass atlocation j.

12/22

The dual problem

Let ui be the dual variable associated to the constraint that Mine i ships allof its coal. Let vj be the dual variable associated to the constraint thatFactory j gets all of its coal.

The dual is the maximization ofn∑i=1

aiui +m∑j=1

bjvj

= [Profit of the company]

subject to

ui + vj 6 Cij︸ ︷︷ ︸The company is cheaper than C

∀(i, j) ∈ {1, 2, . . . ,n} × {1, 2, . . . ,m}.

This is the LP solved by a transport company which charges ui to load oneunit of mass at location i and charges vj to unload one unit of mass atlocation j.

12/22

The dual problem

Let ui be the dual variable associated to the constraint that Mine i ships allof its coal. Let vj be the dual variable associated to the constraint thatFactory j gets all of its coal.

The dual is the maximization ofn∑i=1

aiui +m∑j=1

bjvj = [Profit of the company]

subject to

ui + vj 6 Cij︸ ︷︷ ︸The company is cheaper than C

∀(i, j) ∈ {1, 2, . . . ,n} × {1, 2, . . . ,m}.

This is the LP solved by a transport company which charges ui to load oneunit of mass at location i and charges vj to unload one unit of mass atlocation j.

12/22

Historical time frame

The optimal transport LP was proposed by Kantorovich in 1942 with aninfinite number of decision variables (sums are replaced by integrals). Healso introduced the dual problem. It was not known only a decade later inthe West.

If you ask the question: “when was duality theory invented in LinearProgramming?”, you have different answers.

• 1928. MinMax theorem by Von Neumann.• 1942. Duality in optimal transport by Kantorovich. Statement of a LPwith an infinite number of variables.

• 1947. Simplex algorithm, together with duality theory, by Dantzig.

13/22

Historical time frame

The optimal transport LP was proposed by Kantorovich in 1942 with aninfinite number of decision variables (sums are replaced by integrals). Healso introduced the dual problem. It was not known only a decade later inthe West.

If you ask the question: “when was duality theory invented in LinearProgramming?”, you have different answers.

• 1928. MinMax theorem by Von Neumann.• 1942. Duality in optimal transport by Kantorovich. Statement of a LPwith an infinite number of variables.

• 1947. Simplex algorithm, together with duality theory, by Dantzig.

13/22

Historical time frame

The optimal transport LP was proposed by Kantorovich in 1942 with aninfinite number of decision variables (sums are replaced by integrals). Healso introduced the dual problem. It was not known only a decade later inthe West.

If you ask the question: “when was duality theory invented in LinearProgramming?”, you have different answers.

• 1928. MinMax theorem by Von Neumann.• 1942. Duality in optimal transport by Kantorovich. Statement of a LPwith an infinite number of variables.

• 1947. Simplex algorithm, together with duality theory, by Dantzig.

13/22

2. A special case: Birkhoff VonNeumann theorem

Assignment problem

We consider an assignment problem, that is an optimal transport problemwhere:

• There is the same number of initial and final locations, n = m.

• The mass to be shipped from or to a location does not depend on thelocation, that is ai = bj = 1 for all i, j. By convention we set ai = bj = 1.

In short, we take n an integer, C ∈ Rn×n is the matrix with the transportationcost between locations and we look at the minimization of

n∑i=1

n∑j=1

PijCij

subject to ∑nj=1 Pij = 1 ∀i ∈ {1, 2, . . . ,n}∑ni=1 Pij = 1 ∀j ∈ {1, 2, . . . ,n}

and the non negativity constraints Pij > 0 for all i, j.

14/22

Assignment problem

We consider an assignment problem, that is an optimal transport problemwhere:

• There is the same number of initial and final locations, n = m.• The mass to be shipped from or to a location does not depend on thelocation, that is ai = bj = 1 for all i, j. By convention we set ai = bj = 1.

In short, we take n an integer, C ∈ Rn×n is the matrix with the transportationcost between locations and we look at the minimization of

n∑i=1

n∑j=1

PijCij

subject to ∑nj=1 Pij = 1 ∀i ∈ {1, 2, . . . ,n}∑ni=1 Pij = 1 ∀j ∈ {1, 2, . . . ,n}

and the non negativity constraints Pij > 0 for all i, j.

14/22

Assignment problem

We consider an assignment problem, that is an optimal transport problemwhere:

• There is the same number of initial and final locations, n = m.• The mass to be shipped from or to a location does not depend on thelocation, that is ai = bj = 1 for all i, j. By convention we set ai = bj = 1.

In short, we take n an integer, C ∈ Rn×n is the matrix with the transportationcost between locations and we look at the minimization of

n∑i=1

n∑j=1

PijCij

subject to ∑nj=1 Pij = 1 ∀i ∈ {1, 2, . . . ,n}∑ni=1 Pij = 1 ∀j ∈ {1, 2, . . . ,n}

and the non negativity constraints Pij > 0 for all i, j.

14/22

Permutations

It makes sense to assign each initial location to a single final location.

PermutationA permutation of {1, 2, . . . ,n} is a one to one function of {1, 2, . . . ,n} into{1, 2, . . . ,n}.

The set of permutation of {1, 2, . . . ,n} is denoted by Sn, there aren! = n× (n− 1)× . . .× 2× 1 of them.

If you have a permutation σ ∈ Sn, you can say that the mass at location i istransported only to location σ(i). This corresponds to the following feasibleP:

Pij ={1 if j = σ(i)0 otherwise.

15/22

Permutations

It makes sense to assign each initial location to a single final location.

PermutationA permutation of {1, 2, . . . ,n} is a one to one function of {1, 2, . . . ,n} into{1, 2, . . . ,n}.

The set of permutation of {1, 2, . . . ,n} is denoted by Sn, there aren! = n× (n− 1)× . . .× 2× 1 of them.

If you have a permutation σ ∈ Sn, you can say that the mass at location i istransported only to location σ(i). This corresponds to the following feasibleP:

Pij ={1 if j = σ(i)0 otherwise.

15/22

Permutations

It makes sense to assign each initial location to a single final location.

PermutationA permutation of {1, 2, . . . ,n} is a one to one function of {1, 2, . . . ,n} into{1, 2, . . . ,n}.

The set of permutation of {1, 2, . . . ,n} is denoted by Sn, there aren! = n× (n− 1)× . . .× 2× 1 of them.

If you have a permutation σ ∈ Sn, you can say that the mass at location i istransported only to location σ(i). This corresponds to the following feasibleP:

Pij ={1 if j = σ(i)0 otherwise.

15/22

Main result

Birkhoff Von Neumann theoremTake an assignment problem. Then a feasible solution P is a basic feasiblesolution (it can obtained by setting non basic variables to 0 in a feasibledictionary) if and only if there exists a permutation σ ∈ Sn such that

Pij ={1 if j = σ(i)0 otherwise.

∀(i, j) ∈ {1, 2, . . . ,n}2

This result makes the link between combinatorics (permutations) and linearprogramming. Indeed, the combinatorial problem

minσ∈Sn

n∑i=1

Ci,σ(i)

can be solved by finding the associated assignment problem which is alinear program.

16/22

Main result

Birkhoff Von Neumann theoremTake an assignment problem. Then a feasible solution P is a basic feasiblesolution (it can obtained by setting non basic variables to 0 in a feasibledictionary) if and only if there exists a permutation σ ∈ Sn such that

Pij ={1 if j = σ(i)0 otherwise.

∀(i, j) ∈ {1, 2, . . . ,n}2

This result makes the link between combinatorics (permutations) and linearprogramming. Indeed, the combinatorial problem

minσ∈Sn

n∑i=1

Ci,σ(i)

can be solved by finding the associated assignment problem which is alinear program.

16/22

Number of dictionaries

In particular, the number of dictionaries of the assignment problem is n!.

If for instance n = 20, so 400 decision variables and 40 constraints, thenumber of dictionaries is

20! ≃ 2.43× 1018.

Hopefully the simplex algorithm (in this case the “network simplex” whichexploits the structure of the problem) does not visit all dictionaries.

17/22

Number of dictionaries

In particular, the number of dictionaries of the assignment problem is n!.

If for instance n = 20, so 400 decision variables and 40 constraints, thenumber of dictionaries is

20! ≃ 2.43× 1018.

Hopefully the simplex algorithm (in this case the “network simplex” whichexploits the structure of the problem) does not visit all dictionaries.

17/22

3. Link with geometry:interpolating “shapes”

Geometric embedding

Initial locations andfinal locations, heren = m = 8: this isan assignment prob-lem.

Let xi be the location of the i-th point and yj being the one of the j-th point.The transportation cost between i and j is

Cij = ∥xi − yj∥2.

Other functions of the distance work but the quadratic case gives nicemovies.

Optimal P: by Birkhoff Von Neumann theorem each point is transportedonto a single one: a black line means that two points are connected.

18/22

Geometric embedding

Initial locations andfinal locations, heren = m = 8: this isan assignment prob-lem.

Let xi be the location of the i-th point and yj being the one of the j-th point.The transportation cost between i and j is

Cij = ∥xi − yj∥2.

Other functions of the distance work but the quadratic case gives nicemovies.

Optimal P: by Birkhoff Von Neumann theorem each point is transportedonto a single one: a black line means that two points are connected.

18/22

Geometric embedding

Initial locations andfinal locations, heren = m = 8: this isan assignment prob-lem.

Let xi be the location of the i-th point and yj being the one of the j-th point.The transportation cost between i and j is

Cij = ∥xi − yj∥2.

Other functions of the distance work but the quadratic case gives nicemovies.

Optimal P: by Birkhoff Von Neumann theorem each point is transportedonto a single one: a black line means that two points are connected.

18/22

Interpolation

If xi is transported onto yj, we have a point which travels at a constant speed.19/22

More points

n = 28,m = 32 and I have removed the black lines.20/22

With rendering

Even more points (n = 312, m = 394), shapes are closer, and rendering.

21/22

Next time

Numerical questions: how to solve (approximately) large scales LP comingfrom the optimal transport problem?

Applications:

• More on shape interpolation,• color transfer,• fluid mechanics,• etc.

22/22

Next time

Numerical questions: how to solve (approximately) large scales LP comingfrom the optimal transport problem?

Applications:

• More on shape interpolation,• color transfer,• fluid mechanics,• etc.

22/22