advanced optimization techniques for …irtic.uv.es/~ignacio/aot/01-introduction.pdf · c baltasar...

ADVANCED OPTIMIZATION TECHNIQUESFOR COMMUNICATIONS AND SIGNAL PROCESSING

MASTER ADVANCED SCIENCES OF MODERN TELECOMMUNICATIONS

FIRST QUARTER, COURSE 2009 - 2010

MOTIVATION AND INTRODUCTION

c©Baltasar Beferull Lozano

[email protected], http://www.uv.es/balbelo

Group of Information and Communication Systems (GSIC)

Inst. de Robotica y de Tecnologıas de la Informacion & las Comunicaciones (IRTIC)

Escuela Tecnica Superior de Ingenierıa

Universitat de Valencia

' $c©Baltasar Beferull Lozano - Advanced Optimization Techniques for Communications & Signal Processing, 2009 - 2010 (1st Quarter) 1

Why studying Optimization in the context of Telecommunications ?

• Use of Optimization has become ubiquitous in Communications and Signal Processing

•Many practical problems can be cast or converted into optimization-like problems

• Three different major waves in optimization theory and its applications:

– Linear Programming (Linear Convex Optimization), late 1940s

– General Convex Optimization, late 1980s

– Non-convex Optimization, 1990s

• Each wave has been followed by a period of “appreciation-application cycle”

– More people appreciate the use of a tool =⇒ more look for formulations in applic.

– Then, more work on its theory, efficient algorithms and softwares

– Then, more powerful the tools become =⇒ more people turn to appreciate its use

– This very beneficial cycle has transformed indeed each of these waves.

• A lot of research going on in Convex and (specially) in Non-Convex Optimization

(many research papers being published as we speak and several problems still open)

• Communication and Signal Processing systems benefit enormously from these waves

(there is a huge array of many success stories in many different applications)

& %


Communication Systems

• How to send information from one point to another over a given

(possibly heterogeneous) medium.

• How to send information for various source-destination configurations

& %


• Some of questions that arise:

– How to meet the requirements for a given application ?

(like accuracy, throughput, latency, jittering, power constraints, etc...)

– How to represent and use the information?

– How to utilize the communication medium?

– How to connect users?

– How to reach one point from another?

– How to coordinate among the transmitters and receivers?

– How to regulate competition among users?

– How to make the system robust to failures, attacks, variations, growth across

space and over time?

– How to allocate functionalities to layers/elements and connect them?

– Is Layering in Networking really necessary ?

(is there any connection at all with decomposing optimization problems ?)

& %


Layered Architecture

• Divide and conquer philosophy:

break the overall big problem into smaller ones with standardized interfaces

• Each layer provides services to upper layers & utilizes services given by lower layers

• Performance may not be “optimal”,

but makes the architecture simple and flexible.

APPLICATION

PRESENTATION

SESSION

TRANSPORT

NETWORK

LINK

PHYSICAL

& %


Point-to-Point Communication Channels

Source SourceCoder

ChannelCoder Modulator

Destination SourceDecoder

Channel Decoder Demodulator

CHANNEL

Analogor

Digital

Continuousor Discrete

Time

Digital

DiscreteTime

DiscreteTime

Digital

ContinuousTime

Analog

Noise

Interferences

Distortion

Random effects

Detectar datos modulados digitalmente

ContinuousTime

AnalogDigital

DiscreteTime

DiscreteTime

Digital

Continuousor Discrete

Time

Analogor

Digital

• Compress analog signals into digital data

• Add redundancy to protect against channel impairments

•Map digital data onto physical waveforms suitable for the medium

& %


• Some of the questions that arise:

– How to describe the channel and estimate its characteristics

(twisted pair, coaxial cable, optic fiber, radio, acoustic, storage)?

– How fast can data be sent reliably? (capacity problem)

– How to compress signals efficiently ? (source coding problem)

– Are there any practical results of duality between source & channel coding ?

– How to design efficient decoding algorithms at the receiver ?

– How to mitigate noise (thermal noise, impulse noise ...) ?

– How to manage interference ?

(from other users, from reflections, among symbols)

– How to use optimally the communication resources efficiently?

(time, frequency,engineering design parameters)

– How to design appropriate signal processing blocks ?

∗ Digital filters following certain criteria (absolute error bounds, equalization,...)

∗ Regularization and Denoising methods

∗ Estimation methods

∗ Sampling and Consistent signal reconstruction

& %


Communication Networks (multi-point-to-multi-point communications)

• Not necessarily a direct link, but a networked communication system

•Many more interesting questions than in point-to-point communications

• Links and nodes with various operational constraints

& %


• Some of the questions that arise:

– Fixed or dynamic topology?

– Who are transceivers and who are relays?

– Direct link or multi-hop architecture ?

– Circuit switch or packet switch or something else?

– How to divide into (possibly different types of ) subnetworks?

– End-to-end control or hop-by-hop control?

– How to get from one point to another?

– How to monitor and adjust overall state of the network?

– How to ensure accurate, secure, timely, and usable transfer of information across

space among competing users?

& %


Tools in the context of Communications and Signal Processing

• Experimental tools

– Empirical data from field trials and deployments (useful for modeling)

– Computer simulations

– Test-bed implementations and validations

• Analytical tools

– Information theory, coding theory, communication theory

– Digital Signal Processing algorithms

– Mathematical models of networks

– Queuing theory and other probabilistic tools

– Systems control theory, graph theory, game theory, economics modelling,

physics/biology modelling...

– Optimization theory and distributed algorithms

& %


Optimization

• Optimization variables: x ∈ C.

• Constant parameters describe objective function f(x) and constraint set C

• Algorithms search for optimal solution by moving acrosss constraint set

& %


• How to describe correctly the constraint set?

• Can the problem be solved globally and uniquely?

•What kind of characteristic properties does the solution have?

• How does the solution relate to another optimization problem?

• Can we numerically solve it in an efficient, robust, and distributed way?

• Can we optimize multiple objectives simultaneously?

• Can we optimize over a sequence of time instances?

• Can we find the problem for a given solution (reverse engineering) ?

& %


Many Applications in various disciplines

• Both Theory and Algorithms of Optimization are extremely powerful

– Communication systems

– Other information science areas: signal/image/video processing, systems control,

algorithms, graphics, data analysis, theoretical computer science ...

– Other engineering disciplines: aerospace, mechanical, chemical, civil,

transportation, computer architecture, analog circuit design ...

– Physics, chemistry, biology ...

– Economics, finance, management ...

– Analysis, probability, statistics, differential equations ...

& %


Three major paradigms in Optimization

• Linear Programming (LP)

– Widely known

– Easy to solve (simplex method, etc...)

– Powerful in various applications, as for example:

∗ Digital filter design

∗ Network flow problems (e.g. multicommodity flow solutions)

• Non-linear Convex Optimization

– Watershed between easy and hard optimization problems (modified view)

– Local optimality is also global optimality

– Lagrange duality theory well developed

– A lot known about the problem and solution structures (KKT conditions, etc...)

– Primal/Dual decomposition techniques are available

(favoring distributed implementation)

– Efficient and Reliable algorithms to compute numerically the solutions

(Interior-point, Gradient Descent, Alternating Projections, Semi-definite Prog., ...)

& %


– Surprisingly extremely powerful in several applications, as for example:

∗Water-filling solutions

∗ Robust Tranceiver Design (i.e. Robust beamforming)

∗ Network Utility Maximization (NUM)

∗Multi-user MIMO system design

• Non-Convex Optimization

– It involves new challenges

(non-convex objective functions and discrete integer constraints → NP-Hard, ...)

– Many interesting recent and on-going developments

∗ Geometric Programming (can be converted to equivalent convex problem)

∗ Semidefinite Programming Relaxations, Sum-of-squares relaxation (SOS)

∗ Branch-and-Bound (BB) methods

– Powerful in several applications, as for example:

∗Wireless Network Power Control

∗ Non-concave Network Utility Function (e.g. Internet congestion control)

∗Maximum-Likelihood Decoding, Multi-user Detection.

∗ Duality results between Channel Capacity & Rate-Distortion with side info

& %


The Three meanings of Optimization

PROBLEMS SOLUTIONS

OPTIMIZATION

OPTIMIZATION

OPTIMIZATION

THEORIES

• From problems to solutions:

Recognize & Formulate a problem as an optimization problem and getting solutions

– One needs to identify/recognize the type of problem: convex, non-convex, etc...

(in several scenarios, this is not as easy as one may think...)

• From solutions to problems:

Interpret a given solution as an optimizer/algorithm for an optimization problem

• Building underlying theory to understand better structures and relations between

different problems and solutions.

& %


• Very different uses of optimization (Optimization goes well beyond Optimizing!)

– Modeling

– Architecture

– Reverse engineering

– Optimizing some performance criterion

– Fairness

– Robustness

• Applications in communication systems also stimulate new developments in

optimization theory and algorithms

& %


What is this course all about ?

• Goal:

Study of optimization tools in Communications, Networking and Signal Processing

– Acquire correctly the mentality and language of Optimization.

(surprisingly, this can be useful for other studies you may engage in later on)

– Formulate communication & signal processing problems as optimization problems.

– Understanding underlying theory, concepts and properties of optimization tools

– Design, implement and simulate practical (centralized & distributed) algorithms

to solve optimization problems.

– Analyze the structures/decompositions of problems and solutions, as well as the

relationship between different problems

• The course will cover both classic results as well as more recent results

(over a wide range of problems)

•We will try to present the appropriate background material “just-in-time”

• Training also the ability to do original research in academia or industry

& %


• Of course, I also expect you to:

– Be interested in Communication systems

– Enjoy the class, have fun and be willing to learn a lot!

– Become a real optimizer

(i.e. thinking like an optimizer, as opposed to the try-and-error thinking)

– (Ideally) Turn your final project into conference publications, part of PhD.

dissertations, M.Sc. theses, etc... (thinking optimistically ;)) )

& %


What this course is NOT about

• Not a Math course on Convex Analysis

(we will not do many rigorous proofs, and will skip also many proofs)

• Not an OR course on Non-linear Optimization

(we will cover various additional topics + connection to applications)

• Not a EE course on Digital Communications course

(we will cover only selected topics, other Master courses cover Digital Comm.)

• Not an EE course on Networking

(we will cover only selected topics, other Master courses cover Networking)

• Not a CS course on Algorithms and Complexity

(we will not do an extensive computational complexity analysis)

•What you should NOT do:

– Be lazy =⇒ direct path to failure in this course

& %


Approximate Outline of the Course

• Introduction, Motivation and Course Overview

• Theory and Algorithms:

– Convex Sets and Convex Functions

– Classification of Convex problems: LP, QP, SOCP, SDP, GP

– Convex Optimization and Lagrange Duality

– Pareto Optimization

– Dynamic Programming and Sequential Optimization

– Geometric Programming for Communication Systems

– Gradient Optimization Algorithms

– Interior Point Optimization Algorithms

– Alternating Projections & Composite Mappings

– Decomposition Optimization Methods/Structures and Distributed Algorithms

– Non-convex Optimization

& %


• Applications (interlaced between theoretical lectures):

– Approximation and fitting

– Waterfilling solutions

– Network Flow Problems

– Statistical Estimation and Classification

– Robust Beamforming

– Transceiver Design for MIMO Communications

– Power Control Optimization in Wireless systems

– Multi-user Maximum-Likelihood Detection and Decoding

– Code duality between rate-distortion and channel capacity

– Network Utility Maximization

– Internet TCP Congestion Control

& %


Course requirements and Grading

• Homeworks (20% of grade)

– Although in-group discussion is ok, try to solve all the problems individually

– This will be your best possible preparation for the final exam

– Problems will not be plug-and-chug, will require time and thinking

(but you will hopefully learn a lot!!)

– Problems may involve analytical and/or programming tasks

– Goal: learn well the concepts, acquire intuition and interpret solutions correctly

• Project (40% of grade)

– What is the point of learning materials if you can not use it ?

– A list of topics and material will be provided (students can also propose topics ;) )

– Individual projects (2 projects can collectively tackle a large problem)

– Either Original research or extensive comparative study between existing solutions

– Presentation (15%) during last week of trimester

– 15− 20 page Report (25%) due last day of class (content + presentation evaluated)

– There will be several project progress meetings to interact

& %


• Final exam (40% of grade)

– Set of problems and questions of similar level to the homework assignments.

– Final exam will take place during the last week of class.

& %


Other details: course prerequisites and class material

• Course Prerequisites:

– Algebra

– Calculus

– Probability

– Random Processes

– Basic Digital Communications and Signal Processing

– Matlab Programming

• Not required but would be a plus:

– Linear programming

– Communication Networks

& %


• Class material and Web access:

– Lecture slides, which will be distributed periodically

(some material borrowed from Prof. S. Boyd and Prof. L. Vandenberghe)

– Homework assignments

– Tutorials (including some Refreshers)

– Journal papers

– Web site:

http://pizarra.uv.es

• Several Refreshers to be posted at http://pizarra.uv.es

– Directory “TUTORIALS”

– Refreshers on Algebra over the real and complex domain.

– You should start reading both of them to refresh all concepts

(From next class on, we will assume you know them)

– Matlab tutorials to try to get you familiar with Matlab (in case you are not).

• Professors Ignacio Garcıa and Rafael Sebastian

• Schedule of classes: Wednesdays (10:30 - 13:30)

& %




Bibliography

• There is not a single book covering all the material of the course

• Basic Bibliography:

– S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press

2004. Available online at http://www.stanford.edu/~boyd/cvxbook/bv cvxbook.pdf

Errata list available at http://www.stanford.edu/~boyd/cvxbook/cvxbook errata.html

• Complementary Bibliography:

– D. P. Bertsekas, Nonlinear Programming, Athena Scientific, 2nd Edition, 1999.

– D. G. Luenberger, Linear and Non-Linear Programming, Springer, 2003.

– D. P. Bertsekas, Network Optimization: Continuous and Discrete Models

(Optimization, Computation, and Control), Athena Scientific, 1998.

– D. P. Bertsekas, Convex Analysis and Optimization, Athena Scientific, 2003.

– M. Chiang, Geometric programming for communication systems, Short

monograph in Foundations and Trends in Communications and Information

Theory, vol. 2, no. 1-2, pp. 1-154, July 2005.

– D. P. Bertsekas, Dynamic Programming and Optimal Control, Volume I, Athena

Scientific, 2007.

& %

http://www.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf

http://www.stanford.edu/~boyd/cvxbook/cvxbook_errata.html


– G. L. Nemhauser, L. A. Wolsey, Integer and Combinatorial Optimization,

Wiley-Interscience (series in discrete mathematics and optimization), 1999.

– L. A. Wolsey, Integer Programming, Wiley-Interscience (series in discrete

mathematics and optimization), 1998.

– T. K. Moon, W. C. Stirling, Mathematical Methods and Algorithms for Signal

Processing, Prentice Hall, 2000.

– W. Kocay, D. L. Kreher, Graphs, Algorithms and Optimization, Chapman &

Hall/CRC Press, 2005.

• There will be also periodical reading assignments from some of these books.

GET READY FOR A LOT OF FUN!

Reading Assignment:

Two Refreshers on Algebra + Appendix A

(Boyd & Vandenberghe’s Book “Convex Optimization”)

& %

advanced optimization techniques for …irtic.uv.es/~ignacio/aot/01-introduction.pdf · c baltasar...

Documents