Optimization

Designing systems that optimize a set of metrics subject to constraints.

The optimization process

minimize f(x) subject to x ∈ X

Minimize f(x) can be replaced by maximize -f(x)

                                           +------+
                                       +---+Change<-----+
                                       |   |Design|     |no
                                       |   +------+     |
                   +---------+   +-----v-----+       +--+--+
Design         +--->  Initial+--->Evaluate   +------->Good?|
Specifications     |  Design |   |Performance|       |     |
                   +---------+   +-----------+       +--+--+
                                                        |
                                                        |yes
                                                        v
                                                      Final
                                                      Design

Optimize with respect to data, as intuition can be misleading.

Translating real world problems

There are some books describing the process to transform real world optimization problems to optimization problems

  • Optimization: Algorithms and Applications (R.K. Arora)
  • Optimization Concepts and Applications in Engineering (2nd edition, A. Keane, P. Nair)
  • Computational Approaches for Aerospace Design (P.Y. Papalambros, D.J. Wilde)
  • Principals of Optimal Design (Cambridge University Press, 2017)

Constraints

Constraints can be numerical (for example x ⋝ 4) but should always include the boundary (in the example 4). Excluding it (x > 4), the solution can move infinitely close to 4 without ever reaching it, which means no solution can be found.

Critical Points

A function f(x) may have a global minimum but may have multiple local minima. A zero derivative is a necessary condition for a local minimum but not a sufficient condition. The second derivative has to be >0, so the point is at the bottom of the bowl.