Quadratic Forms

a quadratic form on ℝ^𝑛 is a function 𝑄 defined on ℝ^𝑛whose value at vector 𝑣∊ℝ^𝑛can be computed by:
- 𝑄(𝑣) = 𝑣^𝑇𝐴𝑣
where:
- 𝐴 - symmetric matrix (also called the matrix of the quadratic form)
is a polynomial where all terms are of degree two (e.g. 𝑓(𝑥,𝑦) = 𝑥² + 3𝑥𝑦 + 4𝑦²), where 𝑣=(𝑥,𝑦) ∊ ℝ²)
is a special case of a bilinear form 𝐵: 𝑉⨯𝑉→𝐹 where the two sets of vectors 𝑣,𝑢∊𝑉 are equal (i.e. 𝑣=𝑢):
- for example, consider:
  - bilinear form 𝐵(𝑣,𝑢) = 𝑣₁𝑢₁ + 2𝑣₁𝑢₂ + 3𝑣₂𝑢₁+ 4𝑣₂𝑢₂
  - quadratic form 𝑄(𝑣) = (𝑣₁)²+ 5𝑣₁𝑣₂ + 4(𝑣₂)²
  - thus 𝑄(𝑣) = 𝐵(𝑣,𝑣)
not to be confused with a quadratic equation, which has only one variable and includes terms of degree two or less:
- for example, 𝑓(𝑥) = 𝑥²+ 5𝑥 + 4, is a quadratic equation NOT a quadratic form

Quadratic Form - Examples

Simplest Quadratic Form

The simplest quadratic form is:

𝑄(𝑥) = 𝑥^𝑇𝐼𝑥 = ||𝑥||²

Other Examples

from matrix form to function form

without cross-product terms

with cross-product terms

from function form to matrix form

𝑄(𝑥) = 5𝑥₁² + 3𝑥₂² + 3𝑥₃² - 𝑥₁𝑥₂ + 8𝑥₂𝑥₃

Quadratic Form - Changing Basis such that its Quadratic Form has no Cross-Products

Proof that any symmetric matrix 𝐴 can be rewritten to 𝐷 such that its quadratic form has no cross-products (see also Principal Axis Theorem below)

Click here to expand...

Change of variable/basis is an equation of the form:

𝑥 = 𝑃𝑦
𝑦 = 𝑃^-1𝑥

𝑦 is the coordinate vector of 𝑥 relative to the basis of ℝ^𝑛 determined by the columns of 𝑃

If the change of variable is made in a quadratic form 𝑥^𝑇𝐴𝑥

𝑥^𝑇𝐴𝑥 = 𝑥^𝑇𝐴𝑥
𝑥^𝑇𝐴𝑥 = (𝑃𝑦)^𝑇𝐴𝑃𝑦
𝑥^𝑇𝐴𝑥 = 𝑦^𝑇𝑃^𝑇𝐴𝑃𝑦
𝑥^𝑇𝐴𝑥 = 𝑦^𝑇𝐷𝑦 # Since 𝐴 is symmetric it guarantees an orthogonal matrix 𝑃 such that 𝑃^𝑇𝐴𝑃 is a diagonal matrix 𝐷 (see proof here and Principal Axis Theorem below)

Example

Click here to expand...

make a change of variable that transforms the quadratic form 𝑥^𝑇𝐴𝑥 into a quadratic form with no cross-product term

orthogonally diagonalize 𝐴

eigenvalues and eigenvectors of 𝐴
let
then
𝐴 = 𝑃𝐷𝑃^𝑇
and
𝐷 = 𝑃^-1𝐴𝑃 = 𝑃^𝑇𝐴𝑃

a suitable change of variable is:

𝑥 = 𝑃𝑦

where:

then:

𝑥₁² - 8𝑥₁𝑥₂ - 5𝑥₂² = 𝑥^𝑇𝐴𝑥
𝑥₁² - 8𝑥₁𝑥₂ - 5𝑥₂² = (𝑃𝑦)^𝑇𝐴𝑃𝑦
𝑥₁² - 8𝑥₁𝑥₂ - 5𝑥₂² = 𝑦^𝑇(𝑃^𝑇𝐴𝑃)𝑦
𝑥₁² - 8𝑥₁𝑥₂ - 5𝑥₂² = 𝑦^𝑇𝐷𝑦
𝑥₁² - 8𝑥₁𝑥₂ - 5𝑥₂²= 3𝑦₁² - 7𝑦₂²

𝑄(𝑥) for 𝑥 = (2,-2)

solve 𝑥^𝑇𝐴𝑥:

𝑥^𝑇𝐴𝑥 = 𝑥₁² - 8𝑥₁𝑥₂ - 5𝑥₂²
𝑥^𝑇𝐴𝑥 = 2² - 8(2)(-2) - 5(-2)²
𝑥^𝑇𝐴𝑥 = 16

find 𝑦 coordinates for 𝑥 = (2,-2):

solve 𝑦^𝑇𝐷𝑦:

𝑦^𝑇𝐷𝑦 = 3𝑦₁² - 7𝑦₂²
𝑦^𝑇𝐷𝑦 = 3(6/√5)² - 7(-2/√5)²
𝑦^𝑇𝐷𝑦 = 16

Principal Axis/Axes Theorem - & Geometric View

Click here to expand...

Principal Axis Theorem

The principal Axis Theorem states that the principal axes are perpendicular

Let 𝐴 be an 𝑛✕𝑛 symmetric matrix. Then there is an orthogonal change of variable 𝑥 = 𝑃𝑦 that transforms the quadratic form 𝑥^𝑇𝐴𝑥 into a quadratic form 𝑦^𝑇𝐷𝑦 with no cross-product terms

Principal Axis Theorem - Geometric View

Say: 𝑄(𝑥) = 𝑥^𝑇𝐴𝑥 and 𝐴 is a 2✕2 symmetric matrix

Then the set of all 𝑥's that satisfy

𝑄(𝑥) = 𝑥^𝑇𝐴𝑥 = 𝑐

Either corresponds to a graph looking like a:

ellipse (circle)
hyperbola
2 intersecting lines
single point
no points at all

If 𝐴 is a diagonal matrix then the graph is in "standard form"

If 𝐴 is NOT a diagonal matrix then the graph is rotated out of "standard form"

𝑎𝑥₁² - 𝑏𝑥₁𝑥₂ + 𝑐𝑥₂² = 1

𝑎𝑥₁² - 𝑏𝑥₁𝑥₂ - 𝑐𝑥₂² = 1

Quadratic Form - Matrix Form Duality & Classifying Quadratic Functions

Click here to expand...

see Positive/Negative Definite/Semi-Definite Indefinite Matrix

If Quadratic Form 𝑄(𝑥)	If Matrix Form 𝐴 (whose quadratic form is 𝑥^𝑇𝐴𝑥)	then 𝑄(𝑥) and 𝐴 is	Class
if 𝑄(𝑥) > 0 for all 𝑥 ≠ 0	if eigenvalues of 𝐴 are all positive	then 𝑄(𝑥) and 𝐴 is	positive definite	𝑄(𝑥) is strictly convex function
if 𝑄(𝑥) < 0 for all 𝑥 ≠ 0	if eigenvalues of 𝐴 are all negative	then 𝑄(𝑥) and 𝐴 is	negative definite	𝑄(𝑥) is strictly concave function
if 𝑄(𝑥) can be positive and negative	if eigenvalues of 𝐴 are both positive and negative	then 𝑄(𝑥) and 𝐴 is	indefinite	𝑄(𝑥) is neither convex nor concave
if 𝑄(𝑥) ≥ 0 for all 𝑥	if eigenvalues of 𝐴 are all positive or zero	then 𝑄(𝑥) and 𝐴 is	positive semi-definite	𝑄(𝑥) is a convex function
if 𝑄(𝑥) ≤ 0 for all 𝑥	if eigenvalues of 𝐴 are all negative or zero	then 𝑄(𝑥) and 𝐴 is	negative semi-definite	𝑄(𝑥) is a concave function

Quadratic Matrix Duality & Classification

by the principal axis theorem, there exists a change of variables/basis (𝑥 = 𝑃𝑦) such that:
- 𝑄(𝑥) = 𝑥^𝑇𝐴𝑥 = 𝑦^𝑇𝐷𝑦 = 𝜆₁𝑦₁² + ... + 𝜆_𝑛𝑦_𝑛²
where:
- 𝜆_𝑖- are the eigenvalues of 𝐴
𝑦_𝑖²is always positive
thus, the eigenvalues (𝜆_𝑖) determine the sign of 𝑄(𝑥)

Resources

https://en.wikipedia.org/wiki/Quadratic_form

Quadratic Forms

[data-colorid=oeevepl50e]{color:#808080} html[data-color-mode=dark] [data-colorid=oeevepl50e]{color:#7f7f7f}[data-colorid=fwcz0ipinq]{color:#808080} html[data-color-mode=dark] [data-colorid=fwcz0ipinq]{color:#7f7f7f}Quadratic Forms

Quadratic Form - Examples

from matrix form to function form

from function form to matrix form

Quadratic Form - Changing Basis such that its Quadratic Form has no Cross-Products

Proof that any symmetric matrix 𝐴 can be rewritten to 𝐷 such that its quadratic form has no cross-products (see also Principal Axis Theorem below)

Example

𝑄(𝑥) for 𝑥 = (2,-2)

Principal Axis/Axes Theorem - & Geometric View

Principal Axis Theorem

Principal Axis Theorem - Geometric View

Quadratic Form - Matrix Form Duality & Classifying Quadratic Functions

Quadratic Matrix Duality & Classification

Resources

Quadratic Forms