Finding a Trend Line with Least Squares

Suppose we have a bunch of data points in \(\mathbf{y}\). We create a design matrix, \(X\) which consists of the pattern that we want to match. The first column will be all \(1\)'s because that's the constant term. The second column will be the numbers \(1\) through \(N\) for the linear term. If we wanted a quadratic term, we could include an \(x^2\) sequence as a third column or for a cubic term a \(x^3\) sequence as a fourth column.

\[X = \begin{bmatrix}1 & 1 \\ 1 & 2 \\ 1 & 3 \\ \vdots & \vdots \\ 1 & N \end{bmatrix}\]

To determine the trendline, we need to solve the equation,

\[ X \mathbf{b} = \mathbf{y} \,.\]

Assuming \(X\) is non-invertible,

\[\begin{aligned} X^\top X \mathbf{b} &= X^\top \mathbf{y} \\ (X^\top X)^{-1} (X^\top X) \mathbf{b} &= (X^\top X)^{-1} X^\top \mathbf{y} \\ \mathbb{1}\cdot\mathbf{b} &= (X^\top X)^{-1} X^\top \mathbf{y} \\ \mathbf{b} &= (X^\top X)^{-1} X^\top \mathbf{y} \\ \end{aligned}\]

using Plots

y = [10.0, 7.0, 4.0, 3.0, 3.0, 4.0, 2.0, 1.0, 2.0, 0.0]
X = [ ones(10) 1:10 ]

b = inv(X'*X)*X'*y

scatter(y)
plot!(X * b, lw=3)