Orthogonality and Geometric representation of OLS solution

You will have to go through part  Deriving OLS with Matrices  as prerequisite to understand this post:

From first order condition we have:

\[\begin{align} & \frac{\partial S}{\partial b}=2{{x}^{T}}xb-{{x}^{T}}y-{{({{y}^{T}}x)}^{T}}=0 \\ & or\text{ }2{{x}^{T}}(y-xb)=0 \\ & or\text{ }{{x}^{T}}(y-xb)=0 \\ \end{align}\]

Here: \((y-xb)=e\text{ }\left( Residual \right)\)

So, \(\text{ }{{x}^{T}}e=0\)

Note that \({{x}^{T}}e=0\) is the condition of orthogonality 

What is Orthogonality ?

When conducting econometric or statistical analysis, independent variables that affect particular dependent variable are said to be orthogonal if they are uncorrelated. Orthogonality is a condition when two vectors intersect each other in a right angle. 

 In above illustration 'x' independent variable is orthogonal on 'e' residual

Geometric representation of 'OLS' solution 

We have residual 'e' represented as difference between actual and predicted value. So y (actual) -xb (predicted y) which is equal to 'e' can be written as:

\(\begin{align} & e=y-xb=y-x{{({{x}^{'}}x)}^{-1}}{{x}^{'}}y \\ & =(I-x{{({{x}^{'}}x)}^{-1}}{{x}^{'}})y \\ \end{align}\)

Remember we derived value of 'b' from OLS solution in previous post

Let M = \(\begin{align} &(I-x{{({{x}^{'}}x)}^{-1}}{{x}^{'}}) \\ \end{align}\) 

So the expression becomes 'My',  known as operator

Further we know that $\hat{y}=xb={{({{x}^{'}}x)}^{-1}}{{x}^{'}}y$
Let H= ${{({{x}^{'}}x)}^{-1}}{{x}^{'}}$

So the expression becomes 'Hy', known as 'Hat' matrix or Projection matrix

Note that e ='My' is perpendicular on y(hat)= 'Hy' line which has the projection matrix made of x