Care for a cup of “t” ?

OK. I agree the title has a bad pun. We are not going to discuss about Darjeeling tea or English tea or any other tea for that matter. (I can feel the dissatisfaction among the tea lovers.)

But hold your horses. The ‘t’ that we are going to discuss is the t-statistic. The stalwart for so many of our hypothesis tests. There are indeed many interesting properties of the t-stat but one particular property stands out in regression analysis.

We have always been told that during model building if we drop an independent variable that has an absolute t-value less than 1, the adjusted R-squared will increase. The model is so called better. But why 1, why not 2 or 0.5. Further, why should the adjusted R-squared behave in this manner. Well that is the topic of discussion today. So lets get into it.

But…(there is always a but somewhere) before that, a little background that we are going to need. And we are going to start with R-squared. As we all know (probably) R-squared is a measure of fit of our sample data. We also might have been told that adding more and more independent variables to our regression increases R-squared. That’s where the problem lies with R-squared.

A standard result in regression analysis is the effect on R-squared of adding one more independent variable to the regression. Lets see that.

Say, the original regression is,

y = {\beta}_{0}+{\beta}_{1}{X}_{1} +...+{\beta}_{k-1}{X}_{k-1} + e \qquad Model\qquad 1.

And we add a variable Z to it,

y = {\beta}_{0}+{\beta}_{1}{X}_{1} +...+{\beta}_{k-1}{X}_{k-1} +cZ+ e \qquad Model\qquad 2.

Now if we calculate the R-squared for each and call them,

{{R}^{2}}_{X} \qquad for    model\qquad 1.

{{R}^{2}}_{XZ} \qquad for    model\qquad 2.

Then they are going to be related as,

{{R}^{2}}_{XZ} = {{R}^{2}}_{X} + {{r}^{2}}_{*}\left( 1 - {{R}^{2}}_{X}\right) \qquad Equation\qquad 1.

Here, {{r}^{2}}_{*} is the partial correlation coefficient of y and Z. By partial correlation coefficient we mean that the effect of all the other regressors i.e. X in this case have been removed. Only after removing their effects is the sample correlation between y and Z calculated.

Immediately we can make some important observations.

  1. {{R}^{2}}_{XZ} is always greater than {{R}^{2}}_{X}. This is so because all the terms in Equation 1 are positive and {{R}^{2}}_{X}\le1. This clearly shows that adding more independent variables increase the R-square.
  2. The only case where R-squared doesn’t increase is if {{r}^{2}}_{*}=0 which means that the new independent variable Z has no explanatory power in a linear setting. This also makes intuitive sense.

A word of warning. As explicitly mentioned above correlation measures are a measure of linear explanatory power. It can be the case that non-linear relations exist between y & Z, which may not be captured by the correlation coefficient.

So now we have a relation between, the R-squared and the correlation coefficient {{r}^{2}}_{*} . Next lets see how R- squared and adjusted R-squared are related.

The definition of adjusted R-squared is,

{\hat{R}}^{2} = 1 - {\frac{RSS}{n - K}}/{\frac{TSS}{n - 1}}


RSS = Residual Sum of Squares                                                   TSS = Total Sum of Squares.

K = Total nos of regressors including the constant term. So it will be k for model 1 and (k+1) for model 2.

While R-squared is defined as,

{R}^{2} =  1 - {RSS}/{TSS}

So inserting R-squared into the definition of adjusted R-squared we get,

{\hat{R}}^{2} = 1 - \left(1 - {R}^{2}\right)\frac{n - 1}{n - K} \qquad Equation\qquad 2.

Now for the missing link, the relation of t-stat with the partial correlation coefficient;

{{r}^{2}}_{*} = {{t}^{2}}/{\left({t}^{2}+df\right)} \qquad Equation\qquad 3.

where, df = degrees of freedom = n – k – 1

The proof of equation 3 is a bit involved. I will not give the proof here as it is a standard proof which one can refer to in textbooks such as William H. Greene.

Whoa!!! That’s a lot of equations and nothing yet to show for it. But everything needed is now in place. Let’s recap the 3 equations:

{{R}^{2}}_{XZ} = {{R}^{2}}_{X} + {{r}^{2}}_{*}\left( 1 - {{R}^{2}}_{X}\right) \qquad Equation\qquad 1.

{\hat{R}}^{2} = 1 - \left(1 - {R}^{2}\right)\frac{n - 1}{n - K} \qquad Equation\qquad 2.

{{r}^{2}}_{*} = {{t}^{2}}/{\left({t}^{2}+df\right)} \qquad Equation\qquad 3.

This is all we need to establish our desired result. Only thing left is to manipulate the equations. Let’s do that:

Step 1.

Define constants,

\lambda = \frac{n - 1}{n - k}

\delta = \frac{n - 1}{n - k - 1}

We are doing this so that the equations are easier to manipulate.

Step 2.

Write the equations for the adjusted R-squared for model 1 & 2,

{{\hat{R}}^{2}}_{XZ}= 1 - \left( 1 - {{R}^{2}}_{XZ} \right)\delta

{{\hat{R}}^{2}}_{X}= 1 - \left( 1 - {{R}^{2}}_{X} \right)\lambda

Step 3.

Subtract the second equation in step 2 from the first to get the change in adjusted R-squared,

\Delta{\hat{R}}^{2} = \left( 1 - {{R}^{2}}_{X} \right)\lambda - \left( 1 - {{R}^{2}}_{XZ} \right)\delta

Using Equation 1,

\Delta{\hat{R}}^{2} = \left( 1 - {{R}^{2}}_{X} \right)\lambda - \left(1 - {{R}^{2}}_{X} - {{r}^{2}}_{*}\left( 1 - {{R}^{2}}_{X}\right) \right)\delta

\Delta{\hat{R}}^{2} = \left( 1 - {{R}^{2}}_{X} \right)\lambda - \left(1 - {{R}^{2}}_{X} \right) \left( 1 - {{r}^{2}}_{*} \right)\delta

\Delta{\hat{R}}^{2} = \left( 1 - {{R}^{2}}_{X} \right) \left\{\lambda - \left(  1 - {{r}^{2}}_{*} \right)\delta \right\} \qquad Equation\qquad 4

Step 4.

Now, we are trying to see the change in adjusted R-squared. So let us assume we want to see the condition for an increase in adjusted R-squared. For that, the right side of equation 4 must be positive. The right side of equation 4 is a product of two terms. So either both must be positive or both negative. Since 1 - {{R}^{2}}_{X}\ge 0 , the second term must also be positive, i.e.

\left\{\lambda - \left(  1 - {{r}^{2}}_{*} \right)\delta \right\} > 0

\frac{\lambda}{\delta} > 1 - {{r}^{2}}_{*}

Putting in the values of lambda and delta,

\frac{n - k - 1}{n - k} > 1 - {{r}^{2}}_{*}

1 - \frac{1}{n - k} > 1 - {{r}^{2}}_{*}


\frac{1}{n - k} < {{r}^{2}}_{*}

Step 5.

Now use equation 3 to replace the partial correlation coefficient value in terms of the t-statistic to get,

\frac{1}{n - k} < \frac{{t}^{2}}{{t}^{2}+\left( n - k - 1\right)}

Cross multiplying,

{t}^{2}+\left( n - k - 1\right) < \left( n - k \right) {t}^{2}

\left( n - k - 1\right) < \left( n - k - 1 \right) {t}^{2}

1 < {t}^{2}

1 < \left| t \right|

And there we have it. So we see that if the t-statistic is greater than 1 in absolute terms the change in the adjusted R-squared is positive. Similarly we can establish the result for a decrease in adjusted R-squared.

Just in case you are lost in this mumbo jumbo of equations, let me recap what we are trying to show. We want to evaluate the change in adjusted R-squared. So we added a new independent variable Z and calculated the adjusted R-squared for that new model. We now want that change to be positive. So we manipulate the equations 1 – 4 to show that the new adjusted R-squared value will be higher than the old value if the t-statistic associated with the new variable Z is greater than 1 in absolute terms.

So now we have the proof of the result.

Till next time. Ciao!!!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s