Principles of Econometrics

Index

• Introduction to Statistics and Econometrics
• Simple Regression Analysis
• Multiple Regression Analysis: Basics
• Multiple Regression Analysis: Inference
• Multiple Regression Analysis: Further Issues
• Heteroskedasticity and Other Problems
• Time Series Data
• Panel Data
• Instrumental Variables

1 Introduction to Statistics and Econometrics

Econometrics is the use of statistical methods to analyze economi data, starting typically from non-experimental data.

Common goals of econometric analysis are:

• estimating relationships between variables
• testing economic theories and hypoteses
• evaluating and implementing government and business policy

• Cross sectional data
  • observations that represent individuals, firms, cantons, countries (normally, but non necessarily, at one point in time).
  • observations are drawn randomly from a population (if not, we have a sample-selection problem)
• Time Series Data
  • observations represents periods in time
  • observations are consecutive and hence not random
• Pool Cross-Sections
  • at least two cross sections are combined in one data sectional
  • cross sections are drawn independently of each other
  • can often be treaded similar to a normal cross section
• Panel (or longitudinal data)
  • cross-sectional units followed over time
  • panel data have a cross-sectional and time series dimension
  • useful to account for time-invariant unbobservables and for model lagged responses

The definition of causal effect of $x$ on $Y$ is: "how does variable $y$ if variable $x$ is changed, but all other factors are held costants?
(This concept is called Ceteris Paribus).

Simply establishing a relationship (correlation) netweem variables can be misleading (important distincition between correlation and causation!)

There are multiple types of experiments that allow answering causal questions:

• Randomized controlled trials (RCT)
• Natural experiments

• Chi-Square Distribution

Let $Z_i$ be $n$ independented random variables with $Z_i\sim N(0,1)$, then

has a chi-square distribution with $n$ degrees of freedom and we write $X \sim \chi^2_n.

• t-Distribution

Let $Z\sim N(0,1)$ and $X\sim {\chi }_n^2$, then

has a t-distribution with $n$ degrees of freedom and we write $T\sim t_n$.

• F-Distribution

Let $X\sim {\chi }_k^2$ and $Y\sim {\chi }_l^2$ and assume $X,Y$ independent, then:

has a F-distribution with $(k,l)$ degrees of freedom and we write $F\sim F_{k,l}$.

The standardized average $Z$ of any population with mean $\mu$ and variance ${\sigma }^2$ is asymptotically $N(0,1)$ distributed, or, in other words:

Let $y_1{\dot{y}}_n$ be independent, identically distributed random variables with mean $\mu$, then:

2 Simple Regression Analysis

How can we use the data to describe economic reltions or behaviors?

• $y$: dependent variable (outcome variable)
• $x$: independent variable (regressor, covariate, control variable, "the cause")
• $\mu$: error term (disturbance)
  • it represents strictly unpredictable random behaviors, unspecified or unobserved factors or an approximation error if the relation is not perfectly linear.
• ${\beta }_0$: intercept parameter
• ${\beta }_1$: slope parameter

We start now with Population Modelling and consider the following assumptions:

• SLR.1: Linear parameters
  • in the population model the following relation holds: $y={\beta }_0+{\beta }_{1}x+\mu$
• SLR.2: Random sampling
  • we have a ranbdom sample of dize $n$ of random variables $(x,y)$
• SLR.3: Sample variation in the explanatory variable
  • the sample outcome on $x$ are not all the same value
• SLR.4: Zero Conditional Mean

or, in other words, for every slices of population determined by $x$, the average of $\mu$ i eaul to the population average $\mu$ (which is zero).
This also implies:

also called the Populatio Regression Function.

We now derive the Ordinary Least Squares (OLS).

Intuition: we estimate the population parameters from a data sample of two random variables.

• Let $\{(x_i,y_i)\}:i=1\dot{n}$ denonte a random sample of size $n$
• scatter plot in a $(x,y)$ system
• the regression equation $y_i={\beta }_0+{\beta }_1{x}_i+{\mu }_i$ allocates to each $x_i$ a value $y_i$ including a disturbance term ${\mu }_i$.

But _how can we derive the estimates for ${\beta }_0,{\beta }_1$

We firstly rely on the SLR.4 assumption to derive two equations:

These are called populatio moment restrictions

Using then the sample moments from the population, we can get an estimates of the population moments.

Recall:

• ${\sum }_i(y_i-{\beta }_0-{\beta }_1{x}_i)/n=0(1)$
• ${\sum }_i{x}_i((y_i-{\beta }_0)-{\beta }_1{x}_i)/n=0(2)$

we retwrite $(1$) as: $b_0=\bar{y}-b_1\bar{x}$, then we then plug into $(2)$ and solve for $b_1$:

Finally, we can easily put back $b_1$ into the first equation and derive $b_0$.

Intuition: the slope estimate is the sample covariance $x$ and $y$, divided by the sample variance $x$.

• if $x,y$ are positively correlated, the slope will be positive.
• if $x,y$ are negatively correlated, the slope will be negative.

Moreover, intuitively, the OLS is fitting a line through the sample points such that the sum of squared residuals is as small as possible.

Algebraic Properties of OLS:

• the sum of of the OLS residuals is zero.
• the sample average of the OLS residuals is zero.
• the sample covariance between the regressors and the OLS residuals is zero.
• the OLS regression line alwyas goes through the mean of the sample.

Moreover, we notice that each observation $y_i$ is made up of an explained ${\hat{y}}_i$ and unexplained part ${\hat{u}}_i$ ($y_i={\hat{y}}_i+{\hat{u}}_i$).

Using this terminology, we can define:

• $\text{Sum of Squares Total (SST)}={\sum }_i(y_i-\bar{y}{)}^2$
• $\text{Sum of Squares Explained (SST)}={\sum }_i({\hat{y}}_i-\bar{y}{)}^2$
• $\text{Sum of Squares Residuals (SST)}={\sum }_i(\cap {\mu }_i{)}^2$

To understand how well does the sample fit the regression line, we define:

Be aware that, the term linear in a OLS model does not mean a linear relationship between the variables, but a model in which the parameters enter the model in a linear way.
The following are all linear models (in the parameters):

• $y={\beta }_0+{\beta }_1\ln (x)+\mu$
• $\ln (y)={\beta }_0+{\beta }_{1}x+\mu$
• $y={\beta }_0+{\beta }_1(\frac{1}{x})+\mu$

1. OLS is unbiased (the proof depend on the 4 assumptions, if any fails, the OLS is not necessarily unbiased)
2. The sampling distibution of our estimates is centered around the true parameter

But, following the second implication, how likely is it that the true slope is slightly larger, smaller or zero?

This question can be translated into another assumption:

• SLR.5 Homoskedasticity: assume $\text{Var}[\mu |x]={\sigma }^2$

Using then the Homoskedasticity assumption, we can derive the variance of $b_1$.

Hence, ${\sigma }^2$ is also the unconditional variance, called the error variance.

We can then use this formula to find the variance of $b_1$:

• the larger the error variance $\sigma^2£, the larger the variance of the slope estimate.
• the large the variability in the $x$ ($SS{T}_x$), the smaller the variance of the slope estimate.

Finally, starting from the residuals ${\hat{u}}_i$, we can form an unbiased estimate of the error variance (often called the mean square error (MSE)), denoted by $s^2$:

Intuition: ${\sigma }^2$ is the truth; $s^2$ is our best guess based on the sample we have.

Moreover, we divide by $(n-2)$ because, estimating $b_0,b_1$, we lost 2 degrees of freedom.

3 Multiple Regression Analysis: Basics

The key problem with simple linear regression is that the assumption $\mathbb{E}[u|x]=\mathbb{E}[u]=0$is often problematic.

Consider, for example, that the true population model is:

The previous assumption states that the error term $v={\beta }_2\text{IQ}+\mu$ has $0$ (zero) expected value (no trend) or, in other words, the \text does not correlate with the $\text{wage}$.

This causes ${\beta }_1$ to be biased compared to the "true parameter", so it doesn't measure the causal effect of education to wage.

Thus, instead of assuming that multiple variables are uncorrelated with the output variable, the multiple regression model allows to include them directly in the model.

• ${\beta }_0$ is still the intercept
• ${\beta }_0\dots {\beta }_k$ are the slope parameters
• $\mu$ is the error term.

We still need a zero conditional mean: $\mathbb{E}[u|x_1\dots x_k]=0$, that means, in other words, that all the factors that influence the outcome are included in the model.

In order to estimate the parameters, we still use the Ordinary Least Squares method and we minimize the residuals:

leading to $k+1$ conditions to derive $k+1$ parameters.

The estimate model $\hat{y}=b_0+b_1{x}_1+b_2{x}_2\dots b_k{x}_k+\mu$ alloqs a ceteris paribus interpretation, that means that a change in $x_1$, $\Delta x_1$, leads to a change in $y$ given by $b_1\Delta x_1$, keeping all the other $x_i$ fixed.

Given a multiple (dual for example) regression model y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \mu$,thecoefficient$\beta_1$ can be found in 2 steps:

1. Regress $x_1$ on $x_2$:

Save then the residuals ${\hat{r}}_1=x_1-{\hat{x}}_1$. This represents the part of $x_1$ not correlated with $x_2$.

2. Regress $y$ on the residuals:

The resulting coefficient $\gamma$ will be identical $b_1$ of the original model.

The previous definition of $R^2$ is still valid, but we can add the following remarks:

• $R^2$ is the squared correlation coefficient between $y$ and the predicted $\hat{y}$.
• $R^2$ never decreases when adding independent variables to a regression (usually will increase).
• Because it increases changing the number of parameters, it NOT a good measure to compare different models.

As in the simple regression model, we can formalize the assumptions for the multiple one:

• MLR.1: Linear in parameters
  in the population model the following relation holds:

  $$y={\beta }_0+{\beta }_1{x}_1+\cdots +{\beta }_k{x}_k+\mu$$

• MLR.2: Random sampling
  we have a random sample of size $n$:

  $$\{(y_i,x_{i1},\dots ,x_{ik}){\}}_{i=1}^n$$

• MLR.3: No perfect collinearity (sample variation in explanatory variables)
  no explanatory variable is an exact linear combination of the others, and each regressor has variation in the sample.

• MLR.4: Zero Conditional Mean

  $$\mathbb{E}[\mu \mid x_1,\dots ,x_k]=0$$

  or, for every slice of population determined by $(x_1,\dots ,x_k)$, the average of $\mu$ is zero.

  This also implies:

  $$\mathbb{E}[y\mid x_1,\dots ,x_k]={\beta }_0+{\beta }_1{x}_1+\cdots +{\beta }_k{x}_k$$

  also called the Population Regression Function.

Then, we can derive the following:

Under the previous assumptions, the OSL estimator is unbiased: $\mathbb{E}[b_i]={\beta }_i$.

• If we include irrelevant variables in our model, the OLS estimator remains still unbiased.
• If we exclude relevant variables, OLS will usually be biased.

Let's suppose the true model is:

but our model is:

and we actually estimate:

The slope parameter will then be:

Recall that the numerator is:

Since $\mathbb{E}[{\mu }_i]=0$, it implies that:

Note that the bart after ${\beta }_2$ is the slope from the regression of $x_2$ on $x_1$:

so we have:

Then we can consider two corner cases:

• ${\beta }_2=0$: $x_2$ doesn't affect $y$:
• $d_1=0$, so $x_1,x_2$ uncorrelated in the sample.

Once we know that the estimate is centered around the true parameter, we want to understand how it is distributed.

If we add a fifth assumption (MLR.5 Heteroskedasticity), we know that:

we also know/derive that:

Given the assumptions from MLR.1 to MLR.5,

where $SS{T}_i={\sum }_i(x_{i,j}-{\bar{x}}_j{)}^2$ and $R_j^2$ is the $R^2$ from the regression $x_j$ from all other $x_i$.

• a larger ${\sigma }^2$ leads to larger variance in the estimarors.
• a smaller $SS{T}_i$ implies a smaller variance in the estimators.
• a larger $R_j^2$ (linear dependence between the variables) implies larger variance in the estimators.
  • this is called multicollinearity problem.

Let's analyze now what happens if we mis-specify the model. Consider again the true and the mis-specified models:

In this case the estimated variance equals $\text{Var}[c_1]={\sigma }^2/SS{T}_1$.

On the other hand, the variance using the true model equals: $\text{Var}[b_1]=\frac{{\sigma }^2}{SS{T}_1(1-R_1^2)}$

So, unless $x_1,x_2$ are uncorrelated:

Intuition:

• the variance of the estimaor is smaller in the mis-specified model_.
• the mis-specified model is biased
• As the sample size grows, the variance of each estimator shrinks to zero, making the variance difference less important

• Estimate of the errorr variance

where $df$ represents the degrees of freedom.

• Standard deviation of $b_j$:

• Standard error of $b_j$:

Theorem: Unbiased Estimation of ${\sigma }^2$::

Under the assumptions MLR.1 to MLR.5, the $s^2$ estimator is unbiased, $\mathbb{E}[s^2]={\sigma }^2$.

Under the assumptions MLR.1 to MLR.5, the OLS estimators $b_0\dots b_k$ are the best linear unbiased estimators (BLUEs) of ${\beta }_0\dots {\beta }_k$:

• Best: estimators have the lowest possible variance
• Linear: estimators are a linear function of $y$
• Unbiased: expected value equala the population paramters.

4 Multiple Regression Analysis: Inference

So far, we've seen that, given the MLR 1-5, the OLS is BLUE (mosy precise, most accurate).

To do hypotesis testig, we add another assumption.

MLR. 6: Normality (Classica Linear Model [CLS] Assumption): the disturbance $\mu$ is independent of $x_1,...x_k$ and it is normally distributed with zero mean and variance ${\sigma }^2$: $\mu \sim N(0,{\sigma }^2)$.

Under this assumption, conditional on the sample values of the independent variables we obtain:

(the coefficient estimate is normally distributed around the true beta)

This implies that:

(the standardized average deviation from the true value is standardized normal)

Furthermore, if we use the estimate $s^2$ of the variance of the disturbance (${\sigma }^2$), under the CLM assumptions we obtain:

where ${\text{se}}_{b_j}=s/\sqrt{(}SS{T}_j(1-R_j^2))$ and $n-k-1$ is the degree of freedom.

This result will be useuf to determine how likely an estimate $b_j$ is similar to ${\beta }_j$.

Both the $t$-Student and the Normal distribution are symmetric bell-shaped but the $t$-Student:

• has fatter tails than the normal
• converges to the normal for an infinite sample
• is conditional to the degree of freedom
• can be approximated with a normal distribution where the degree of freedom $>30$ or similar.

Before starting with the test, let's take a look to the different errors:

• Type I Error: we reject the null hypotesis when it is true (false positive).

• Type II Error: we don't reject the null hypotesis when it is false (false negative).

1. Set up the hypotesis. $H_0,H_1$ can be one-sided ($H_0:b_j\le 0,H_1:b_j>0$) or two sided ($H_0:b_j=0,H_1:b_j\ne 0$).
2. Determine the $t$-statistic using the estimates for $b_j, se(b_j).

For example, as seen before: $t=\frac{b_j-0}{se(b_j)}$

3. Select a significant level, or, in other terms, the chane to make Type I error and determine the critical value, depending if it is one or two sided hypotesis.

4. Decide: reject $H_0$ if ansolute value of the $t$-statistic is larger than the critical value.

For example, let's consider the following model:

1. We set up the hypotesis: $H_0:b_1\le 0,H_1:b_1>0$ ($H_0$: education doesn't increase employment)
2. We calculate $t=b_1/se(b_1)$
3. We consider $c$ using $\alpha$ (confidence level)$anddegreeoffreedom$ = n - k - 1$
4. We reject $H_0$ if $t>c$ (if $H_0:b_1>0$ use $t<-c$)

• If we reject the null hypotesis, we typically say: _$x_j$ is statistically significant/has statistically significant effect on $y$ at the $\alpha$ level.

Graphically:

More generally, we can test wheter an estimate fits a specific value: $H_0:b_j=a$.

In this case, we use the appropriate $t$-statistic: $t=(b_j-a)/se(b_j)$, where $a=0$ for the standard test.

Another way to use. stastistical testing is to construct confidence intervals using the same critical value for a two-sided test.

A $(1-\alpha )\%$ confidence interval is defined as $b_j\pm c\cdot se(b_j)$ where $c$ is the (1 - \alpha / 2)$percentileina$t_$ distribution.

An alternative approach is to calculate what is the smallest significance level at which the null hypotesis would be rejected given the data.

So we compute the $t$-statistic and we look up at which percentile it is in the appropriate $t$-distribution. This is called the p-value.

The p-value represnets the probability that we would observe the $t$-statistic we did, if the null were true.

Suppose we want to test if $b_1$ is equal to another parameter, that is: $H_0:b_1=b_2$. Then the statistic test is $t=(b_1-b_2)/se(b_1-b_2)$.

But, if we expand the formula:

So we need $s_, which we don't usually have.

To avoid this, we can use the following "trick".

We set $H_0:q_1=b_1-b_2=0$. To do that, we also have to substitute $q_1=b_1-b_2$ in our model.

So, for example, we can consider:

So far we tested a single linear restriction ($b_1=0,b_1=b_2$). Now we want to jointly test multiply hypotesis about the parameters.

A typical example is testing "exclusion restrictions", that means knowing if a group of parameters are equal to zero.

Then the null hypotesis might be something like: $H_0:b_{k-q+1}=0,...b_k=0$, but we cannot check each statistic separately because we want to know fi the $q$ parameters are jointly significant each other.
We instead need to estimate the restricted model without all the $x_{k-q+1}..x_k$ as the un-restricted model with all the $x$ included.

Intuition: we want to know if the change in SSR is bigh enough.

This is called $F$-statistic, defined as:

where $r$ is the restricted, $ur$ the un-restricted. Note that the $F$ statistic is always positive.

_Intuition: the $F$ statistic is measuring the relative increase in the SSR when moving from the unrestricted to the restricted model. -$q=d{f}_r-d{r}_{ur},quadd{f}_{ur}=n-k-1$.

To decide if the increases in SSR is "big enough", to reject the exclusions, we compare if with the $F$ distribution, indeed we know that: $F \sim F_ where:

• $q$ is the numerator degrees of freedom
• $n-k-q$ is the denominator degrees od freedom.

Imder the Gauss-Markov assumptions the OLS is BLUE, however they are not always fullfilled with real data. Large samples (big data!) come to our rescue! It can be shown that some nice properties keep intact if $n\to \infty$. (Larger samples allow to relax some assumptions).

• Consistency: if $n\to \infty$ estimators are consistent, that means that the distribution of the estimator collapses to the parameter value.
  This implies that when $n\to \infty$ we can use MLR.4 (zero mean and zero correlation).

Just as we derived the omitted variable bias earlier, we can think about the inconsistency (asymptotic bias).

Consider the true model $y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+v$ and the estimated model: $y={\beta }_0+{\beta }_1{x}_1+\mu$, so that $\mu ={\beta }_2{x}_2+v$.
In this case, $b_1={\beta }_1+{\beta }_2\delta$, where $\delta =\frac{\mathrm{Cov}(x_1,x_2)}{\mathrm{Var}(x_1)}$, which tells us how much our estimate ($b_1$) deviates from the true parameter (${\beta }_1$).

Intuition: inconsistecy is a large sample problem: it does not go away as we add data.

So far, we relied on assumption about normal distribution of errors, but this assumption can often break down! Again, large samples come to the rescue: as $n\to \infty$, the central limit theorem shows that OLS estimates are asymptotically normal.

Thus, we no longer need to assume normality with a large sample, we get it anyway.

If $\mu$ is not normally distributed, we sometimes will refr to the standard error as the asymptotic stanfard error. In general, we can expect stanfard errors to shrink at a rate proportional to the inverse of $\sqrt{n}$.

Thereare other estimators besided OLS that are consistent. However, under the Gauss-Markov assumptions, the OLS estimators will have the smallest asymptotic variances, therefore we say that the OLS is asymptotically efficient.

5 Multiple Regression Analysis: Further Issues

To test hypotheses about estimates, we previously relied on assumption about normal distribution of errors (MLR.6) in order to redive $t$ and $F$ distributions.

This implied that the distribution of $y$ given $x$ was normal as well.

However, this assumption about normality can often break down! (example: amy clearly skewed variable, like wages, arrests, savings etc... cannot be normal, since normal distributions are symmetric).

Also in this case, large samples are the solution: if $n\to \infty$ the Cental Limit Theorem shows that OLS estimates are asymptotically normal-.

In other terms, for any population with mean $\mu$ and standard deviation $\sigma$, the sampling distribution of the sample mean is approx normal with mean $\mu$ and standard deviation $\sigma /\sqrt{(}n)$.

Secondly, the $t$-distributoin approaches a normal distribution for a large $df$ (degree of freedom), so we no longer need to assume normality with a large sample.

As we said, if the error is NOT normally distributed, we sometimes will refer to it as the asymptotic standard error. (We can expect standard errors to shrink at rate proportional to the inverse of $\sqrt{(}n)$).

Changing the scale of the $y$ variable will lead to a change in the scale of the coefficients and the standard error, without a meaningfull change in significante/interpretation. The same applies for a change in $x$.

Occasionally, we will see references to standardized coefficient, calculate usig the standardized version of $x,y$, so coefficients reflect a standard deviation change of $y,x$.

OLS can also be used for relationship that are not strictly linear in $x,y$ by using non-linear functions of $x,y$ (as long the model is linear in the parameters).

• they are invariant to the sale of the variables since it's all about percentage changes
• they give a direct estimate of the elasticity
• for models with $y>0$, the conditional distribution is often Heteroskedastic or skewed, whil $\ln (y)$ is much less.
• the distribution of $\ln (y)$ is narrower, limiting the effect of outliers.

Note that, when using the log form, variables have to be positive!

For a model like $y={\beta }_1{x}_1+{\beta }_2{x}_2^2+\mu$ we know that:

• if ${\beta }_1$ is positive and ${\beta }_2$ is negative, $y$ is increasing in $x$ at first and then decreasing.
• if ${\beta }_1$ is negative and ${\beta }_2$ is positive the opposite happens.
• The turning point is calculated setting the derivative to zero and lies at: $x^{\ast }=-{\beta }_1/{\beta }_2$

Recall that $R^2$ always icrease as more variables are added to the model since $SSR$ never decreases with more variables:

The adjusted $R^2$ takes then into account the number of variables in the model:

The adjusted $R^2$ penalies models with more variables, especially for low $n$ and high $k$ and increases only when additional variables are added whose $t$-statistic is larger than $\approx$ 1. $⟹$ adding variables with poor explanatory opower decreases adjusted $R^2$.

In other terms, we usually use adjusted $R^2$ to compare models with the same $y$, but never with models with different $y$.

Suppose we want to use our estimates to obtain a specific prediction, in this case we want to estimate:

If we want to have an idea of the precision of the estimate (easily done by substituting values) we can:

• rewrite ${\beta }_0={\theta }_0-{\beta }_1{c}_1-...{\beta }_k{c}_k$
• substitute in to obtain: $y = \theta_0 + \beta_1 (x_1 - c_1) + ... + \beta_k (x_k - c_k) + \mu
• regress $y_i$ on $(x_{ij}-c_{ij})$ and the intercept will gie the predicted value and its standard error $⟹$ the minum standard error is obtained when the $c$ equal the means of the $x$.

A dummy (binary) variable is variable that takes on the value of $1$ o $0$.

Consider a simple model with one continuous variable $x$ and a dummy $D$:

• $D=1$ if female, $0$ otherwise
• $x$: education in years
• $y$: wage

This can be interprete as an incercept shift:

• if $D=0⟹y={\beta }_0+{\beta }_1{x}_1+\mu$ (male, BASE GROUP)
• if $D=1⟹y=({\beta }_0+{\delta }_0)+{\beta }_1{x}_1+\mu$ (female)

Any categorial variable can be turned into a set of dummy variables. Because the base group is represented by the intercept, if there are $n$ categories there should be $n-1$ dummy variables.

Note that we can model interaction between dummies to divide in subgroups and between dummies with continuous variables $x$ to model a change in slope.

Testing wheter a regression function is different for one group versun one another can be thought as of testing for the joint significance of the dummy and its interaction with all other $x$ variables.

So we can estimate the model with and without the interactions and form a $F$-statistic (very tedious in practise!).

We can compute the $F$-statistic without running the unrestricted model with all interactions with $k$ continuous variables, but intead we can:

• run the restricted model for group 1 _(using observations $n_1$) and get $SS{R}_1$
• run the restricted model for group 2 _(using observations $n_2$) and get $SS{R}_2$
• run the restricted model for all _(using $N=n_1+n_2$) and get $SSR$
• compute the $F$-statistic as:

$\mathbb{P}(y=1|x)=\mathbb{E}[y|x]$ when $y$ is binary model, so we can write our model as:

• ${\beta }_j$ represents the change in the probability when $x_j$ changes by 1 around the mean.
• the predicted $y$ is the probability

However, potential problems arise since the prediction can be outside $[0,1]$.
Also, this model will violate the assumption of homoskedasticity, which will affect inferece.

Despite everything, OLS is usually a good starting point when $y$ is binary.

6 Heteroskedasticity and Other Problems

• MLR.1: Linear in parameters
  • In the population model, the following relationship holds:

• MLR.2: Random sampling
  • We have a random sample of size $n$:

• MLR.3: No perfect collinearity
  • None of the independent variables is constant, and there are no exact linear relationships among regressors.
• MLR.4: Zero conditional mean
  • The error has zero conditional mean:

• This implies $\mathbb{E}[u]=0$.

• MLR.5: Homoskedasticity
  • Assume constant conditional variance of the error:

• MLR.6: Normality
  • The disturbance is independent of $x_1,\dots ,x_k$ and normally distributed with zero mean and variance ${\sigma }^2$:

Recall that the assumption of homoskedasticity implied that, conditional on the explanatory variables, the variance of the unobserved error $\mu$ was costant.
If this is not true, then the variance is different for different values of $x$ and errors are said to be heteroskedastic.

OLS is still unbiased and consistent, even if we do not assume homoskedasticity.

However, the standard errors of the estimates are biased if we have heteroskedasticity.

• If the standard errors are biased, we cannot do inferece based on the usual $t$, $F$, $LM$-statistic. (The $LM$-statisitc is $LM:n{R}_{\mu }^2$, and $R_{\mu }^2$ is obtained regressing $\mu$ on all variables. The LM statistic has a $X_k^2$-distributio.)

For the simple bivariate case, heteroskedasticity implies that:

OLS slope decomposition:

Conditional variance of $b_1$ under heteroskedasticity:

Note that this differs from the homoskedastic case

A valid (consistent) estimator of the variance of $b_1$ when ${\sigma }_i^2\ne {\sigma }^2$ is:

where ${\hat{u}}_i$ are the OLS residuals.

Note that this is different from the homoskedastic estimator:

Consider the model:

With heteroskedasticity, a valid (consistent) estimator of $\mathrm{Var}(b_j)$ is:

where:

• $r_{ij}$ is the $i$-th residual from regressing $x_j$ on all other independent variables.
• $SS{R}_j={\sum }_{i=1}^n{r}_{ij}^2$ is the sum of squared residuals from this auxiliary regression.
• ${\hat{u}}_i$ are the OLS residuals from the original model.

So, the corresponding robust standard error is:

Sometimes a finite-sample correction is used:

As $n\to \infty$, this correction becomes negligible.

Note that robust standard errors are justified asymptotically. In small samples, $t$-statistics based on robust SE may not be close to a $t$ distribution.

We want to test:

If we assume the relationship betwee ${\mu }^2$ and $x_j$ to be linear, we can test it as a linea restriction:

In this test, we do not observe the error, but we can estimate it with the residuals from the OLS regrssion.

After regressing the residuals squared on all $x$, we can use the $R^2$ to form a $F$ or $LM$-test.

• The $F$-statistic is distributed as $F_{k,n-k-1}$ and it is equal to:

• THe $LM$-statistic follows a $X_k^2$-distribution and is:

_Note that the Bresuch-Pagan test detect any linear forms of heteroskedasticity.

The White Test allows for non-linearities by using squares and cross-products of all $x$.

Consider thenm the fitted values from OLS $\hat{y}$ are a function of all $x$. Therefore ${\hat{y}}^2$ will be a function of the squares and cross-produts, so we can:

• regress the residuals squared on $\hat{y},{\hat{y}}^2$
• use the $R^2$ to form a $F$ or $LM$-statistic

While it is always possible to estimate robust standard errors, if we know something about the specific form of heteroskedasticity we can obtain a more efficient estimates.

Intuition: transforming the model into one that has homoskedastic errors $⟹$ if we do it, we call the estimators weighted least squares.

Suppose the original model is: $y_i={\beta }_0+{\beta }_1{x}_{i1}+\cdots +{\beta }_k{x}_{ik}+{\mu }_i$

and heteroskedasticity has the form:

We can define the variable:

Because $h_i$ is a function of $x_i$, conditional on $x_i$ it is a constant and has costant variance.

Hence the transformed error is homoskedastic.

We now trasform the whole equation model by dividing the original model by by $\sqrt{h_i}$:

We can then define the transformed variables as:

so we obtain:

So OLS on this transformed model is BLUE (under the usual assumptions).

But why this is called “weighted” least squares?

OLS on transformed data minimizes:

Therefore WLS minimizes weighted residual squares with weights $w_i=\frac{1}{h_i}$

Intuition:

• If $h_i$ is large, observation $i$ has high error variance $\Rightarrow$ lower weight.
• If $h_i$ is small, observation $i$ has low error variance $\Rightarrow$ higher weight.

In reality, we often do NOT know the exact form of heteroskedasticity ($h(x)$), so we need to estimate it.

We generally assume a flexible variance model:

So, under this assumption, we have:

Note that exponential form guarantees $h(x_i)>0$, so variance cannot be negative.

From this assumption: