Forecasting guide

I wrote this guide when I joined a team where few people had formal training in forecasting methods, but many people had PhDs in quantitative fields and so weren’t afraid of a good-looking equation. As a result, this guide aims to help develop a solid mathematical basis on which to build data literacy and intuition for forecasting problems.

There are plenty of guides online for how to build forecasting model pipelines. Scikit-learn, a python library, made it possible to build a whole pipeline in 50 or so lines of code. However, the difficulty is in knowing: a) which 50 lines to write and b) what these 50 lines really do.

This guide aims to bridge the gap between what the python libraries tell you and what they don’t.

Here you won’t find: a cookbook with ready-made code examples available to plug and play with your data; strong opinions on which method is the best for all problems; or deep details on the underlying mathematics. However in here you will find: a wide survey of methods along with an overview of the mathematical underpinnings of forecasting, some historical context for the methods, and some free and paid resources to help further your learning.

The goal is to help you learn what is possible and what is available.

Note – Types of datasets: this guide sticks to dealing with tabular data (or data that can be transformed into a usual tabular format). This is in opposition to so-called unstructured data, like text or images.

The dictionary definition of forecasting is to predict or estimate a future event or trend. In our context, it’s specifically to make an educated guess about future data given our current data. This can mean predicting future prices with past prices, predicting future battery performance with past performance, or predicting future sales with past sales.

Note how this is different from building a physical model. Forecasting relies on capturing the information that exists in your data (sometimes you will also see people talk about capturing the signal in your data). Although physical modeling and forecast modeling are very different, they are both called modeling because they are both ways to represent the world.

Note – Physics-informed forecasts: Although I just spent a whole paragraph telling you that physics modeling and forecast modeling are different, there is a way to use both together. In fact, a physical model can inform the structure of your forecasting model. Steven Brunton has a nice youtube channel I recommend on this topic (and also a book which has been on my reading list for over a year, but who’s got time to read?)

It turns out there are many different problems we would like to forecast, and so there are many different forecasting methods. And adjacent to the forecasting methods, there are approaches to test for the validity of the forecasting methods. A full forecasting model is usually, a method (like linear regression), a testing framework (like cross-validation), along with some data (and associated feature engineering).

Fundamentally, forecasting methods rely on the assumption that there is a stochastic element to data collection. Out of all the possible asset prices, we saw only the ones that did happen. The question becomes, how different could the prices have been? And in the future, how different can they be? And how do past prices relate to future prices? Forecasting is about exploiting the inherent stochasticity in our data to answer these questions.

Note – Frequentists VS Bayesians: The above paragraph is mostly written from a frequentist perspective, which is the dominant framework people learn. This reformulation relies on the idea of repeated sampling while a bayesian version of this would rely on the idea of measuring uncertainty (i.e. what is our belief about what the prices could have been).

First, you would perform Exploratory Data Analysis (EDA). Often, this means plotting your data and looking at simple metrics: min, max, mean, standard deviation, quantiles, correlation across variables, … This allows you to start to guess at some of the rules that your data follows. Then, using domain expertise you would start to narrow down which kind of problem we are facing. What do we want out of our data? How much data do we have? How explainable do we need our solution to be? What’s the measure of success for our forecasting model? The answers can change over time, and this is reflected in our choice of modeling approach.

Generally, before building a complex model and once the questions we want to answer are clearer, we start by building a simple benchmark. This benchmark should be as simple as: using a mean; using the previous day’s data; using a fixed prediction; a very bare bones linear regression. Without such a benchmark, we don’t even know what we are aiming to beat.

Once we have a benchmark and we have some idea for a more complex model, there are generally three avenues to augment that more complex model:

• Feature engineering:

  That is, modify your data in some ways. This is usually driven by the results of your EDA and your domain expertise. For instance, if you know that markets behave in a certain way, you can modify your market data to reflect that. Modifications can be about removing outliers, scaling the data, generating interaction variables, or passing your data through a complex function. A variety of metrics from the world of information theory can help you assess if your transformed data is better for the forecasting task. Note that usually the number of possible transformations is much higher than what any computer can handle, so you need to exercise judgment when coming up with them.

• Tune your model’s parameters:

  We will cover more examples of model parameters later, but this tuning is an essential step. Although it depends strongly on your data, it is also very much about the specific model you are using.

• Finally, try a different model:

  There are different models to try for regression or classification, and their quirks will fit some problems better than others. A priori it is usually hard to guess which model will perform best.

Note – What constitutes a model: is it the base equation? Is it just the parameters? Is it the trained output of your code? Is it how you train and test the model? Is it your data? Most people nowadays agree that it is all of that, and so you should have a versioning system to make sure you know at all times which version of your data goes with which version of your model training approach.

Note – Feature engineering: Machine Learning will often perform drastic data transformations to build models with better out-of-sample performance. This data transformation is often called feature engineering, or feature extraction, or sometimes more specifically dimensionality reduction. Feature engineering can mean taking a continuous variable (e.g. age number) and making it into a categorical variable (e.g. age category like child, adult, …) via heuristics, or it can mean using statistical methods to create the features. Econometrics however tends to be much more restrained with its data transformations, typically takings logs or first-differences only.

Scalars are like regular numbers, while vectors are collections of scalars. We say vectors are of shape \( n \times 1 \) where \( n \) is the number of scalars in the vector. Matrices are collections of vectors. We say a matrix has shape \( n \times k \) when it has \( n \) rows and \( k \) columns.

Typically, scalars are designated by a single lowercase letter, while vectors are a single lowercase letter with either an arrow above or in bold, and matrices are a single uppercase letter, sometimes in bold. When the context makes it obvious, the arrow or the bold font are omitted for legibility. Confusingly, random variables are also denoted by a single upper case letter. Welcome to statistics.

If you want to use words, tensors are the 3+ dimensional equivalent of matrices.

The transpose of a vector or of a matrix is that same vector or matrix but flipped symmetrically along its diagonal. So a vector that is \( n \times 1 \) becomes \( 1 \times n \) and a matrix that is \( n \times k \) becomes \(k \times n \). The notation for the transpose of matrix \( X\) is \( X’ \) or sometimes \( X^T \). In this guide I use \( X’ \) for legibility.

To indicate that a random variable \( X \) follows a given distribution \( \mathbb{F} \), the notation is: \( X \sim \mathbb{F} \).

Multiplying two numbers together is easy, but to multiply two matrices or vectors, their sizes need to match up: if \( X\) is \( n \times k \) and \( \beta \) is \( k \times 1 \) then I can compute \( X \times \beta \) and the result is of size \( n \times 1 \), but I can’t compute \( \beta \times X \). Note how squaring matrices works: \( X^2 \) can be either \( X’X \) or \( XX’ \). You choose the one you want based on the dimensions you want and/or need for your result. Note that this means that squaring a vector can lead to a scalar or a square matrix.

In a forecast, we typically estimate parameters. If the true parameter is written as \( \beta \) then the estimated parameter will be written with a hat: \( \hat{\beta} \). Some data transformations are written using a tilde above the variable that represents the data. Data transformations that involve differencing typically use dots instead of tildes, which is a convention that comes from the field of differential equations.

• Hat: \( \hat{\beta} \)
• Tilde: \( \tilde{x} = f(x) \) where f is some function
• Dot: \(\dot{x}_{t} = x_{t} – x_{t-1}\)

Usually, a derivativate is denoted using \( ‘ \), but to not create confusion with the transpose notation and to be more explicit on the number of times the derivative is taken, I use the notation \( X^{(n)} \) to indicate the \( n-th \) derivative.

Throughout the guide, I talk about population data and sample data. The population is the entire data that can, while the sample data is the part of that total that I collected.

For instance, if I measure the height of everyone in a building, that is a sample of all the heights that are possible for people. Note that the definition of population data can be difficult: it could be all the people currently alive, or all the people ever. How far back do we go? Up until homo sapiens first appears? Such considerations are usually important insofar as it’s important to understand if the data that was collected was collected because it was special in some way (e.g. is there some selection bias at play).

Consider a maybe less contrived example. In the stock market, we observe only one price for each time of the day for each asset. What is the population distribution for that data? Is it a distribution for each time? What if some times have no new prices? Is it a distribution for each day? Is it a distribution for each asset? It’s unlikely that there is a true answer here, rather what matters is what we’re trying to model.

The operator \( \mathbb{E} \) is called the expectation, or expected value, and it is (usually) equivalent to the mean. The variance operator is most often written \( Var \), and the covariance operator \( Cov \).

Sometimes, we want to express what we know about a random variable given some knowledge we already have, as opposed to without that knowledge. Consider the random variable \( Y \), and some information \( X \). We write formally that \( Y \) given \( X \) is \( Y | X \).

Nowadays, almost all the best forecasting methods come from the field of Machine Learning (ML). So why bother with an old field like Econometrics? Although it’s often true that Econometrics methods perform less well (specifically when measuring out-of-sample performance), they are often much more explainable. Moreover, the mathematics of ML tend to focus on numerical computation while the mathematics of Econometrics tend to focus on analytical solutions to modelling problems. As a result, the two tool sets complement each other well.

Historically, Econometrics came first. It can be considered a sub-branch of Statistics and Economics. Some classic questions for Econometrics are:

• Given some past price data, can I predict future price data?
• Given an equation for how unemployment and inflation are linked, can I estimate the parameters of this equation?
• Can I evaluate the existence/validity of the causal link between seemingly unrelated data?

Because Econometrics is older, it concerns itself a lot with:

• Causality and how to recover it from data, in the context of explaining in-sample effects.
• Estimation with small datasets (Economics departments used to have armies of grad students invert matrices by hand).
• Analytical solutions and other theoretical results (e.g. BLUE estimators).

Machine Learning is a more recent field and is usually considered a sub-branch of Statistics and Computer Science. Machine Learning concerns itself more with issues of pattern recognition. Some classic questions for Machine Learning are:

• Given different classes for my data, can I predict the class of a new piece of data?
• How well does my prediction generalize outside of my training data?
• Can I use the features of my data to come up with a classification?

Because Machine Learning is more recent and is more in line with modern computing advances, it concerns itself a lot with:

• Numerical solutions to problems (e.g. gradient descent).
• Out-of-sample predictions and predictions on very different data (i.e. overfitting).
• Applied heuristics (e.g. how to select a best model in practice, regardless of theory?).

Nowadays both fields can and should use each others’ tools. In practice, both can tend to look down on each other. ML practitioners think Econometricians don’t know how to use computers properly and Econometricians think it’s crazy that people don’t seem to care about causality or statistics anymore.

In the next section we will spend a significant amount of time on regression analysis and its mathematical underpinnings. The reason for this choice is four-fold:

• Regression is a somewhat simple model with strong mathematical underpinnings. These assumptions are very useful for understanding what we want in a “well-behaved” model, not just for linear regression.
• Regression has a lot of extensions. It can be linear or have many different shapes, it has extensions that allow for parameter shrinkage and automatic selection (see elastic net later on), and overall offers a surprisingly flexible and explainable framework. Even in cases where the ultimate model is not linear regression, I still find myself thinking about a lot of the concepts I explain here when making sure my random forest or support vector machine is working correctly.
• A lot of classification models can be represented with a regression set up (e.g. logit or probit).
• That’s how I learned forecasting, so I’m biased.

Note – What does the word regression mean?: You may have heard the term regression towards the mean, and you might know that regression in most contexts means going back or reverting. Regression towards the mean is when you sample data from some process, let’s say you measure people’s heights, and you observe a really large number, say someone measures above 2 meters. Then you would expect your next sample to not be so extreme, and instead be \closer\ to the mean value, in our example the mean height.

Regression was officially discovered and published by Adrien-Marie Legendre in 1805, although Gauss is rumored to have used the method in his work up to 10 years before and is credited with significant advancements to the method. The method originally came from the fields of astronomy and geodesy, and in particular came about to solve the challenges of navigating the Earth’s oceans during the Age of Discovery.

The problem to solve was the following: given some features \( X \), a \( n \times k \) matrix, and some outputs \( y \), a \( n \times 1 \) vector, can you find the link between \( X \) and \( y \)? Nowadays Econometrics also really cares about whether that link is causal (which is arguable not a mathematics question), but the fundamental mathematical problem is to estimate that relationship.

The problem they set out to solve is to find \( \beta \) such that:

\[y = X \beta + \varepsilon \]

This equation is often called the regression anatomy formula.

Another way of looking at it, is to think about defining the shape of the error that you are willing to live with:

\[\varepsilon = y – X \beta\]

Note that \( \varepsilon \) is \( n \times 1 \) and \( \beta \) is \( k \times 1 \).

Furthermore keep in mind that the fundamental goal of linear regression is to reduce the dimensionality of your problem. Reducing a scatter of points to a single line loses some information. The data is not allowed to move in some directions as a trade-off to make the problem more explainable. Ideally all the movement removed is represented by the error term, but that is not always true.

Note – Linear VS non-linear regression: What makes the regression linear? Above, we saw that we can interpret regression as defining the shape of error we can live with. If that function is linear in \( X \), then the regression is said to be linear. In practice, this is visible because if you draw your regression line, it will be straight. This consideration is separate from how we measure our error term (vertical, perpendicular, squares, etc), as discussed in the next section.

Looking at the image below with a linear scatter: where would you put the line of best fit?

If we draw a straight line through these points, then the distance between the points and the line is our error. Ideally we would want the total error, i.e. the sum of those distances, to be minimized. This idea of minimising a sum of error terms is key to most ML methods.

Visually, that error can be represented by line segments that start at our data points and go towards the fitted line. But which orientation should the segments have?

The segments could intersect with the line of best fit perpendicularly:

Or the segments could all be parallel to one of the axes:

Gauss and Legendre’s insight was to draw not line segments, but squares, and to find the line that minimizes those squares:

Look at how different those lines look like:

Fundamentally, there are two considerations:

1. Which direction the error lines are pointing – we saw vertical and perpendicular, but horizontal is also possible. A vertical line (i.e. parallel to the y-axis) implies that we only consider errors for the \( X \) variables, while a horizontal line (i.e. parallel to the x-axis) implies that we only consider errors for \( y \) variable. A perpendicular line implies that we consider the errors for both the \( X \) and \( y \) variables. Whether the line segment is vertical, horizontal, or perpendicular changes the regression anatomy formula.
2. How we sum our errors – the most common ways are the Mean Squared Errors (MSE) and the Mean Absolute Errors (MAE). For the MSE, we square our errors and then sum them, while for the MAE we take the absolute value of our errors and then sum them. More formally:

\[MSE = \frac{1}{n} \sum_{i=1}^{n} \left( \varepsilon_i \right)^2 = \frac{1}{n} \sum_{i=1}^{n} \left( y_i – x_i \beta \right)^2 \] \[MAE = \frac{1}{n} \sum_{i=1}^{n} | \varepsilon_i | = \frac{1}{n} \sum_{i=1}^{n} | y_i – x_i \beta | \]

Where \( y_i \) is a scalar and \( x_i \) is a vector, making the MSE and MAE scalars.

Using vertical lines and squaring them to minimise the MSE is how we obtain the most commonly known regression formula: Ordinary Least Squares (OLS). But of course, all other set-ups are allowed: horizontal errors with the MAE, perpendicular errors with the MSE, etc. That last one is called total regression, or sometimes orthogonal regression or even Deming regression. It is rare to see it in practice as a form of regression, and instead this approach is more commonly seen in ML methods such Principal Component Analysis (PCA) or Support Vector Machines (SVM).

For now, we focus our attention on OLS, and why the insight of minimising squares is so powerful:

• Squaring your errors before adding them together means that positive errors and negative errors do not cancel each other out.
• The square function is easy to differentiate, so we can look at our estimated errors, square them, and taking the derivative, we can easily derive our estimator for \( \beta \) analytically. Compare this with the MAE, which has a kink around 0 and is therefore not easily differentiable.
• Squaring is how we calculate the Euclidian distance in an arbitrary vector space. Looking at the vector space spanned by \( X \), squaring the elements is like computing the \( L2 \) norm. In particular, you can prove that minimizing the \( L2 \) norm is a type of projection onto a vector space. And since minimizing the \( L2 \) norm is how we solve for our OLS estimator, we can understand OLS as a projection of \( y \) onto the vector space spanned by \( X \). This gives us a strong geometric representation of what our approach does.

We now turn our attention to the two key results needed for classical linear regression to work nicely: the Law of Large Numbers (LLN) and the Central Limit Theorem (CLT). Both are limit theorems but the former is about point estimation while the latter is about distributional estimation.

There are many equivalent ways to derive the Ordinary Least Squares estimator for a linear equation. Here we take the square of our errors and differentiate them. Setting the derivative equal to 0 allows us to solve for the \(\beta\) that minimizes the errors. We check that this is indeed the minimum by taking the second derivative. Note that the following is written in matrix notation, so \(\varepsilon’ \varepsilon\) is really \(\varepsilon^2\). Note how \(\varepsilon’ \varepsilon\) is a scalar. We start with the first order condition:

\[\mathop{\arg\min}\limits_\beta \hspace{2mm} \varepsilon’ \varepsilon = \mathop{\arg\min}\limits_\beta (y – X \beta)'(y – X \beta) \quad (1)\] \[\Leftrightarrow \frac{d}{d\hat{\beta}} \left[(y – X \hat{\beta})’ (y – X \hat{\beta}) \right] = 0 \quad (2)\] \[\Leftrightarrow \frac{d}{d\hat{\beta}} \left[y’y – y’ X \hat{\beta} – \hat{\beta}’ X’ y + \hat{\beta}’ X’ X \hat{\beta} \right] = 0 \quad (3)\] \[\Leftrightarrow \frac{d}{d\hat{\beta}} \left[y’y – 2 \hat{\beta}’ X’ y + \hat{\beta}’ X’ X \hat{\beta} \right] = 0 \quad (4)\] \[\Leftrightarrow – 2 X’ y + 2 X’ X \hat{\beta} = 0 \quad (5)\] \[\Leftrightarrow X’ X \hat{\beta} = X’y \quad (6)\] \[\Leftrightarrow \hat{\beta} = (X’X)^{-1}X’y \quad (7)\]

From (1) to (2), take the first derivative to find the \( \arg\min \). Use \( \hat{\beta} \) to indicate the estimator as opposed to the true value. Note that this whole expression is a scalar, it is equal to the number 0, not a matrix of 0s. This means we can take transposes of each element.
From (2) to (3), expand the terms.
From (3) to (4), rearrange the scalar terms to group them by exponent of \( \beta \).
From (4) to (5), apply the derivative.
From (5) to (6), move over the terms by adding on both sides and dividing by 2 on both sides.
Finally from (6) to (7), multiply by the inverse on both sides.

Now we look at the second order condition: \[\frac{d^2}{d\hat{\beta}^2} \left[(y – X \hat{\beta})’ (y – X \hat{\beta}) \right] = X’X > 0\]

So the estimator does in fact minimize the squared error.

The complete derivation is only included to satisfy the more curious readers. For those who want some extra credit, look into two other ways of deriving this same estimator: the method of moments and the maximum likelihood estimation method. Regardless, the most important thing to remember from this derivation is the final formula: \[\hat{\beta} = (X’X)^{-1}X’y\] Most crucially, note the need for matrix inversion. A lot of estimation methods rely under the hood on matrix inversions, which is one of the reasons why inverting matrices is such an important aspect of modern computing.

In the next section, we look in detail at the assumptions that make OLS the best estimator.

A few assumptions are required so we can use our theorems to make OLS a valid estimation method. These assumptions are sometimes labeled A1 to A5 (A for assumption) and have a few variations. I list all these assumptions in quite a lot of (mathematical) detail here in an attempt to provide a mathematically solid intuition.

We now consider what happens if the above assumptions are violated. Fundamentally, if any of these is violated, then OLS is no longer the best estimator. But each assumption violation comes with its own flavor of trouble.

Now that we have a strong understanding of how OLS works, let’s discuss ways to expand or transform the base model to account for more use-cases. Once again, although I use OLS as an example here, all of these methods are valid for any estimation approach. Here I present a variety of methods individually, but they can be combined as you see fit for your own problem.

You will sometimes encounter the terms “classification” or “clustering” used interchangeably. This is fine but there is actually a mathematical distinction:

• Clustering relies on some defined measure of distance to a hypothetical “cluster of data”.
• Classification relies on building a decision boundary in your data’s space.

The problem both approaches are trying to solve is that of bucketing or labeling your data. Examples of this can be:

• Distinguishing a buying customer from or a browsing customer.
• Distinguishing a machine that is still behaving properly from one that has a strange behavior.
• Classifying events as “expected” or “outliers”.

Whereas a regression approach tries to find the relationship between continuous or discrete features and a continuous outcome, classification tries to find the relationship between continuous or discrete features and a discrete outcome. In short, the tasks are decided by the type of outcome variable we want.
Note that a lot of classification methods are binary: either you’re in a specified class, or you’re out of it. An easy way to transform any such binary classification into a multi-class algorithm is to do the classes one by one, considering each class on its own versus all the other classes.

In this section we mostly cover the intuition behind the implementation of the major classification algorithms. We focus in particular on the ones from the field of Machine Learning (as opposed to Econometrics). I won’t spend as much time on these models as I did on regression analysis, because the fundamental advice is the same: make sure your features fit your assumptions and is well-behaved.

As discussed in the introduction, the time serial-correlation of the data has a huge impact on our A1-A5 assumptions. A1 is unharmed but the following changes are notable:

• A2: Although the base model is still linear, the output is now the future feature: \( X_{t+1} = X_t \beta + \varepsilon_t \) with \( \mathbb{E}[\varepsilon_t] = 0 \)
• A3: Although in general homoskedasticity can still hold, it works differently because past errors remain over time: \( X_{t+1} = X_t \beta + \varepsilon_t = \left(X_{t-1} \beta + \varepsilon_{t-1} \right) \beta + \varepsilon_t = \dotso \)
• A4: Independence no longer holds: \( \varepsilon_t = X_{t+1} – X_t \beta = X_{t+1} – \left(X_{t-1}\beta + \varepsilon_{t-1} \right) \beta \). Now the \( \varepsilon_t \) are correlated with each other across timestamps.
• A5: Although the errors can generally still be distributed normally, the standard deviation assumption is different because of the changes to A4.

Unit roots are the crux of what makes time series hard to deal with. The concept of unit root actually comes from differential equations. In fact, considering the simple linear problem of \( X_{t+1} = X_t \beta + \varepsilon_t \), we can rearrange the terms to have a differential equation set-up: \( X_{t+1} – X_t \beta = \varepsilon_t \). If we divide by a small \( \delta t \), and bring it to 0 at the limit, this is in fact exactly a differential equation. The general context for a unit root is as follows:
Consider constant weights \( {a_i}_0^n \) and \( y^{(i)} \) the \( i-th \) derivative. The general model is:

\[a_n y^{(n)} + a_{n-1} y^{(n-1)} + … + a_1 y’ + a_0 y = 0\]

Such a general model has characteristic equation:

\[a_n r^n + a_{n-1} r^{n-1} + … + a_1 r + a_0 = 0\]

Where \( r_1, r_2, …, r_n \) are the potential roots of the general solution to the differential equation. The time series version of this problem is:

\[y_{t+n} = b_{1} y_{t+m-1} + … + b_n y_t\]

With characteristic equation:

\[r^n – b_1r^{n-1} – … – b_n = 0\]

As an example, if \( r_1, r_2 \) are the roots of a characteristic equation, then we can derive a closed-form solution for \( y_t \):

\[y_t = c_1 e^{r_1t} + c_2 e^{r_2t}\]

Note how this closed-form solution no longer relies on past data to model future data, but just on \( t \) itself. Although it would be nice, in general, we can’t solve these equations for price prediction.
We say that this differential equation has a unit root if \( r = 1 \) is one of its solutions. This unit root breaks a very subtle but important part of assumption 5. In fact, if we have a unit root, assumption 5 tells us that we get extremely fast convergence, but to a random value. In the next section, we look at a White Noise process, a very simple example to help us understand the effect of a unit root.

The simplest time series process is called the White Noise. The concept of White Noise is central to signal processing and many areas of physics that use concepts such as Brownian motion. Here is how it is defined:

\[\varepsilon_t = \rho \varepsilon_{t-1} + u_t\]

Where:

\[u_t \sim \mathbb{N}(0,1) \text{ and } \mathbb{E}[u_t] = 0\]

If we replace each term at \( t \) by its preceding term at \( t-1 \), we get an equation of the form:

\[\varepsilon_t = \rho^t \varepsilon_0 + \sum_{i = 0}^{t} \rho^iu_i\]

If \( \rho < 1 \), we can immediately see that old errors disappear over time as \( \rho^n \) for a large \( n \) is close to 0. This means only recent data matters. If \( \rho > 1 \), similarly we can immediately see that errors will compound exponentially.
The problems happen when \(\rho = 1\). In that case, the stochastic process looks like it could have a useful signal, but it doesn’t. In particular, from A5 we can derive that the estimator for \( \rho \) in a White Noise process will have the following property:

\[\sqrt(T) (\hat{\rho} – \rho) \sim \mathbb{N}(0, 1- \rho^2) \text{ if } |\rho| < 1\]

\[T(\hat{\rho} – \rho) \rightarrow \frac{1}{2} \frac{w(1)^2 – 1}{\int_0^1 w(r)^2 \,dr} \text{ if } |\rho| = 1\]

Where \( w \) is a Wiener process. This equation roughly means that the OLS estimator will converge fast to a random value that has nothing to do with your data.
The details of how to derive this result are not the most important. What matters is that you now know that if your errors have a unit root component, they will produce a non-trivial random signal that changes the fundamentals forever, as there is no decay over time.
This concept of unit root is linked to another concept: stationarity.

Note – \( \rho = 1 \text{ or } \rho \simeq 1 \): In theory, if \( \rho \neq 1 \), then we can assume there is no unit root. In practice however, it’s very hard to distinguish between \( \rho = 1 \) and \( \rho \simeq 1 \).

In a statistics context, stationarity is the idea that a given probability distribution is constant over time. Implicitly, this is partly what A4 gives us for OLS since it forces the standard deviation to be constant over time, regardless of cross-correlation effects. Notationally, there is no \( t \) component on the standard deviation.
There can be many reasons for why your data exhibits non-stationary behavior: a change in market regulation means the data-generating process fundamentally changed, technology improved and so trading methods changed, the type of fuel used changed, etc.
In particular, if you believe there is a unit root in your data, it will exhibit non-stationary behavior. This is very bad, because your regression estimator will almost invariably fit to noise instead of signal. Formally speaking, stationarity is defined as follows:
Let \( {X_t} \) be a stochastic process and \( F_X(x_{t_{1}}, …, x_{t{_n}}) \) be the cumulative distribution function of the unconditional joint distribution of \( {X_t} \). Then \( {X_t} \) is said to be strongly stationary if:

\[F_X(x_{t_{1}+\tau}, …, x_{t{_n}+\tau}) = F_X(x_{t_{1}}, …, x_{t{_n}})\]

Note the indeces: this equality means that the distribution of our data is constant across linear time translations. This equality can be broken by trends which over time affect the mean, or the variance of our data. Other types of changes can occur, where for instance the function \( F \) itself changes.
In practice, we mostly care about a weaker form of stationarity, covariance stationarity:

\[\text{Cov}(x_{t_{1}+\tau}, x_{t{_n}+\tau}) = \text{Cov}(x_{t_{1}}, x_{t{_n}})\]

This is very similar to the previous equality, except we allow for the full distribution to change as long as the covariance (and implicitly the mean) stay constant across time. And of course, unit roots break this equality.

The main ways in which we test for stationarity use hypothesis testing. There is a refresher on hypothesis testing in the next section.

Now that you know that your data is non-stationary, we look at a list of common data transformations you can perform that can get rid of your stationarity issues. Once you’ve performed one of these transformations, you can run your ADF and KPSS tests again to see the difference.

Most of the transformations introduced below work on the basis of removing some information from the data in the hopes of mostly removing the noise and keeping most of the signal of interest intact. A good way to check for that is to look at the cross-entropy or at the Pearson correlation between your original data and your transformed data, and to choose the transformation that maximizes the cross-entropy or correlation while still succeeding the hypothesis tests. There is no hard rule for how much signal to sacrifice to add significance to your hypothesis test, you have to decide on that trade-off based on your use-case.

When we presented OLS, we first talked about how best to place a line in a scatter of points. We later explained how this decision is essentially made by defining our error. More accurately, the decision is made by how we minimize our defined error. In a traditional Machine Learning course, instead of focusing on defining a linear function and looking at the assumptions needed to make our estimator good, we start with a function that we want to minimize and we see how well it performs.

In fact, in general, ML spends a lot of time thinking about which error function is optimal for which problem. Deciding on your error function to minimize effectively changes your forecasting approach. Even more than that, how you solve that minimization problem can change your forecasting approach. As a result, ML cares a lot about how to solve those minimization problems.

You will often hear talk about the shape of your problem being good, which refers to the geometric shape of the error function to minimize being well-behaved, i.e. being easily differentiable (not having kinks and being smooth), and having easy-to-find optima. This is important because unlike our analytical solution for the OLS estimator, ML wants to be able to minimize arbitrary functions. As such it concerns itself with numerical solutions as opposed to analytical ones.

A formal mathematical exploration of these terms and numerical solutions is out of the scope of this guide, but the intuition is that it’s easier to find a max or a min on a smooth curve than on a curve with lots of sharp twists and turns, and that different approaches deal with sharp twists and turns more or less well.

For the curious, and in no particular order, here are some numerical optimization algorithms to look up:

• Stochastic Gradient Descent
• Adam
• Nesterov Momentum
• Adagrad
• Adadelta

There are a lot of heuristics and methods guiding how to write your own error function, choosing the right optimizer, and optimally training your model. I encourage you to look at the extra resources listed at the end of the guide if you want to learn more about this topic.

One of the most useful techniques from Machine Learning is the train-test split. Like many other ML methods, its focus is on improving out-of-sample performance by reducing overfitting. The core idea is to separate your data into two parts. Using your training data, you fit your model the way you usually would. But measuring the performance of your model on your training set could easily lead to over-estimating your goodness of fit by fitting too closely to that specific dataset. Therefore, we measure the validity of the model on a separate, not-before-seen, testing data set. The training data is biased one way or another, and in fact the out-of-sample data is also biased one way or another. The reason why the out-of-sample bias is better is because a priori, not having trained on it, our model is equally likely to perform better or worse on it. That bias is much easier to live with than overfitting bias.

You can also divide your data into more chunks, usually a train, a validate, and a test set (note the terms validate and test are used interchangeably across different textbooks). That way you train your model on your train set, you pick the best (hyper)parameters for that model on your validate set, and you pick the best overall model on the test set. Separating the data between the part we know and the part we don’t know for each decision ensures that we don’t overestimate how much we actually know about unknown future data.

To measure a model’s performance, more data is always better because it gives you more confidence in the measure of the performance. Therefore, losing some of your data to training reduces your certainty on how well your model will perform. Using only a test set means that you sacrifice how confident you are in your test to be more confident about your lack of overfitting bias. So really there are two different types of biases we are juggling: the bias of our measure of fit and the bias of our out-of-sample performance. With less data to test over, our confidence in our performance is lower, but we have more confidence that whatever bias the test set has is symmetric, neither overestimating nor underestimating performance a priori.

Cross-validation aims to solve the problem of the test set being too small to gain good confidence. It works by randomly selecting data points in any order to form a subset of your data. That subset is then split into train and test and your model is fitted and evaluated. This process is repeated many times, sometimes hundreds of times for larger datasets. Each of these subset is called a fold. Then you use the average of your many test measures of fit across all your folds as your overall measure of fit. This method effectively resamples your data to maximally exploit its stochastic nature, as if you were drawing samples from the population many times.

Although cross-validation is very powerful to get a better (i.e. less biased, less over-fitted) measure of fit, you must exercise care when using it. If you measure on the same testing set again and again, in the end the test set becomes part of your training data and is no longer out-of-sample. Therefore, if you resample your data so many times that you train and test over your entire data set hundreds of times, your cross-validation approach will actually result in overfitting.

Of course, everything we just discussed becomes even harder when applying it to time series.

The time series nature of your data needs to be respected for your model to properly capture the signal you want. As such, the cross-validation step of randomly selecting your data in any order destroys any hope of training a sensible model. There are a variety of ways of still performing cross-validation, but fundamentally they all amount to selecting a block of data instead of selecting data randomly. Then you move that block of data across time. The decisions to make revolve around: how much data goes into each block, how much do you move the block across for the next fold, and how often do you retrain.

Note – A well-specified model: It is not impossible that the features you define for your model specify it so accurately that you can omit the time aspect of your data. In practice, I’ve never seen this to be the case, but I have seen many cases where there wasn’t enough data to work with and so pushing the analysis with this assumption was better than not doing anything.

I explore some time-series cross-validation options below:

There are plenty of tutorials online that explain how to perform a simple z-test or how statistical significance works. Instead I want to focus on the logic of hypothesis testing, what such a test can tell us, and most importantly what it can’t. Fundamentally, hypothesis testing is a framework for asking questions about our data which must always be in the following form: given the data that I have and what I assume to be true about it, how unlikely are other options?