The following notes come from the course Machine Learning in Finance and Insurance, taught at ETH Zurich by Patrick Cheridito, Professor of Mathematics at ETH Zurich and Director of RiskLab Switzerland.
Feel free to reach out to me with any comments, clarifications, or corrections.
Index
- Basic Notions of Statistical Learning
- Linear Regression
- Gradient Descent
- Logistic Regression
- Kernel Methods
- Neural Networks
- Classification and Regression Trees
- Bagging and Random Forests
- Gradient Boosted Trees
- Dimensionality Reduction and Autoencoders
- Generative Models
Projects
1 Basic Notions of Statistical Learning
Definition of a -algebraFirstly, we remember that a -algebra is a non-empty collection of subsets of that contains itself and is closed under complementation and countable unions, i.e. it satisfies the following:
- contains the whole space
- closed under complementation:
- it is closed under countable unions:
From this definition, we give some examples:
- Trivial -algebra: (the smallest -algebra)
- Power set: (collection of all subsets of , the largest possible -algebra)
- Borel -algebra (smallest -algebra generated by all open sets in , it is the essential foundation for measure theory and real analysis)
Intuition: in , consider all the intervals and perform all three actions that define the -algebra (complements, countable unions, countable intersections, where the last one nicely derives from the definition of -algebra): the resulting set is the Borel -algebra and represents every subset of that can be obtained using standard mathematical operations).
We also recall that a function is Borel measurable if:
Intuition: if the preimage of every Borel set in is a Borel set in .
The prediction problem
Consider the following:
- a probability space , i.e.:
- a non-empty set , the sample space
- , a -algebra of subsets of ,
- a probability measure
- a feature space with Borel -algebra (example: )
- a closed or open target space with Borel -algebra (example: )
- some random elements (inputs), (outputs)
We then consider , , and as the distributions of , , and , respectively, and as the regular conditional distribution of given , that is:
- is a probability measure on
- , is -measurable
- , -almost surely.
It then follows that, for all -integrable , we have the disintegration of :
We now go back to the prediction problem: we want to find a measurable function s.t. , i.e. a function that allows us to minimize the expected loss (also called risk), defined as:
for a measurable loss function .
Then, in machine learning, it is common to solve the empirical loss minimization problem!
Regression Functions and Irreducible Error
Lemma (theoretically best predictor with respect to a given loss function )
Let be a measurable function such that, for -almost every , minimizes
Then
for every measurable function .
Note that is the ideal predictor we would use if we knew the true conditional distribution of given . For each , if we predict a value , the conditional expected loss is
The function chooses the prediction that minimizes this conditional risk:
From now on, we suppose these assumptions from the lemma hold and we call:
- regression function (or, more generally, the Bayesian-optimal predictor)
- irreducible error (even with the exact conditional distribution, randomness in given can prevent perfect prediction)
Example: Square loss
We now consider the following example with , and a random variable with .
Then, :
so the mapping is strictly convex with:
- a unique minimizer (under square loss: the best constant prediction of is its mean)
- a minimal value of (the minimal possible error is its variance)
We now assume , by disintegration we obtain:
Therefore we have
for -almost every . By the Cauchy–Schwarz inequality, this also implies
for -almost every .
The conditional risk under squared loss is then:
The second term does not depend on , and the first one is minimized when:
Therefore, the unique minimizer is:
Then, we easily obtain the least-squares regression function and its minimum loss (risk):
and
Example: pinball loss
We now consider and a loss function , with and the pinball loss function defined as:
In this formula, is the quantile we want to predict and it determines the asymmetry of the loss. Indeed, in the pinball loss function, the penalty is proportional to , therefore:
- for : underprediction is penalized more
- for : overprediction is penalized more
- for : errors in both directions have the same cost
We first note that:
We now consider a random variable with . In this case, is convex in and:
So is a minimizer of if and only if and , if and only if is an -quantile of .
Now let , for , be an -quantile of the conditional distribution . Then we have:
- (quantile regression)
For the special case , is a median of , and is the minimal MAE.
Empirical loss (empirical risk)
Let with be independent copies of , called training data.
The empirical loss/risk of a measurable function is . In the case of , we can write:
and we can call this the weak law of large numbers.
Intuition: as the amount of training data grows, variance approaches zero.
Hypothesis Classes
A hypothesis class is a family of measurable functions , e.g.:
- all affine functions (i.e. weighted sums of features )
- second-order polynomials
- splines (local polynomial pieces joined together, producing a smooth flexible curve)
- trees (sequential if-then rules with piecewise constant regions)
- support vector machines (separate classes with a maximum-margin boundary)
- neural networks (several nonlinear transformations)
Ideally, we would like to minimize over all measurable functions .
However, in practice, we try to find a numerical solution to the empirical loss minimization problem:
Note that can be produced by either:
- a deterministic algorithm, in the form , where is a measurable function of the training data;
- a stochastic optimization algorithm, in the form , where is a random element independent of the training data and is measurable (note that a stochastic algorithm can produce different results with the same training data due to the presence of the noisy term )
Given the numerical solution , we now formalize the expected loss decomposition for the square loss function.
We first define the conditional expected loss given that :
If we define:
then we can write the conditional expected loss as:
The first term is the conditional bias squared, the second term is the conditional variance, and the last term is the conditional irreducible error.
Furthermore, the expected loss is (the calculations are omitted for economy of space; readers are encouraged to compute them by hand):
The first term is the average bias squared, the second term is the average variance, and the last term is the irreducible error.
For a given size of the training set, we can then define:
- the bias as the distance of the average prediction function from the true regression function
- the variance as the variation of the data-dependent (noisy) prediction function
and... of course, we should aim for a good balance between them!

Goodness of fit
If we consider the same square loss function, a training data set , with , and a test dataset , with , and a prediction function trained on the training data, we can define the in-sample goodness of fit as:
where
- SSR = (sum of squared residuals)
- SST = , with (total sum of squares)
Note that, with the same approach, we can define the out-of-sample , using and from the testing dataset.
As an obvious intuition, measures prediction power (higher is therefore better!).
Approximation and Sampling Error
If is not flexible enough to approximate well (i.e. the hypothesis class does not contain the ideal predictor ), we can incur an approximation error, that measures the limitation of the current model family itself in comparison with the ideal model and it is defined as:
If the empirical loss minimization problem, i.e. , has a solution, we call that solution (note that in this case is random, since it depends on ).
If exists, it is random because it depends on the random training sample. In this case, its sampling error (estimation error) is the difference between its true loss and the population loss available in (it results from minimizing the empirical loss instead of the expected loss; it typically decreases with increasing sample size ), i.e.:
Optimization error
Consider a numerical solution of , with:
- measurable and a -valued random element independent of .
We can then define the direct optimization error as:
Note that the first term is the empirical loss achieved by the numerical solution, the second is the lowest empirical loss achievable by any function in . Note also that this error is different from the sampling error:
- the optimization error measures failure to solve the training problem
- the sampling error measures the gap between training performance and test data performance
Decomposition of the Generalization Error
Once we have defined the previous metrics, we can define the following:
For a given and an increasing (i.e. the set of models we can choose from is increasing):
- approximation error is decreasing
- sampling error tends to increase
- there's a tradeoff between approximation error and sampling error (similar to the bias-variance tradeoff)
- for small data should be simple (and for big data, it can be complex!)
Training Error and Test Error
Once we have defined , we condition on it and so it is no longer random.
If we consider training and test data sets, we can define:
If is significantly smaller than , the model is most likely overfitted.
We can then calculate the sample variance:
which, if , follows the Law of Large Numbers and converges, as , to .
Furthermore, by the Central Limit Theorem, we have:
Therefore, we have the following property:
That is a confidence interval with confidence level , where is the -quantile of the standard normal distribution.
We now come back to the overfitting problem: what can we do about it? First, we can reduce the complexity of the hypothesis class, i.e., by considering simpler models. Alternatively, we can regularize the empirical loss minimization problem by penalizing the complexity of potential prediction functions , which, practically speaking, means:
for hyperparameters (strength of the Lasso penalty), (strength of the ridge penalty).
We call LASSO and ridge regression; together, they compose the elastic net.
In general, LASSO penalizes the absolute value of the coefficients and can set them to zero, and it is defined as:
Therefore, LASSO performs feature selection, i.e. limits the features with lower explanatory power.
On the other hand, ridge regression defines a penalty equal to:
It discourages large coefficients but does not make them exactly zero; it smoothly shrinks them around zero.
Ridge regression can be useful when features are correlated. In this situation, LASSO may select one feature and discard another similar one, while ridge tends to distribute the coefficients across them.
When both penalties are present the method is called elastic net, since it combines LASSO's ability to set coefficients to zero and ridge regression's shrinkage of correlated features.
2 Linear Regression
Ordinary Least Squares (OLS)We consider the model , where is independent of .
Let , for , be independent and identically distributed realizations of . In this case, we can write the model in matrix form:
Here, is the design matrix and is the vector of coefficients. Typically, we have (i.e. we have more observations than parameters).
In this scenario, we define the empirical loss minimization under the square loss function (OLS, ordinary least squares) as:
where for all .
Since the function is convex in , we can minimize it using partial derivatives. A minimizer satisfies
which gives the following normal equation:
To solve the normal equation, we note that if the columns are linearly independent, is invertible, and the unique solution of is defined by:
If we condition on (i.e. if we rewrite ), we obtain:
Thus, has a multivariate normal distribution with mean and covariance matrix ; in other words, is unbiased.
We also note that is always a symmetric positive semidefinite matrix, for (the number of columns of ). In this case, we consider the spectral decomposition of :
for an orthogonal and a diagonal with diagonal entries .
In this case, we define the pseudoinverse of as:
Lemma (columns of linearly dependent)If the columns of are linearly dependent, the normal equation has several solutions, and
is the solution with the smallest -norm.
Recall that the -norm is defined as:
Recall: if the columns are linearly dependent, and is singular, so its determinant is zero.
In practical terms, this means that the coefficients are not uniquely determined by the data, so the pseudoinverse then selects the coefficient vector with the smallest Euclidean norm.
Consider, as a real-world example, that we want to predict house prices and use a model that considers both:
- house area in square meters
- house area in square meters
In this case, the variables contain the same information, i.e. , therefore the corresponding columns in the design matrix are linearly dependent and the model cannot uniquely determine separate coefficients (in practice: one variable should be removed).
Proof of the LemmaWe want to prove that satisfies the normal equation defined above.
Let
If , then
Therefore, . Conversely, if , then . Hence,
It follows from the rank-nullity theorem that
Since and the two spaces have the same dimension,
Recall that is the image of a linear map, also called its range or column space:
Moreover, by the spectral decomposition:
where is orthogonal and
Therefore we have:
where are the first columns of , corresponding to the positive eigenvalues. In particular, for some coefficients , we have:
Denote now by the orthogonal projection onto . Consequently,
because already belongs to . Thus, satisfies the normal equation.
We now prove that is the solution with the smallest -norm.
Since we have the following:
Since is singular, there are other solutions, and any other solution can be written as . Since both and satisfy the normal equation,
Subtracting gives , so we have:
So is orthogonal to , by the Pythagorean theorem,
Therefore, is the solution of the normal equation with the smallest Euclidean norm.
Singular Value Decomposition (SVD) Theorem
Let be an matrix of rank . Then all of the following hold:
- there exists an orthonormal basis of consisting of eigenvectors of with eigenvalues and (the eigenvalues corresponding to the independent directions are positive, while the others are zero);
- for , is an eigenvector of with eigenvalue ;
- are orthonormal eigenvectors of with eigenvalues ;
- with (this is the SVD of );
- the pseudoinverse of is given by ;
- in particular, for all , is the minimizer of with the minimal norm.
The proof is left to the reader... or to the Giacomo of the future with some more time!
CorollaryLet be an matrix of rank .
Then has a singular value decomposition of the form
- is an orthogonal matrix;
- is an orthogonal matrix;
- and are diagonal and matrices, respectively, of the forms:
For , where are the positive eigenvalues of .
Regularization by Truncating the Pseudoinverse
If the design matrix (recall: composed of , with observations and features) has small positive singular values:
- is sensitive to the noise
- the numerical computation of becomes unstable (the pseudoinverse contains the term , so if the singular value is small, the noise component in that direction is multiplied by the large factor ).
In this case, we can define a threshold and define the truncated SVD and the truncated pseudoinverse as:
Conditioned on , we can also define as:
Intuition: is the signal that survives the truncation, and is the amplified noise.
We can consider as a threshold hyperparameter that balances bias and variance. Indeed, for an increasing :
- the bias of , , increases because contains less information
- the Frobenius norm of the covariance matrix decreases because more terms are removed (recall that the Frobenius norm is defined as ).
Ridge Regression
As briefly mentioned in the introduction chapter, we can introduce a penalty term to force the estimate to have a small Euclidean norm. That is, we can solve the following optimization problem for a hyperparameter :
As always, is a minimizer if and only if:
If we solve for , we obtain the following solution :
where are the positive eigenvalues of .
Conditioned on , we obtain:
We clearly see that, for an increasing :
- the bias of , , increases;
- the Frobenius norm of the covariance matrix decreases.
Generally speaking, we do not penalize the intercept in ridge regression, since it represents the baseline of the response, so we exclude it from the penalty. The optimization problem therefore becomes, for a hyperparameter :
LASSO (Least Absolute Shrinkage and Selection Operator)
Again, as mentioned in the introductory chapter, we can introduce a penalty term to force to have a small -norm; i.e., we can solve the following optimization problem for a hyperparameter :
Note that this is a non-smooth convex optimization problem, i.e. the absolute-value function has a kink (the derivative does not exist) at zero and is convex but not differentiable there.
For this reason, the -penalty tends to shrink coefficients to zero and, therefore, leads to a sparse solution (intuition: minimizing the sum of absolute values makes exact zeros attractive, so some coefficients can vanish entirely).
Variable Standardization
Consider now a solution that solves . The first line of the vector equation gives:
Here, and denote the sample averages.
If we consider the case , then , so the first column of 1s in (recall that the design matrix has a first column full of 1s) is redundant!
To avoid this, we can standardize the variables and ; i.e. we can transform them as follows:
Here,
Therefore, we can proceed with standardized linear regression, i.e. solving the following optimization problem:
Once we find a minimizer for this problem, the prediction for a new data point is
Generally speaking, standardization is unnecessary
- for OLS without regularization
- when the original units are meaningful and we want interpretable coefficients
Furthermore, standardizing the input variables is useful:
- in regularized regression (e.g. truncated SVD, ridge, LASSO, and elastic net) because penalties depend on the magnitude of the coefficients
- for comparing coefficient sizes, since standardization makes them easier to compare across different scenarios/input variables
Secondly, standardizing both the input and the target variables is useful:
- if we want standardized regression coefficients
- if we are comparing models across targets with different scales
- in this case, the intercept is not optimized (since it is zero from standardization of ), so the regularization does not penalize the intercept (since it estimates only the coefficients).
Cross validation
Cross-validation refers to a family of model validation methods for assessing how a trained model generalizes to unseen data. One particular cross-validation method is called k-fold cross-validation, and it consists of:
- randomly partitioning the data into subgroups
- retaining a single subgroup as validation data and using the remaining subgroups to train the model
- repeating the process times and calculating the empirical loss each time
- using the average validation loss over groups as an estimate of the generalization error
Note that this method can be used to find the optimal regularization parameters (e.g. the threshold or the hyperparameters in LASSO and ridge regression) by picking the regularization parameters that yield the smallest average validation loss.
In this case, if cross-validation is used to find hyperparameters, the resulting prediction function should be tested and evaluated on an independent test dataset.
Numerical and Categorical Data
We now spend some time understanding the different types of data common in ML.
- Numerical data: data in or for which the natural order has meaning; they can be both discrete and continuous
- Categorical data: discrete data without a natural order (e.g. "married"/"unmarried", postal codes, etc.)
- Unclear cases: some data do not fit into one category (e.g. could be treated as either categorical or numerical data)
One-hot and Dummy Encoding
Suppose categorical data take values in , but the natural order does not have any meaning.
If there are only two categories, such as A and B, they can be encoded with 0 and 1 easily.
However, things get a bit more complicated when , i.e. when we have more values available. In this case, we have the following alternatives:
- One-hot encoding: if , it can be embedded in via:
- Dummy encoding: if , it can alternatively be embedded in via:
(note the position of the 1s in the vector!)
- Random embeddings: sometimes can be embedded in for with alternative approaches.
Feature Engineering
Sometimes the performance of a linear regression can be improved significantly by transforming and combining features.
Suppose we have the original features .
Given this initial data set, there are operations we can perform, i.e. many we can apply to the original _(e.g. , , , ), but note that these operations often require expert knowledge.
Data cleaning, missing data, and imputation
Data cleaning refers to the practice of removing duplicates or irrelevant observations or, more complexly, identifying extreme values that could skew the statistics.
Furthermore, in the presence of missing data, i.e. given a row vector , if it has a missing component , we can:
- remove the whole row (but in this case we are reducing the amount of data we have!)
- impute the missing data by:
- using the average of
- using the most frequent category
- predicting the missing data point from existing data
- drawing the missing data point from the learned conditional distribution
3 Gradient Descent

Imagine you are lost on a foggy mountain and you have to reach the lowest valley point (i.e. you have to come back to the parking lot after your hike).
Since you cannot see anything, by carefully analyzing and feeling the slope of the terrain under your feet, you find the direction that goes downhill more steeply... this could be a nice starting point to understand gradient descent, but now, let's deep dive into the technicalities!
Suppose we have a function h we want to minimize and suppose is a L-Lipschitz function, i.e. for some constants :
We try now to minimize h with the Grid Method:
- we evaluate h at the grid points
- we solve , i.e. we choose the smallest value
- we now want to compare with the true minimizer ; to do that, denote by the grid point closest to
- so we have:
- if we apply the Lipschitz condition, because h is L-Lipschitz, we have
This tells us how much worse the grid solution can be than the optimum solution .
Note that, if we want an accuracy of , we need:
so we need function evaluations... and this, of course, is infeasible in high dimension (we call this problem the curse of dimensionality!)
Convex Function
We now consider much nicer functions... i.e. convex functions!
Recall that a function is convex if, for all and :
Classical 2D intuition: if you take two points , the point lies on the segment between them. The convexity condition says that the function value at this intermediate point is the straight line joining and .
Recall also that every norm is a convex function.
We define now the subgradient of h at as the vector such that:
Note that the symbol represents the dot product. Recall also that, from a geometric point of view, represents the plane touching the graph of at . The inequality says that the plane always lies before the graph of .
We now define the subdifferential as the set of all subgradients of h at x.
If h is convex, is, for all , a non-empty subset of .
To prove that, we fix . Its epigraph is then convex:
Furthermore, the point lies on the boundary, so the supporting hyperplane theorem says that there are such that:
Since the epigraph extends upward, , so we can set and, taking , we get:
Therefore, and so we can conclude that .
Furthermore (and most importantly), if a convex function is differentiable at x, then is a singleton and we have .
Furthermore, is a minimizer for a convex function if and only if .
_Recall that, given a function , the gradient is the vector of all partial derivatives: it tells how much h changes when each coordinate changes and points in the direction of the steepest increase (therefore, points in the direction of the steepest decrease).
Gradient Descent
Consider now , a convex function with a minimizer . We apply the gradient descent method.
- First, we start with (which can be deterministic, i.e., chosen with a heuristic, or random)
- we apply the gradient steps. For all and learning rate :
- Then, from the definition of the gradient with vector and the dot product given above, we have the following:
- that translates into:
- Therefore, we obtain the following:
Furthermore, if for all , then:
So if and , then for (or, in other words, the minimum found with gradient descent converges to the true minimizer).
(Note that the required condition for can be easily satisfied, for example by setting : we have and .)
However, in practice, computers do not allow ... so if the number of gradient steps is fixed, we choose to minimize:
In this case, for an accuracy , we need (and note that this does not depend on the dimension):
(Note that doubling the desired precision requires approximately four times as many gradient steps.)
Stochastic Gradient Descent
Let's jump back to the foggy mountain: we are still at the top and we still need to find the parking lot. Instead of carefully testing the incline across the entire terrain before taking a step, we quickly feel the slope under one foot and immediately take a step.
Now let's introduce some technicalities: consider a probability space and a loss function for a random observation using the model parameters :
where is convex in , measurable in , and for all .
The overall objective function is , which is the expected loss over all possible observations.
We first state that is convex and that, for every measurable subgradient ,
is also a subgradient of at .
However, might be difficult (or sometimes impossible) to compute. So we can use independent samples and approximate the gradient. Technically, let , with , be independent copies of .
Then fix and set:
For , we call it standard stochastic gradient descent (SGD); for , we call it SGD with mini-batches, which can reduce the variance of the gradient estimates.
(Intuition: instead of calculating the exact gradient using all data, we estimate it from one or more randomly selected samples and use that estimate to update the parameters.)
Formally, we follow these steps:
- we start at (that, again, can be deterministic or random, independent of )
- we take the stochastic gradient steps, for all , with random:
- we denote by the sigma-algebra generated by (i.e. all the information known from the algorithm up to iteration ). Note that is -measurable; therefore, once the current parameters and past history are fixed, the stochastic gradient is an unbiased estimator of the true gradient. Formally, we write:
Intuition: SGD may take a wrong step because its gradient estimate is noisy, but, thanks to independent copies of the random loss function that are independent of the past, on average its direction is the same as the exact direction of non-stochastic gradient descent.
- we now assume has a minimizer . Then we have:
which implies
- Again, if , we can write:
Again, if and , then as .
One can also show a stronger claim: for a convex , almost surely as (and not only for its expected performance , as done before).
Note that SGD is also used for non-convex problems, e.g. to train neural networks, even if it is not guaranteed to converge to the global minimum.
Indeed, a convex loss function has no "bad" local minima: every local minimum is also global; therefore, if SGD converges, it can reach the best possible loss .
Neural networks often contain flat regions, local minima, and saddle points (critical points where the slopes in orthogonal directions are zero, but the point is not a local maximum or a local minimum). However, SGD is still used because it is computationally practical and often finds parameters with good predictive performance, but in theory it cannot guarantee a global minimizer (it only reaches a stationary point).
ValidationLet , with , be independent copies of , independent of , and call them the validation data set, i.e. they have the same distribution as the training loss but are independent.
To validate the model, we monitor the empirical loss, defined as , on the validation dataset.
- if the validation loss stops decreasing, we decrease the learning rate
- if the validation loss starts to increase, we stop SGD
- if a validation dataset is used in training, the resulting must still be evaluated on an independent (out-of-sample) test dataset.
Generally speaking, we can say that with training data we fit parameters, with validation data we choose the training settings, and with testing data we perform a final unbiased evaluation.
Example: suppose we have data and we want to predict .
We start using the squared error loss, defined as .
We then split the data into training and validation data:
- we train on training data only, by repeatedly choosing a random mini-batch of training pairs and calculating its average loss and gradient; then we recursively update:
- after each epoch (i.e. a complete iteration through the entire training dataset; if we have training observations and a mini-batch contains observations, the epoch contains approximately gradient updates), we evaluate the current model on all the validation data and obtain:
- we then store the parameter (chosen from all the epochs) that achieves the lowest validation loss.
4 Logistic Regression
We first recall some basics of probability.
A random variable is Bernoulli distributed with parameter (and in this case we write ) if:
Now we consider , where:
- (logistic function)
In other words, we have:

Therefore, the probability mass function of is:
And, following the same reasoning, the probability mass function of independent (note that they all have the same parameter ) is:
Recall also that the sum of Bernoulli random variables follows a binomial distribution, formally:
Going further, the likelihood function of is:
The log-likelihood function of is:
Recall that the likelihood measures how plausible a parameter value is given the observed data, so is treated as unknown and the observations are fixed. Remember also that a larger implies that the proposed makes the observed sequence more likely (maximum likelihood chooses the value of that best explains the data).
Training and Estimation with Logistic FunctionWe now suppose that we want to regress , a binary outcome that follows:
where
We consider training data , independent and identically distributed realizations of .
We initially set, for and :
We define the empirical loss minimization problem (conditional on negative log-likelihood minimization):
We immediately notice that this is a convex minimization problem for that can be solved with (stochastic) gradient descent.
We then obtain the solution of the empirical loss minimization problem, , and the predicted probability of given :
Note that the prediction does not give the value of , but just the probability , so we must choose a decision threshold such that we predict if , and otherwise.
We now consider test data , with , independent and identically distributed realizations of independent of the training data.
Then, for each case , we can give the following classification:
- (positive) if
- (negative) if
- (true positive) if and
- (false positive) if and
- (true negative) if and
- (false negative) if and
We can then define accuracy (i.e. the percentage of correctly predicted cases) as:
However, note that when dealing with imbalanced data, the accuracy metric can be misleading. Indeed, if one of the two classes is much smaller or larger than the other, the accuracy ratio might not be the right measure since we may have very high accuracy just because we are always predicting the larger class (and never predict the smaller class).
Therefore, we can define other performance measures:
- True positive rate (recall, sensitivity):
- False Positive Rate
- ROC (Receiver Operating Characteristic): it shows the profile for
- AUC (Area under the curve): larger is better; randomly guessing produces the diagonal with AUC , therefore AUC is worse than random guessing
- Positive predictive value (precision)
- F1 Score

Suppose that we want to classify a good (P) and a bad borrower (N).
In this case, we may aim to minimize the FDR (the false discovery rate) or, similarly, maximize the true positive rate.
Furthermore, note that in this case a small AUC is desirable, since predictions should be independent of a protected attribute.
5 Kernel Methods: Support Vector Machines and Reproducing Kernel Hilbert Spaces
A support vector machine (SVM) is a method for supervised learning that began attracting attention in 1990. The name comes from:
- machines that turn inputs into outputs
- vectors that support decisions
We recall that supervised learning uses labeled data (inputs with known answers) to learn to predict an output; it is usually used for classification and regression. On the other hand, unsupervised learning uses unlabeled data (data without known answers) to discover hidden patterns or structures, often for clustering, dimensionality reduction, and anomaly detection (some common algorithms: K-means, hierarchical clustering, PCA, autoencoders, etc.).
In contrast to other ML methods, numerical optimization is well understood for SVMs, so they are reliable and interpretable. Furthermore, the theory behind SVMs is frequently used in the field of ML; for example, large neural networks can be viewed through the lens of kernel methods (neural tangent kernels).
SVM for Binary ClassificationSuppose we want to classify points in into two categories, . That is, suppose we have points with labels , and our goal is to find a linear classifier separating yellow and red points.

Different classifiers are possible, and each is represented by a black line below.

The geometric idea behind SVMs is that each classifier has a safety margin, represented in the picture by a gray shaded area. In this context, we want to choose the classifier that maximizes this margin.

For our data, we have the following optimal linear classifier, where the dashed lines are parallel shifts of the classifier that touch both point clouds. We call these touching points support vectors of the classifier.

However, note that data points are not always that well behaved... For example, see the following:

So the idea behind SVMs for nonlinear data is to lift data into a higher-dimensional space where they are linearly separable...

... so a linear classifier in the high-dimensional space corresponds to a nonlinear classifier in the low-dimensional space.

Technically speaking, we need to define the decision hyperplane.
Let and for , and assume that linear classification is possible, i.e. there exist (weights) and (bias) such that:
Intuition: the classifier defines a score , and the sign of the score gives the predicted class.
So each classifier specifies a decision hyperplane, i.e. the dividing boundary between the two classes, where the vector controls the orientation and shifts it:
In particular, we want to choose a valid classifier , normalized such that , that maximizes the distance between the data and the decision plane. For a correctly classified point, this distance is , so:
Equivalently, let solve the hard-margin optimization problem:
Then .
Lifting the data and slack variablesWith the following optimization problem, we could have two issues:
- linear separation not possible (i.e. not all the constraints can be satisfied)
- we may not want to satisfy all constraints to avoid overfitting
Therefore, we can do the following:
- lift to a possibly infinite-dimensional Hilbert space with a feature map and use as the classifier
- introduce a slack variable , which allows the margin constraints to be relaxed in exchange for a penalty in the objective function, leading to the adjusted optimization problem:
We recall that a Hilbert space is a vector space with an inner product (i.e. a generalized dot product) where limits of converging sequences stay in the space. Ordinary Euclidean space is an example, but Hilbert spaces can also have infinite dimensions.
Intuition: the slack variable measures how far point falls short of the required margin. A positive slack can mean that the point is correctly classified but inside the margin, or that it is misclassified.
Furthermore, a large C (bias-variance knob) means violations are expensive, so the fit is tight to the data, on the other hand, a small C implies that the margin is wide, the fit is smoother and more errors are tolerated (with higher risk of overfitting).

- Points correctly classified on or beyond the margin have .
- Points correctly classified inside the margin have .
- Points on the decision boundary have , and misclassified points have .
Note that the constraints and the requirements imply:
where the hinge loss is and .
At the optimum, the slack variables equal these lower bounds, so we can rewrite the optimization problem as:
We now recall that , so we can use the fact that the function space is itself a Hilbert space with norm:
so the optimization problem becomes:
Intuition: we stop optimizing over weights and optimize over functions. Until now, the unknown was the vector and the classifier was . But, in general, we care about , not about , so we parametrize the problem in terms of by considering that each gives a function.
Therefore, we can talk in terms of a space of functions that, in this case, is called the reproducing kernel Hilbert space (RKHS) associated with its feature map.
Support Vector Regression (SVR)
We initially recall that in regression we want to find a function f(x) that predicts a continuous y. However, OLS penalizes every error. In contrast, SVR takes a different approach and penalizes only errors bigger than a threshold (later denoted by ) and looks for a function that is as simple (flat) as possible.
In particular, the goal of SVR is to find a hyperplane that best approximates the relationship between the lifted inputs in (the feature map) and the continuous output .
In this setting, we also notice that the model is not linear in the original space, but this becomes linear in the lifted space, as .
Tube-SVR looks for a hyperplane that keeps the predictions in an -wide tube around the fitted function. Deviations inside the tube () do not bring any penalty, but deviations outside the tube are penalized in proportion to their distance from the boundary.
The optimization problem for Tube-SVR, assuming is fixed and represents the -tube in which prediction errors incur no penalty and/or the noise the model tolerates, is:
subject to
where is the regularizer, since small implies a flat and smooth function that is less sensitive to input changes and therefore less prone to overfitting (same role as the margin term in the SVM). are the slack variables and they measure how far each point falls outside the tube. Then, the term penalizes the total slack (again: with large C, errors are expensive, so the model fits tightly, increasing the risk of overfitting).
Similarly to classification, if we substitute the optimal slack , we can rewrite the optimization problem in as:
where is now the -insensitive loss function, defined as , which is "insensitive" since it produces zero loss for errors smaller than .
Note that the user-friendly square loss is a very popular choice for SVR.
Example: in the following graph, we can nicely see SVR for .

Kernel Functions and Reproducing Kernel Hilbert Spaces The previous discussion was veeeeery theoretical in the following ways:
- what is a good feature map ?
- how should we choose the Hilbert space?
- what is the resulting function space H?
The kernel trick makes it possible to work directly with a kernel function without explicitly using , , and the feature map.
Definition: KernelLet be a non-empty set. A function is called a kernel on if there exists a Hilbert space and a map such that:
Intuition: a kernel is a function that takes two points and returns a number. That number is an inner product taken in some other space.
In particular, the raw inputs from the set can be anything (vectors, text, graphs, etc.); there's a feature map that maps each input into a Hilbert space where an inner product is defined.
Note that "kernel" terminology is totally unrelated to the kernel in linear algebra!
Lemma (necessary and sufficient condition to be a kernel)A function is a kernel if and only if it is symmetric and positive semidefinite, i.e.
We now provide some examples of kernel functions on R^d:
- linear kernel
- polynomial kernel for all and
- Gaussian kernel for all
- compactly supported radial basis kernel for
In general, kernels of the form for are called radial basis function kernels. Moreover, a scaling factor such that is called the bandwidth.
Reproducing Kernel Hilbert Spaces
Let be a Hilbert space of functions . Then is a reproducing kernel Hilbert space (RKHS) if , given by , is continuous for all .
Furthermore, a function is a reproducing kernel of if:
- it is a member of the space: for all
- it satisfies the reproducing property: for all and
If this function exists, we call , given by , the canonical feature map.
_Intuition: an RKHS is a space of functions where evaluating a function at a point is a well behaved operation, i.e. for each fixed x, f(x) cannot change while the function norms stays norm.
As a consequence, we have the reproducibility property: we can recover a function value at x by taking an inner product with a special function associated with x.
So, an RKHS is a function space with a norm and inner product that make pointwise evaluation possible through a kernel.
Theorem (Reproducing Kernel Hilbert Spaces)- If has a reproducing kernel , then is an RKHS
- every RKHS has a unique reproducing kernel
- for every kernel , there exists a unique RKHS having as its reproducing kernel.
RKHS Learning: the Kernel Trick
Let be a non-empty set. For given data , we want to solve the learning problem over an RKHS :
Representer Theorem
Let be a continuous function in its first argument, and let be the RKHS of a kernel . Then there is a minimizer of .
This minimizer is of the form, for some :
Moreover, if is convex in its first argument, is unique.
Intuition: we can search over every function in a possibly infinite-dimensional space , but the optimal function can be written only using the training examples as defined above. Each function is a kernel centered at a training input, and the coefficients are the weights. Therefore, we can solve the learning problem using just kernel evaluations between data points.
Furthermore, we also have the following theorem:
Representer Theorem with InterceptLet be a continuous function in its first argument and the RKHS of a kernel . Then, if has a solution, it is of the form where:
Corollary (Kernel Ridge Regression)
Let be the RKHS of a kernel . The kernel ridge regression problem
has a unique solution given by
where is given by , is the identity matrix, and .
Corollary (Kernel Ridge Regression with Intercept)Let be the RKHS of a kernel . The kernel ridge regression problem with an intercept
has a unique solution , where
Numerical Methods
We start by noticing that the square loss admits closed-form solutions, so common implementations include the scikit-learn package in Python (sklearn.kernel_ridge.KernelRidge). This is computationally efficient, but memory requirements are quadratic in the dataset, and usually the coefficients are nonzero (no sparsity).
from sklearn.kernel_ridge import KernelRidge
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
# A small nonlinear dataset
rng = np.random.default_rng(42)
X = np.linspace(-3, 3, 120).reshape(-1, 1)
y = np.sin(X[:, 0]) + rng.normal(0, 0.1, size=len(X))
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42
)
model = KernelRidge(kernel="rbf", alpha=0.1, gamma=0.5)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
For binary classification with the hinge loss, , the Sequential Minimal Optimization (SMO) algorithm is usually used. SMO is a form of coordinate descent, where, at each step, two indices are chosen and we optimize only over . Generally speaking, SMO belongs to a class of algorithms that are based on solving simpler sub-problems, called decomposition methods, chunking, or subset selection. Notice that, for large datasets, many may simply be equal to , so we have a sparse solution. For implementing SMO, again the scikit-learn package (sklearn.svm.SVC) is very common.
from sklearn.datasets import make_moons
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_moons(n_samples=500, noise=0.25, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y
)
model = make_pipeline(
StandardScaler(),
SVC(kernel="rbf", C=1.0, gamma="scale"),
)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
svc = model[-1]
For the -insensitive loss , we can apply a variant of the SMO algorithm to general regression problems using the same scikit-learn package. In this case, the model treats outliers more robustly and produces a sparse solution.
However, unlike squared-error kernel ridge regression, -SVR doesn’t have a simple linear solution: its loss function has corners and is not differentiable at the band’s edges, so it uses an optimization algorithm. The problem is still convex and tractable: non-differentiability just means there’s no ordinary smooth-loss formula to solve it directly.
import numpy as np
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVR
rng = np.random.default_rng(42)
X = np.linspace(-3, 3, 160).reshape(-1, 1)
y = np.sin(1.5 * X[:, 0]) + rng.normal(0, 0.08, size=len(X))
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42
)
# Simulate one unusually noisy training observation.
y_train = y_train.copy()
y_train[0] += 2.0
model = make_pipeline(
StandardScaler(),
SVR(kernel="rbf", C=1.0, epsilon=0.15, gamma="scale"),
)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
SMO Algorithm for Binary Classification
For the empirical RKHS problem with intercept is:
s.t. .
The Lagrangian is:
if we apply the First Order Conditions with respect to a, and b, we end up with the dual formulation (quadratic program with box constraint):
If we go back to the optimization problem, we notice that the objective wants to keep the boundary simple by penalizing more complex functions () and penalizing margin violations (). In this context, the Lagrangian attaches a to each margin constraint. In the dual form, the optimization works with the weights and the pairwise kernel similarities () instead of directly with the optimizing function.
SMO Algorithm from Platt, 1998Note that the following algorithm comes from the paper Sequential Minimal Optimization: A Fast Algorithm for Training Support Vector Machines, written by Platt (Google) in 1998. The idea is that, instead of optimizing for all coefficients , the algorithm can repeatedly:
- selects two coefficients
- optimizes only over those coefficients while keeping the constraints valid
- updates the model and repeats until convergence
This comes from the fact that the dual SVM problem has many variables, one for each training point, so solving all of them at once can be very expensive.
Technically speaking:
- we initialize
- we iterate until the convergence criterion is satisfied (for example, if the KKT conditions are approximately satisfied):
- choose indices with according to a given rule or heuristic
- set for
- choose to maximize the dual objective over the feasible values of this pair, keeping all other coefficients fixed
- update according to the first-order conditions
- the output is the optimal function , where the term follows from the first-order conditions.
Numerical Approaches: Large-Scale Methods for Square Loss
Again, we recall that analytical solutions for the square loss are not feasible for large data sets. So we propose some approaches to fix this limitation.
We start with the feature selection / random feature method, whose basic idea is to select a subset , with , where the selected points are called landmarks (inducing points), and derive an estimator of the form:
Intuition: many training points can be redundant, and a carefully chosen subset can represent most of the important structure in the full kernel matrix.
Note that methods differ in how they choose the set . For example, using the Nyström method, we choose a subset of rows and columns from the full kernel matrix by setting:
- : selected landmark indices
- : similarities between all points and the landmarks
- : similarities among the landmarks
so we can approximate the full matrix as
where is the pseudoinverse of .
Secondly, we can opt for preconditioning methods. A popular method is called FALKON (see Rudi et al.). To illustrate preconditioning, let and consider an ideal preconditioner and a vector such that:
Then solves , the linear system for the closed-form solution of the above corollary.
However, note that constructing this ideal can be costly. FALKON instead uses a Nyström approximation of the problem and an efficiently computed preconditioner for the resulting system.
Function approximation with RKHS Definition: universal kernel
Let be a kernel on a topological space . We say that its RKHS (or, equivalently, ) is universal if is dense in the set of all continuous functions with respect to the sup norm, i.e. for every continuous function and , there exists such that:
Intuition: a universal kernel is a kernel whose RKHS is expressive enough to approximate any continuous function on the input space , where
is the largest approximation error and is a function produced by the kernel model.
To better understand it, consider to be an arbitrary smooth curve and to be the hidden prediction function learned using the kernel. If the kernel is universal, given enough suitable kernel functions with appropriate coefficients, we can make uniformly close to . In this case, the RKHS may contain infinitely many functions, but it does not necessarily contain . (You can think of this as analogous to polynomial approximation: polynomials may not be equal to every continuous function, but on a compact interval they can approximate every continuous function.)
We now consider to be a compact subset of ; the following examples are universal kernels:
- exponential kernel: for
- Gaussian kernel: for
- binomial kernel: for
Function approximation with RKHS: Sample efficiency
Consider a Gaussian kernel (other smooth universal kernels would also work) and .
Theorem: Worst-Case Learning Rate for Kernel RegressionLet be "sufficiently smooth" and the optimal regression function be -times continuously differentiable.
Let be i.i.d. such that .
Let be the optimal kernel regression function for data and .
Then there exists such that:
Intuition: the error is roughly of order , so the error decreases polynomially with the amount of data and the exponent determines how quickly this happens. Furthermore, the parameter describes the regularity of the true regression function (small : the function can vary sharply; large : the function is smoother and easier to approximate), so the rate gets faster as increases. On the other hand, the rate gets slower as increases (curse of dimensionality).
Theorem: Minimax Lower Rates of ConvergenceLet be a sufficiently rich family of smooth measures.
Then, there exists such that:
where is the optimal least-squares regression function for .
Intuition: the previous theorem says kernel regression can achieve the specified rate; the minimax theorem says that no method can generally do better over the whole problem class. Therefore, the rate is called minimax-optimal.
...To sum up, we have seen that:
- kernel functions are analytically and numerically tractable
- under suitable smoothness and distribution assumptions, kernel regression achieves the best possible worst-case learning rate, up to constants.
However, we've also seen some problems:
- for kernel methods, the number of parameters () grows with the number of data points
- the kernel must be chosen from the start and affects solution quality
- even though learning rates of kernel methods are optimal in general, in some specific situations other learning methods can adapt to the structure of the data better than kernel methods.
6 Neural Networks
Even if now (as of October 2026) we only speak about frontier-AI models driven by powerful neural networks, they are actually not new: the first models date back to the 1940s and backpropagation to the 1980s. More specifically:
- in 1940–1960: first mathematical models of biological neurons (McCulloch and Pitts, 1943, Hebb, 1949)
- in 1980–1990: connectionism with the first backpropagation algorithm, (Rumelhart et al., 1986) and the long short-term memory (LSTM) architecture (Hochreiter and Schmidhuber, 1997)
- from 2006 to now: the deep learning revolution, with efficient training of deep belief networks (Hinton et al., 2006) and efficient training of deep neural networks (Bengio et al., 2006 and Ranzato et al., 2006)
However, things started getting serious due to increased computational power (which means larger networks can be trained) and increased data availability (which means networks can learn well).

We now spend some time diving deep into the different categories of neural networks.

We consider an input layer and a first hidden layer, defined as:
The second hidden layer is defined as:
The output layer is defined as:
Once we've done so, we can formally define a feedforward neural network as a "system" consisting of:
- weight matrices
- biases
- activation function
The realization of an FNN is the function given by:
Here, the affine transformations are defined by .
Furthermore, we have that:
- is the number of neurons in the -th layer, and is the network's architecture
- are the network parameters
- the parameter space is , where
Remarks: If the network outputs need to be in , , or , we can add a final transformation on top of the last layer. If we do not specify one, we can assume .
Another remark: a small recap of the terminology:
- is the input layer, is the output layer, and the others are the hidden layers
- is the number of layers, and is the depth
- the network is said to be shallow if (only one hidden layer)
- the network is said to be deep if
- is the width of the network
Intuition: we can think of the network as a sequence of layers that transform the input: in the first layer, each neuron combines the inputs using weights, then adds a bias. Weights control how strongly each input matters; the bias shifts the result. Then, via the activation function, we apply to each neuron's result; this nonlinearity lets the network "learn" more than a simple linear relationship.
For the next layers, we "simply" repeat the same process, using the previous layer's activations as inputs.
Furthermore, we denote by (a vector space) the family of all FNNs with activation function and architecture .
For example, we can consider as the class of all shallow FNNs with -dimensional input and one-dimensional output. In this case, they can be represented by functions of the form:

Feedforward Neural Networks in Finance
We now provide some examples of neural network applications in finance:
- Option pricing: learn the pricing function for simulated or market option prices with a neural network, as shown in the 1994 paper by Hutchinson et al., A Nonparametric Approach to Pricing and Hedging Derivative Securities Via Learning Networks.
- Empirical asset pricing: forecast excess returns from firm characteristics (size, value, momentum, etc.), as shown in the 2020 paper by Gu et al., Empirical Asset Pricing via Machine Learning. (In this case, we curiously see that networks with few hidden layers outperform linear models out of sample.)
- In particular, past research focused on the "factor zoo" to determine returns, with scarce results, so Gu and Kelly treated this problem as a traditional prediction problem.
They targeted 30,000 US stocks from 1957 to 2016, with roughly 94 features, and kept the training, validation, and testing data in chronological order to avoid look-ahead bias. Then, in their paper, they compared OLS, penalized linear models (elastic net), PCR and PLS, generalized linear models with splines, random forests, and neural networks with 1 to 5 hidden layers, evaluating them on out-of-sample . - Their biggest finding is that machine learning beats linear models out of sample and, in particular, that shallow networks win, with performance peaking at about 3 hidden layers and then declining (deeper networks overfit because the signal-to-noise ratio in returns is very low). The overperformance of neural networks comes from capturing nonlinearities and cross-interactions between features, which simple linear regression cannot represent. Furthermore, they discovered that across all methods, the most important variables are momentum, liquidity, volatility, and size... many other variables counted little!
- In particular, past research focused on the "factor zoo" to determine returns, with scarce results, so Gu and Kelly treated this problem as a traditional prediction problem.
- Deep hedging: estimate the hedge ratio with a network, as in the 2019 Buehler paper, Deep Hedging.
Universal Approximation Property
The first proof of the UAP is credited to Cybenko in his 1989 paper. Four years later, Leshno et al. gave a necessary and sufficient condition on the activation function for the UAP to hold for shallow neural networks.
In particular, we can state the UAP for shallow FNNs as follows.
Theorem 1 (UAP, Leshno et al., 1993)Let be compact, and let be a continuous activation function. The set is dense in with respect to the supremum norm if and only if is not a polynomial.
_Note that the UAP also holds for FNNs with a fixed number of hidden layers (if the width is allowed to go to ). Furthermore, the UAP also holds for FNNs with a fixed width (if the depth is allowed to go to ).
Note also that in many applications FNNs provide a more efficient approximation than shallow FNNs.
Intuition: consider neurons as the building blocks, so each neuron is a "ridge function" (i.e., a step in one direction). Choosing and moves and orients the step, and scales it.
If we sum many neurons, we can build any shape (in 1D, this is analogous to approximating a function with steps).
However, if is a polynomial of degree at most , the space of functions we can approximate is finite-dimensional. This implies that we need some non-polynomial (nonlinearity) to approximate complex shapes.
We recall the empirical loss minimization problem: we want to solve:
for a training set of i.i.d. realizations of , a loss function , and a hypothesis class .
If we set , we have:
We can define .
We can now say that this optimization problem is highly nontrivial:
- the parameter space can be very high-dimensional
- if is almost everywhere differentiable, we can use (stochastic) gradient descent with
- However, is typically non-convex; therefore, convergence to a global minimum is not guaranteed
- Furthermore, computing is efficient even in large networks using backpropagation
Also, note that during training, each network layer learns to produce features that are useful for the next layers to minimize the loss.
Gradients and the Chain RuleWe've seen that a neural network is a computational graph: the forward pass computes , , , and the loss one layer after the other (in the figure: black arrows). The backward pass computes the derivatives of the loss in the opposite order (in the image: red arrows).

The chain rule then says that from one layer to the next one, the corresponding Jacobian matrix, with entries , is:
Therefore, the Jacobian over several layers is the product of these matrices:
Back propagationThe main problem of training neural networks with gradient descent is that we have a loss function that depends on many parameters (the weights and biases in every layer), so we would need for all of them, which would be too expensive. To find an efficient solution, we first represent the network as a chain of functions and the loss as a nested composition.
For a data point , let , , and for .
We define the gradient , so we have:
- for the output layer:
- for the hidden layers:
That can be nicely rewritten as:
- for the parameters: since , we have:
i.e.
Notice that one forward pass (storing all and ) and one backward pass give all partial derivatives.
Intuition: represents how much the loss cares about each neuron in layer , or the error signal (blame) associated with each neuron:
- for the output layer: the blame is easy to compute because is just the gradient of the loss with respect to the prediction;
- for the hidden layers: a neuron in layer affects the loss only through the following neurons in layer , so the blame is a sum over those neurons.
So, with the backward pass, we can compute in sequence, reusing each one for the next layer. Each also gives that layer's parameter gradients immediately.
Gradient Descent and Mini-Batch Training
We recall that with gradient descent (GD),
every step requires a pass over all data points, which can be too expensive for a large number of data points.
(Also recall that is the loss averaged over the whole dataset of points: .)
We can now consider mini-batch SGD, which uses the average gradient over a random mini-batch of points. Let with :
We clearly see that the mini-batch gradient is a noisy but unbiased estimate of the full gradient, i.e. the noise level (the variance of the estimate) scales like . A bigger batch gives a more accurate direction but costs more.
Note also that (batch size) should be treated as a hyperparameter and tuned accordingly.
Therefore, the mini-batch stochastic gradient descent algorithm can be simplified as follows:
t = 0
for epoch in range(E):
perm = random_permutation(m)
for start in range(0, m, n_batch):
idx = perm[start:start + n_batch]
loss = mean(loss_fn(model(X[idx], theta), y[idx]))
g = backprop(loss, theta) # ∇_θ loss
theta = theta - eta[t] * g
t += 1
More about the libraries:
- PyTorch: Automatic differentiation package (torch.autograd)
- PyTorch Tutorial: Automatic Differentiation with torch.autograd
- TensorFlow Guide: Introduction to gradients and automatic differentiation
- TensorFlow Guide: Introduction to graphs and tf.function
The Learning Rate
We first recall that the Hessian is the matrix of second-order partial derivatives of a scalar function. For a loss , its entries are:
It describes how the loss curves in different parameter directions: near a minimum, it helps approximate the loss and understand how quickly optimization converges.
The learning rate (step size) is usually the most important hyperparameter.
If we consider a quadratic model, we clearly see that near a minimum we have that:
with Hessian with eigenvalues .
In this case, the gradient descent satisfies:
which implies
so gradient descent converges if and only if .
Furthermore, we can see that the best constant step size (i.e. the best learning rate) is , which gives the rate (i.e. the factor by which we reduce the error per step) equal to
where is the condition number.

However, notice that with mini-batch gradients and a constant , SGD eventually stops converging: the iterates keep fluctuating around in a region that grows with and with the gradient noise.
Furthermore, the Robbins-Monro conditions give that
SGD converges (i.e. for strongly convex objectives with bounded gradient noise, we have convergence).
In practice, below are some learning rates often used in practice:
- step decay: (i.e. every steps)
- cosine annealing: ,
- learning rate reduction on plateaus: halve when the validation loss has not improved for some epochs
- warm-up learning rate: increase linearly during the first iterations
- rule-of-thumb learning rate: choose on a logarithmic grid (e.g. ) and take the largest value that does not diverge, then reduce it.
Hyperparameters and Model Selection
Hyperparameters (such as architecture (depth , widths , activation ), learning rate, batch size, number of epochs, and regularization (, dropout rate ), etc.) are not learned in SGD.
Instead, they are usually chosen by minimizing the error on a validation set, distinct from the training and test sets.
For financial data, we usually do NOT shuffle across time to avoid look-ahead bias: instead, we usually use a walk-forward split (via an expanding or rolling window) and use the test set only once, at the very end.

Optimizers: Momentum and RMSProp
Consider , the mini-batch gradient at , and note that squares, square roots, and divisions of vectors are taken component-wise. We now spend some time mentioning some optimizer techniques.
Momentum (heavy ball optimizer), from Polyak, 1964The idea is to accumulate an exponentially weighted velocity, given , typically :
In the case , we have that
is an exponentially weighted sum of past gradients that is able to average out part of the mini-batch noise.
In this case, consistent gradients add up: if for all , then
so the effective step size is , which is when .
Furthermore, gradients that change sign cancel out: if , then
so the step size reduces to , which is approximately when .
RMSProp (Root Mean Square Propagation)For this optimizer, we consider a specific per-coordinate step size from a running average of squared gradients. Formally, if we set and , we can write:
In this case, coordinates with large (small) gradients get small (large) steps; the update is (almost) invariant to rescaling individual parameters, which helps with badly scaled problems.
Adam Optimizer, from Kingma and Ba, 2015The idea behind this is that it combines momentum, RMSProp, and bias correction in a single optimizer: we consider a momentum step divided in each coordinate by the typical size of the gradients in that coordinate, . To understand it better, we start from the algorithm in pseudocode:
eta = 1e-3
beta1 = 0.9
beta2 = 0.999
epsilon = 1e-8
theta = theta0
m = 0
s = 0
for t in range(1, num_iterations + 1):
# Mini-batch gradient at theta
g = backpropagation_gradient(theta)
m = beta1 * m + (1 - beta1) * g
s = beta2 * s + (1 - beta2) * (g ** 2)
m_hat = m / (1 - beta1**t)
s_hat = s / (1 - beta2**t)
theta = theta - eta * m_hat / (s_hat**0.5 + epsilon)
Formally, we can see this and the bias correction by unrolling the recursion and starting with :
The weights sum to
So, is a weighted sum of whose weights sum to less than one.
If for all , then . For example, ; dividing by gives .
Note that the same holds for with the factor , which stays far below 1 for much longer ().
To understand the step size, note that, in each coordinate, estimates and estimates . By Jensen's inequality, , so the Adam step is at most about in magnitude in each coordinate, and is smaller when gradients are noisy (i.e. change sign).
Multiplying the loss by multiplies and by and by , so the Adam step does not change (up to ), whereas the SGD step is multiplied by .
Therefore, is roughly the largest change of each parameter per step, so the same (e.g. ) works for very different problems; this is why Adam is the usual default.

As we can see from the picture above, on an ill-conditioned quadratic (with ), GD with its optimal constant step size zigzags across the valley and converges slowly, while momentum and Adam optimizers converge much faster.
On the other hand, on quadratics, momentum with specified parameters improves the rate (error reduction factor per step) of GD from to .
However, note that no optimizer is best for every problem: SGD with momentum is often used for CNNs, while Adam or a variant of Adam (AdamW) is used for transformers (and, again, recall that the learning rate must be tuned in any case).
Practical Training Issues: Standardization
Often, standardizing each input feature using the mean and standard deviation over the training set improves performance:
Apply the same to validation and test data (alternatively, use min-max scaling to ).
This works because, for linear least squares, the Hessian is . Features on very different scales make large and gradient descent extremely slow. For example, moneyness may appear alongside days to maturity in option datasets.
Least squares on (moneyness, days to maturity), with 50 gradient descent steps and :
- Raw inputs: gradient descent quickly reaches the valley floor (the steep direction is approximately , the weight of days to maturity), then barely moves toward the minimum .
- Standardized inputs: , and gradient descent reaches the minimum .

Targets: scale them too, so that they are of order 1, like the output of the network at initialization. Otherwise, the loss and its gradient (e.g. for the squared loss) are dominated by the largest targets, and a learning rate that works for one scale fails for another.
Example: option prices. Use instead of . If the price is homogeneous of degree one in , i.e. for all (e.g. in the Black-Scholes model), then setting gives
So the target is of order 1, and the network needs one fewer input (moneyness instead of and ) (Hutchinson et al., 1994).
Time series: compute and using past data only (e.g. on rolling windows). Statistics computed on the full sample contain information from the test period.
Heavy tails: returns, volumes, and spreads have heavy tails. A few extreme values inflate and dominate the loss, so winsorize (clip at, e.g., the 1% and 99% quantiles of the training set) or use log transforms.
Initialization
In general, the critical point reached by SGD depends on the initialization , because the loss function is not convex in .
If we initialize all parameters to a common constant , i.e. set , then all neurons in a layer compute the same output (e.g. for all ) and receive the same gradients (in backpropagation, is the same for all ). Therefore, they stay identical during training, and the layer effectively has a single neuron. We must initialize the neurons differently, e.g. randomly.
We can use random initialization, but if is too small:
- gradients vanish (in the chain rule, we multiply weights that are on average smaller than one)
If is too big:
- gradients explode (in the chain rule, we multiply weights that are on average larger than one).
An initializer that works well is the Xavier (Glorot) initialization [Glorot and Bengio, 2010]:
where and are the widths of layers and .
Why? With independent centered weights and , has variance
If near 0 (e.g. ), then . Thus, the variance neither vanishes nor explodes across layers if . The same argument for the backward pass (a sum over the neurons of layer ) gives ; Xavier takes a compromise between the two.
ReLU sets about half of the pre-activations to 0, so . He initialization, [He et al., 2015], compensates with the factor 2.
Other initializers can also be used (see, e.g., torch.nn.init and tf.keras.initializers). Typically, one repeats the training for different sampled values of .
Batch Normalization
Problem: during training, parameter updates in one hidden layer often shift or rescale the input of the next hidden layer, which slows down training (internal covariate shift).
Main idea: stabilize training by normalizing the activations of each hidden layer using the empirical mean and standard deviation over each batch.
If is a batch of size and is the activation of the -th neuron in the -th layer, then:
where
and is a small constant () to prevent division by zero.
Note that normalized activations are usually then rescaled, , with learnable parameters and , so the layer can undo the normalization if useful. At test time, and are replaced by averages from training.
Practically speaking, in PyTorch and TensorFlow, batch normalization is implemented as a layer (see torch.nn.BatchNorm2d and tf.keras.layers.BatchNormalization).
Regularization
Loss functions are typically regularized, i.e., we prefer to minimize a modified objective constructed from the hyperparameter :
Popular choices of are:
- , which shrinks in directions that do not minimize the objective enough
- , which encourages a sparse
Note, of course, that we only regularize the weights and do not touch the biases in order to avoid underfitting, and that different regularization parameters can be applied to different layers.
In particular, we can now consider weight decay, which shrinks parameters toward 0 at every step by setting:
Note that, for stochastic gradient descent, this translates to regularization: the gradient of is , so .
Instead, for adaptive methods, weight decay and regularization are not equivalent: with Adam, for example, the term is divided by , implying that parameters with large gradients are regularized less.
Secondly, note that training techniques can be regularizers too. For example, consider:
- early stopping: for a fixed (patience parameter), we stop training if, after steps, the error on a validation set has not decreased, and we go back to the parameters with the minimum validation error. (Note that this can equivalently be seen as choosing the number of SGD steps based on a validation set.)
- dropout: before each gradient step, we set each neuron to zero with probability (independently of all others), then perform backpropagation as usual. This encourages sparse and redundant representations (this can be viewed equivalently as training exponentially many neural networks).
To see this in practice, we now present a dropout method, proposed by Srivastava et al. in 2014, in the inverted form: during training, we multiply the activations of each hidden layer by a random mask, _that is:_1
so that the expected activation is unchanged and no dropout is applied at test time. (Note that is element-wise (Hadamard) multiplication.)
Choosing Evaluation MetricsWe first recall that the training loss is chosen for optimization such that it is cheap to compute and differentiable. Choosing the evaluation metric is another story and should reflect how the model will be used and, more importantly, it is computed on the test set. We now present some evaluation metrics.
-
For regression, we can often use:
- MSE:
- RMSE:
- MAE:
- (for return forecasts, where is the benchmark forecast)
-
For classification, we prefer methods that look for accuracy and precision (F1, ROC and AUC; click here to review them). For predicted probabilities, we can use the log-loss: , or the Brier score: .
-
For finance-specific topics, we consider pricing errors in implied-volatility points rather than in prices, hedging errors and, for training signals or strategies, we can consider P&L, Sharpe Ratio, turnover and drawdowns of the optimized strategy, better after transaction costs.
Lastly, always compare with simple benchmarks (historical mean, linear regression, GARCH, Black–Scholes, etc.) and evaluate over several periods and market regimes.
Little off-topic on implementation: tensors and automatic differentiation in PyTorchWe recall that a tensor is an n-dimensional array on a CPU/GPU.
Operations on tensors with requires_grad=True are recorded in a computational graph; backward() then runs reverse-mode automatic differentiation (backpropagation):
We can consider the following example, where we try to compute a portfolio's variance and then estimate variance changes depending on weights using backward().
backward() starts from the variance, and PyTorch walks the graph backward and applies the chain rule at each node. This is reverse-mode automatic differentiation, and it is the same algorithm as backpropagation in neural networks. The result is exact. It is not a numerical approximation like .
import torch
Sigma = torch.tensor([[0.04, 0.006],
[0.006, 0.09]]) # covariance matrix of two assets
w = torch.tensor([0.6, 0.4], requires_grad=True) # portfolio weights
variance = w @ Sigma @ w
variance.backward()
print(variance.item(), w.grad)
# portfolio variance w' Sigma w
# reverse-mode automatic differentiation
# 0.03168 tensor([0.0528, 0.0792])
Note that automatic differentiation is exact up to floating-point errors (no finite-difference error).
7 Deep Learning
Convolutional Neural Networks (CNNs)In this chapter we focus on a type of machine learning that uses artificial neural networks with many layers.
Recall that a neural network learns by receiving input data, producing a prediction, comparing the prediction with the correct answer, and adjusting its internal parameters to reduce the error.
In this way, the layers gradually learn increasingly complex patterns.
Similarly, a convolutional neural network (CNN) is a type of deep-learning model designed especially for images and other data composed on a grid.

Indeed, instead of analyzing each data point (pixel) independently, a CNN uses a "filter" that scans across the image to detect patterns (edges, corners, textures, etc.):
In particular, a CNN consists of a layer with three stages (on the left) or a composition of three different layers (on the right). In either case, we have three main components: the convolutional layer, the detector, and the pooling layer.
The Convolutional LayerThe convolutional layer implements a matrix operation known as cross-correlation.
Indeed, let and be matrices such that and .
Then their cross-correlation is a matrix, denoted by , whose entries are given by
These are exactly the for which the window stays inside , so
Furthermore, is called the input and is called the kernel (or filter).
Note that in machine learning, cross-correlation is related to matrix convolution, so the terms are often used as synonyms.
Intuition: is an image represented by a matrix, and is a smaller matrix acting as a filter. The kernel slides over the image and, at each position, multiplies the numbers element by element and then adds the results into a sum in the output.
For example, suppose the following:
We then place the kernel on the top-left submatrix and multiply element by element, so we have:
Then, we move the kernel across every position where it fits (positions , , , and ), so we have:

As a further example, we can consider the following convolution:

In this case, the Sobel Y kernel takes the difference between the row above and the row below each pixel, so it is able to approximate the vertical derivative and measure vertical intensity changes.
The output is large at horizontal edges (back, belly, tail) and has the same shape as the
input.
In machine learning, kernels are usually square matrices , and we usually treat as a hyperparameter.
Therefore, if has shape and has shape , then the output of the cross-correlation has shape .
However, to keep the same shape as the input (as in the example of the dog before), we can pad the input with layers of zeros.
For example, for and , we have
So the padded input has shape , and the output has shape
This means that, if with odd, we can keep the input size unchanged.
Also in this case, we often treat as a hyperparameter.
Additionally, you can compute convolution with strides of length . The idea is that specifies how many cells the kernel moves after each computation.

For stride , the output has shape
so the window must stay inside the input and leftover rows and columns are dropped.
As usual, we treat the stride as a hyperparameter.
However, note that normally convolutional layers consist of more than one kernel ( kernels), each with its own set of parameters.
Therefore, if the input has shape and we have kernels with the same shape , padding , and stride , then the final output will have shape:
The Detector Layer
The detector layer applies a nonlinear activation function componentwise, so the output of this layer has the same shape as the input, and the detector layer has no trainable parameters.
The Pooling Layer
The pooling layer slides over a window of shape , with stride and padding , over the input and summarizes each window by one number.
We can consider two common types of pooling: max pooling and average pooling.

The idea behind pooling is that we can downsample: for example, with , we can halve and , so there are four times fewer values and therefore less computation.
Furthermore, note that pooling is approximately invariant to small shifts, since the maximum of a window often stays the same when a feature moves within the window. What matters is whether a feature is present, not its exact position.
Therefore, the output of the pooling layer is approximately invariant under small translations (this is particularly useful for image-recognition tasks).
Indeed, this means that if an object moves slightly within an image, the pooling output often remains similar.

Example: Architecture LeNet-5
LeNet-5 is one of the earliest convolutional networks, from the paper by LeCun et al. in 1998.
It was trained on the MNIST dataset, which contains 70,000 pictures of handwritten digits.

The model is able to classify input images and understand which digit they represent.

CNNs and Data Priors
8 Bagging and Random Forests
9 Gradient Boosted Trees
10 Dimensionality Reduction and Autoencoders
11 Generative Models
Projects
All the projects were assigned during the course and self-made in groups. The code is fully available and reproducible on GitHub here.
Below you find a brief description and comment on the project, for the technical implementation please refer to the codebase.
Credits to Giacomo Maggiore, Ugo Buzzacchino, Kostas Zisis, Konstantin Krstevski.
P1. Linear Regression with housing price data
P.1.1 Data preparationThe project is built around a dataset of house price data, which consists of a dependent target variable Y, "SalePrice", and 79 features, 35 of which are numerical and 44 categorical.
The first step was determining whether the target variable followed a Gaussian distribution. To do that, we plotted a frequency histogram, a Q-Q plot, and calculated moments, resulting in:
Skewness: 1.883
Excess kurtosis: 6.536
Mean: $180,921
Median: $163,000

It can be immediately observed that:
- Skewness > 0 and mean > median the distribution has a long tail to the right (right-skewed)
- Excess kurtosis > 0: there are more extreme values than in a normal distribution
- In the Q-Q plot (which represents theoretical vs. sample quantiles): the data points form a curve, not a straight line
Therefore, we standardized the data and applied a log transformation to reduce skewness and heavy tails, resulting in the distribution that can be seen in the following plots:

Secondly, we split the dataset into training and testing sets. We then filled missing numerical values with the sample mean of the remaining series and categorical data with the most frequent value.
P1.3 Linear Regression on Numerical Features
We first fit an OLS regression using only numerical predictors with the scikit-learn library and then saved the coefficients by running:
# The numerical predictors were imputed and standardized using training-set statistics in 1.d.
ols_numerical = LinearRegression(fit_intercept=True)
ols_numerical.fit(X_train_num, y_train)
ols_numerical_coefficients = pd.DataFrame({
"feature": ["Intercept", *NUMERICAL],
"coefficient": [ols_numerical.intercept_, *ols_numerical.coef_],
})
Recall that in this first run the target variable is log(SalePrice) and the features were standardized.
Intuition: given a coefficient b, the price is about 100*b higher when the corresponding feature increases by one standard deviation, ceteris paribus.
We then calculated MSE and for both training and testing data, on both log(SalePrice and SalePrice scales, resulting in the following:
| Sample | Target scale | MSE | R² |
|---|---|---|---|
| In-sample | log1p(SalePrice) | 0.02200 | 0.85810 |
| Out-of-sample | log1p(SalePrice) | 0.02187 | 0.87109 |
| In-sample | SalePrice (USD) | 1,433,790,871 | 0.76177 |
| Out-of-sample | SalePrice (USD) | 829,116,180 | 0.88118 |
We immediately notice that, on the log scale, the MSE for training and testing is almost equal, so the model DOES NOT overfit.
On the other hand, on the dollar scale, the training /MSE are lower/higher compared to the testing set, but the reason comes from two extreme-value training houses (outliers).
Generally speaking, we can say that the log metrics show typical relative accuracy and the dollar-scale performance may be influenced by a few large errors (either in the training or in the testing set, depending on luck!)
We then estimate the coefficients "manually", using NumPy and a numerically stable linear system (not relying explicitly on the inverse matrix to solve the normal equation).
We first define the design matrix and y vector as:
A = np.column_stack([np.ones(len(X_train_num)), X_train_num.to_numpy()])
y_vec = y_train.to_numpy()
Secondly, we get the number of observations and coefficients, then compute for the normal equation:
# Get the number of observations and coefficients.
m, p = A.shape # p = number of features + 1 intercept
# Compute AᵀA for the normal equation.
AtA = A.T @ A
We can now solve the normal equation and compare the coefficients with those from the previous scikit-learn model:
# Solve the normal equation for the regression coefficients.
beta_solve = np.linalg.solve(AtA, A.T @ y_vec)
# Extract the intercept and coefficients from the scikit-learn model.
beta_sklearn = np.r_[ols_numerical.intercept_, ols_numerical.coef_]
# Compare the coefficients obtained by both methods.
q2b_coefficients = pd.DataFrame({
"feature": feature_names,
"beta_numpy_solve": beta_solve,
"beta_sklearn": beta_sklearn,
"abs_difference": np.abs(beta_solve - beta_sklearn),
})
By plotting the result, we could see that most of the coefficients are approximately equal in the two models, but 8 are very different.
This happens because these columns are linearly dependent; therefore, is a singular matrix and the normal equation has infinitely many solutions. However, the "manual" method using the NumPy solver does not return an error because rounding errors make invertible, and it then returns an arbitrary solution (dependent on rounding errors).
In contrast, scikit-learn uses an OLS solver that returns the unique solution with the smallest norm.
Therefore, we conclude that the system is NOT numerically well-behaved and we should not consider the coefficients from the 8 dependent columns.
For the same reason, when looking at the standard errors (SE), we should only consider the finite and small SEs and ignore the huge or NaN SEs, as they come from the (redundant) linearly dependent variables.
Given the results, we spent some time investigating the numerical structure of A.
rank_A = np.linalg.matrix_rank(A)
_, singular_values, Vt = np.linalg.svd(A, full_matrices=False)
P.1.2 Regularization Techniques
We now try to implement and tune four different methods:
- Truncated Pseudoinverse
- Ridge
- Lasso
- Elastic Net
For each method, we use an 8-fold CV, MSE as the model-selection criterion, and the same CV folds.
Firstly, we build one reusable preprocessing component that applies numerical imputation (i.e. fills missing data), numerical standardization, categorical imputation, one-hot encoding, and correct treatment of NA/None where the values are allowed categories.
preprocessor = ColumnTransformer([
("num", Pipeline([("imputer", SimpleImputer(strategy="mean")), ("scaler", StandardScaler())]), NUMERICAL),
("cat", Pipeline([("imputer", SimpleImputer(strategy="most_frequent")),
("onehot", OneHotEncoder(handle_unknown="ignore"))]), CATEGORICAL),
], sparse_threshold=0)
cv = KFold(n_splits=8, shuffle=True, random_state=42)
We then construct one pipeline for each method using the SAME preprocessor.
- For Ridge, Lasso, and Elastic Net, we start from the raw data, apply the preprocessor, and then fit the model.
- For Truncated Pseudoinverse, we start from the raw data, apply the preprocessor, compute the truncated SVD, and then fit a linear regression.
(Note that all methods include the intercept and it is not regularized.)
We then set the hyperparameter grids and tune the following with GridSearch:
- regularization strength for Ridge;
- regularization strength for Lasso;
- regularization strength and mixing parameter for Elastic Net;
- number of retained singular directions for Truncated Pseudoinverse (with scikit-learn's TruncatedSVD, this is n_components: the number of retained singular components, not a singular-value cutoff).
models = {
"Truncated pseudoinverse": (
Pipeline([("preprocessor", preprocessor), ("svd", TruncatedSVD(random_state=42)), ("model", LinearRegression())]),
{"svd__n_components": list(range(10, 251, 10))},
),
"Ridge": (
Pipeline([("preprocessor", preprocessor), ("model", Ridge())]),
{"model__alpha": np.logspace(-4, 5, 37)},
),
"Lasso": (
Pipeline([("preprocessor", preprocessor), ("model", Lasso(max_iter=100_000))]),
{"model__alpha": np.logspace(-4, 0, 17)},
),
"Elastic Net": (
Pipeline([("preprocessor", preprocessor), ("model", ElasticNet(max_iter=100_000))]),
{"model__alpha": np.logspace(-4, 0, 17), "model__l1_ratio": [0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.5, 0.75, 0.95]},
),
}
To find the best parameter, we apply the following logic:
- run an initial broad grid
- once a grid is done, inspect that grid and the selected parameter
- if the parameter is at or near the border, expand that side and re-run
- stop only when the selected values are inside the final grid (we do not automatically stop at the first grid if the optimum parameter looks constrained by the chosen grid, in order to not introduce selection/prior bias)
searches, rows = {}, []
for name, (pipeline, grid) in models.items():
search = GridSearchCV(pipeline, grid, scoring="neg_mean_squared_error", cv=cv, refit=True, n_jobs=1)
search.fit(X_train_raw, y_train)
searches[name] = search
i = search.best_index_
train_mse, train_r2, test_mse, test_r2 = evaluate(search.best_estimator_, X_train_raw, X_test_raw)
rows.append([name, search.best_params_, -search.cv_results_["mean_test_score"][i], search.cv_results_["std_test_score"][i],
train_mse, train_r2, test_mse, test_r2])
# At or near the boundary = one of the two lowest or two highest grid values.
boundary = [key for key, values in grid.items() if search.best_params_[key] in [*sorted(values)[:2], *sorted(values)[-2:]]]
print(f"{name}: {'extend and rerun ' + str(boundary) if boundary else 'selected values are not at or near the edge of the final grid.'}")
After some time (if I had a MacBook Pro M5, it would have taken 15 seconds), we end up with the following results and representative plots:
| Model | Train MSE | Train R² | Test MSE | Test R² |
|---|---|---|---|---|
| OLS: numerical only | 1,433,790,871 | 0.7618 | 829,116,180 | 0.8812 |
| OLS: all features | 291,430,340 | 0.9516 | 880,885,573 | 0.8738 |
| Truncated pseudoinverse | 565,788,140 | 0.9060 | 691,717,575 | 0.9009 |
| Ridge | 601,558,822 | 0.9000 | 632,575,164 | 0.9093 |
| Lasso | 444,200,698 | 0.9262 | 483,227,109 | 0.9308 |
| Elastic Net | 676,880,511 | 0.8875 | 614,607,818 | 0.9119 |

We immediately notice that all four tuned models beat the previous OLS on the test set, obtaining a higher test and a lower RMSE.
Notice that the intercept is never penalized, since it sets the general price level, while the penalty should only limit the effect of the features (scikit-learn's Ridge, Lasso, and ElasticNet center the data and fit the intercept separately, so they never penalize it. In the truncated pseudoinverse, the LinearRegression after TruncatedSVD has no penalty at all.)
Recall also that, in the Python pipeline, pipeline parameters have the name <step name>__<parameter>.
- for example,
model__alphais the parameter alpha of the step "model", which is Lasso, Ridge, or Elastic Net svd__n_componentsis the parametern_componentsfor the step "svd" (singular value decomposition)- the names are "automatically" resolved by GridSearchCV to set the values
Also, when GridSearchCV has chosen the best hyperparameters, it fits the whole pipeline again on the entire training data, since each model is usually tested on only 7/8 of the training rows, while the refit uses all of them.
Briefly, we can also understand how each model deals with sparsity and truncation in the presence of singular values.
- We first notice that the models have different numbers of non-zero feature coefficients (excluding the intercept): out of 311 columns, Lasso has 122, Elastic Net has 111, and Ridge has 311.
(Recall that we initially have 79 features [36 numerical + 43 categorical], but after one-hot encoding, each categorical feature becomes one column per observed value [e.g. if category "town" has 4 different values 4 different columns]) - Indeed, the L1 penalty sets the coefficients of weak/redundant columns to zero depending on the strength.
- On the other hand, Ridge only shrinks coefficients, so none of them are exactly zero, but they are much smaller (lower L2 norm) compared to the pseudoinverse.
- The truncated pseudoinverse keeps all 150 singular directions; for this reason, out of the initial 311 coefficients, all are non-zero.
To sum up, when deciding on a final model for recommendation, we ended up with Elastic Net (alpha = 0.003162 and l1_ratio = 0.25) for the following reasons:
- it has the lowest mean CV RMSE (this is the same criterion used to tune and compare each model)
- with so many features (311), Elastic Net can reduce the number of non-zero coefficients and therefore reduce variance and make the model easier to interpret
P2. Credit Analytics
Before starting with the practical coding project, we give a short introduction of the topic.
In everyday language, we often associate risk uncertainty. However, risk management is more precise, since it defines risk as the exposure to the possibility of loss.
For the sake of this project, we define credit risk as the risk coming from losses from borrowers who default. Therefore, later in the code, we will model each borrower with a default probability , where is the probability the borrower repays.
We can then consider three different ways to represent risk, for a random loss :
- Variance:
- Value at Risk (VaR) at confidence level (usually ): . At 95%, this is the loss threshold exceeded in the worst 5% of outcomes.
- Expected Shortfall (ES) at confidence level : . This is the average loss in the worst fraction of outcomes.
VaR is simpler and widely used, but... ES is getting more and more popular because it looks into the tail.
There are two main common methods to assess risk:
- Monte Carlo scenarios, where we simulate many outcomes (in this way, we ask the question "who repays?" multiple times) and, for each scenario, compute the corresponding P&L so we can understand how bad and how often it happens, given that the model is right.
- Stress scenario, where we choose a specific adverse situation and compute P&L in that specific situation, without knowing how likely this bad scenario is.
To sum up... in the following project, we model each borrower with a repayment probability and, given specified lending decisions, simulate different scenarios to compute the portfolio P&L distribution and then its expected P&L and VaR.
P2.1 Synthetic Data
For this project, we use synthetic data: to do so, we simulate 30'000 borrowers with three features: age and monthly income, both uniformly distributed, and employment status, distributed as a binomial variable with .
# simulate explanatory variables x (the age and monthly income are assumed here to follow a continuous uniform distribution)
m = 20000
n = 10000
# age in [18, 80], monthly income in [1, 15] (in CHF 1000), both uniformly distributed
x_1 = np.random.uniform(18, 80, size=(n+m, 1)) # age
x_2 = np.random.uniform(1, 15, size=(n+m, 1)) # monthly income
# 0: salaried, 1 = self-employed with probability of being self-employed = 0.1
x_3 = np.random.binomial(1, 0.1, size=(n+m, 1)) # employment status
We then put vectors together and we split the data in training and testing:
# vector in R^3 of explanatory variables using only m training data points (the first m rows of x_1, x_2, and x_3)
x_ = np.hstack((x_1, x_2, x_3))
x_train = x_[:m, :]
x_test = x_[m:, :]
P2.2 Dataset and Logistic Regression
Given the input synthetic data, we now build the dataset using logistic regression with two different probability models for repayment.
Indeed, given the borrower's tendency to pay, represented by the logistic function , we set:
# logistic sigmoid function
sigmoid = lambda x: 1. / (1. + np.exp(-x))
# random variables uniformly distributed
xi = np.random.uniform(0, 1, size=x_1.shape)
# Probability that borrower i is "good" in datasets 1 and 2
p_1 = sigmoid(13.3 - 0.33 *x_1 + 3.5 * x_2 - 3 * x_3)
p_2 = sigmoid(5 - 10 * ((x_1 < 25).astype(int) + (x_1 > 75).astype(int)) + 1.1 * x_2- x_3)
# build first and second datasets (with different underlying probabilities)
y_1 = (xi <= p_1).astype(int)
y_2 = (xi <= p_2).astype(int)
Then, by considering both , , and the uniformly distributed random variable , we can sample each payment outcome randomly: when a uniform random draw is below the borrower's probability, and otherwise.
y_1 = (xi <= p_1).astype(int)
y_2 = (xi <= p_2).astype(int)
We now fit the models using the scikit-learn logistic regression library:
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss
In particular, we initially fit the models on both datasets, then predict on the entire dataset (training and testing):
# Fit the models on both datasets
model_1 = LogisticRegression().fit(x_train, y_1_train.ravel())
model_2 = LogisticRegression().fit(x_train, y_2_train.ravel())
# Predict the probabilities on the entire dataset (both training and test)
pred_X_1 = model_1.predict_proba(x_)
pred_X_2 = model_2.predict_proba(x_)
However, we then compute cross-entropy on both datasets separately for training and testing:
cross_entropy_loss_1_train = log_loss(y_1_train, pred_X_1[:m]) # Use only the training data for y_1
cross_entropy_loss_2_train = log_loss(y_2_train, pred_X_2[:m]) # Use only the training data for y_2
cross_entropy_loss_1_test = log_loss(y_1_test, pred_X_1[m:]) # Use only the test data for y_1
cross_entropy_loss_2_test = log_loss(y_2_test, pred_X_2[m:]) # Use only the test data for y_2
# Cross-entropy loss on the training data:
# Dataset 1: 0.0297
# Dataset 2: 0.1535
# Cross-entropy loss on the test data:
# Dataset 1: 0.0349
# Dataset 2: 0.1486
Secondly, we replicate using normalized data, i.e., we divide by the empirical standard deviation of the training data, for both the test and training sets.
x_train_std = np.std(x_train, axis=0)
x_train_normalized = x_train / x_train_std
x_test_normalized = x_test / x_train_std
We now implement a support vector classifier (SVC) using scikit-learn:
from sklearn.svm import SVC
# Parameters given
gamma = 10e-2
C = 0.2
model_1_normalized = SVC(kernel='rbf', gamma=gamma, C=C, probability=True)
model_2_normalized = SVC(kernel='rbf', gamma=gamma, C=C, probability=True)
# Fit the models for both datasets (this can take up to 60 seconds with SVC)
# Note: "model.fit(...)" modifies the model object in place, so use a separate model object
# for each dataset if you want to keep both fitted models.
model_1_normalized.fit(x_train_normalized, y_1_train.ravel())
model_2_normalized.fit(x_train_normalized, y_2_train.ravel())
We then predict probabilities on the combined training and testing data, then split at :
x_normalized = np.vstack((x_train_normalized, x_test_normalized))
model_1_pred_normalized = model_1_normalized.predict_proba(x_normalized)
model_2_pred_normalized = model_2_normalized.predict_proba(x_normalized)
# Cross-entropy loss on the training data (SVM, normalized):
# Dataset 1: 0.0319
# Dataset 2: 0.0711
Note that using standardized data (i.e., ) instead of normalized data would not change the results.
In particular, we can show that, for two observations and , with and , we have:
So we have that:
Then the SVC fit depends on the data only through these kernels, so the dual coefficients and the intercept are identical:
Once we've done this, we plot the FDR and TPR curves, with the TPR on the horizontal axis, to compute the AUC.

Dataset 1 - Logistic regression: 0.00021
Dataset 1 - SVM: 0.00025
Dataset 2 - Logistic regression: 0.00777
Dataset 2 - SVM: 0.00085
Dataset 1: Logistic regression has the lower FDR/TPR AUC.
Dataset 2: SVM has the lower FDR/TPR AUC.

Recall that, given a threshold , a borrower is "good" if , so we interpret:
- TPR() = true positives / all good borrowers
- FDR() = false positives / (true positives + false positives)
Furthermore, recall that the AUC is the average FDR across all the TPR levels, i.e., the average share of bad loans in the accepted portfolio. This implies that, in this case, lower is better (the opposite of the usual ROC-AUC). Note that the values are very small because, luckily, most borrowers are good.
P2.3 Scenario simulation
We now define a scenario matrix D: each row is a borrower, each column a scenario.
# Define Portfolio and possible outcomes for this portfolio using matrix D
p_good_test = np.asarray(p_2[m:]).reshape(n, 1)
D = np.empty((n, n_scenarios), dtype=bool)
for start in range(0, n_scenarios, chunk_size):
stop = min(start + chunk_size, n_scenarios)
uniforms = np.random.random((n, stop - start))
D[:, start:stop] = uniforms <= p_good_test
We then consider the following three lending strategies:
- We give out a loan to every person in the test data set in the amount of CHF 1000, charging an interest rate of 5.5%.
- Select applicants with logistic-regression predicted repayment probability .
- Select applicants using normalized-feature SVM probabilities .
For each strategy, we then compute the P&L and the VaR using the same scenario matrix, i.e., by doing:
Portfolio_i = np.ones(n, dtype=bool)
repaid_count = D.sum(axis=0)
profit_i = (
repaid_count * loan_amount * interest_rate
- (Portfolio_i.sum() - repaid_count) * loan_amount)
# Plot histogram of profits and losses
import matplotlib.pyplot as plt
plt.hist(profit_i, bins=50)
plt.xlabel("Profit & loss (CHF)")
plt.ylabel("Number of scenarios")
plt.title("Strategy (i): Profit & loss across scenarios")
plt.show()
expected_profit_i = profit_i.mean()
var95_i = -np.quantile(profit_i, 0.05)



We notice that the highest estimated expected P&L is achieved with Strategy (3) (Normalized SVM, 1% (CHF 61,960.54)), the lowest estimated 95%-VaR: with Strategy (CHF -54,070.00).
Furthermore, we notice that selective strategies issue fewer loans and use the same 1% rate, so their performance depends on how well each model identifies reliable borrowers.
P3. Deep Hedging
In the following project, we show a nice application of neural networks in hedging a European call, comparing this approach with the classical Black–Scholes delta-hedging.
We firstly define a European Call Option as the right of the holder to buy a stock for a fixed strike price at a future expiration time . Its payoff at maturity is then
Note that a European call can be exercised only at maturity, while an American call can be exercised at every time until maturity.
In the project, we take the point of view of a trader that sells the option, so we receive a premium at the beginning and, at maturity, we must pay the option payoff . In between, we have to hold shares of the underlying stock to offset the risk of the future payment (in other words: we have to hedge ourselves).
In particular, in the interval , we hold shares, so the gain from those shares is
and the total profit and loss (P&L) of the hedging strategy is:
Even if the term "P&L" may sound misleading, the goal of the hedging strategy is to make the P&L as close to zero as possible across many possible paths, as this would mean that the premium and the trading gains cover the option payment.
Note that, as in major finance literature, we assume no transaction costs, zero interest rates and, furthermore, we consider trading at discrete dates and, at least for the first part, a Black–Scholes stock price model.
For most of the project, we will work with PyTorch, a very common Python library that provides tensors (array-like objects, I wrote something about them here), neural-network layers, automatic differentiation and different optimization algorithms.
In particular, torch.nn contains the neural network building blocks, including the Linear, Tanh, Sequential and ModuleList objects used below.
We assume a zero interest rate, so we model the stock price as
with volatility, Brownian motion and the initial stock price.
Since Brownian motion is continuous with independent random increments, we can write for a small time interval :
so that we have proportional, rather than fixed-size, price changes. (Note that under the risk-neutral measure, the drift of the stock is and, since we assumed , there's no drift term for the stock.)
We firstly show the solution of this stochastic differential equation by applying Itô's formula.
We indeed define , so we have .
If we take the relevant derivatives, we have:
that can be substituted in Itô's formula so we end up with:
We now get our hands dirty with the code. Firstly, we define the parameters:
K, N, s0, T, sigma, r = 1.0, 30, 1.0, 0.25, 0.35, 0.0
dt = T / N
Note that we will have 30 intervals but 31 stock prices, from to , where the last one determines the payoff.
We then simulate the geometric Brownian motion (stock price) as
def simulate_gbm(m, rng):
# generate shocks
# m rows for m simulate paths
# N cols, one for each time interval
Z = rng.standard_normal((m, N))
# log increments
log_incr = -0.5 * sigma**2 * dt + sigma * np.sqrt(dt) * Z
log_S = np.concatenate([np.zeros((m, 1)), np.cumsum(log_incr, axis=1)], axis=1)
return s0 * np.exp(log_S)
Note that we use log returns since they are more convenient: multiplication of prices nicely becomes addition:
We define the training and test paths with 100,000 and 10,000 paths.
rng = np.random.default_rng(0)
S_train = simulate_gbm(100_000, rng)
S_test = simulate_gbm(10_000, rng)
print(S_train.shape, S_test.shape)
We can now finally build the neural network using PyTorch.
In particular, we build networks (each one with its parameters) whose goal is to predict how many shares to hold at time when the price is .
In each network, the first layer transforms one stock price input into 32 values, then nn.Tanh applies the hyperbolic tangent elementwise, so it maps values between , introducing nonlinearity.
The following two layers repeat the pattern. At the end, another linear layer produces the output number of shares. Note that this final layer has no activation function, so the output is NOT constrained in .
torch.manual_seed(1)
N = 30
nh1 = 32
nh2 = 32
def make_network():
return nn.Sequential(
nn.Linear(1, nh1),
nn.Tanh(),
nn.Linear(nh1, nh2),
nn.Tanh(),
nn.Linear(nh2, 1)
)
H = nn.ModuleList([make_network() for _ in range(N)])
In the above code, ModuleList stores all 30 networks and registers them as PyTorch modules. Therefore we can access each network's parameters simply by H.parameters().
Note that each network has:
- params in the first layer
- params in the second layer
- in the last layer
- so we have 1,153 parameters for each network and parameters in total.
Furthermore, note also that time is not an explicit input to a single shared network; it is represented by which network is selected. Indeed, each network sees only the current price, not future prices.