PEP: Evaluating Tackles in American Football

Robert Bajons

Vienna University of Economics and Business

May 24, 2024

Collaborators

Jan-Ole Koslik
PhD Student
Universität Bielefeld

Rouven Michels
PhD Student
Universität Bielefeld

Marius Ötting
Postdoc
Universität Bielefeld

Introduction

How can we value a tackle?

Evaluate a hypothetical scenario: What would have happened, if the player had missed the tackle?

Introduction

How can we value a tackle?

Evaluate a hypothetical scenario: What would have happened, if the player had missed the tackle?

Measure the yards saved by a tackle:
- Predict the end-of-play yard line (EOPY) when removing the tackler from the data.
- Compare to real EOPY (or predicted on actual data with tackler).
Problem: Value of yards saved depends on the game state.
- Example 1: 2 yards saved close to the defenders end zone vs. 2 yards saved close to the opponents end zone.
- Example 2: 2 yards saved on 4th and 10 vs. 2 yards saved on 4th and 2.

Introduction

How can we value a tackle?

Evaluate a hypothetical scenario: What would have happened, if the player had missed the tackle?

Measure the prevented expected points (PEP) by a tackle:
- Mapping from EOPY to expected points.
- Point prediction of mean EOPY lacks uncertainty quantification.

Introduction

How can we value a tackle?

Evaluate a hypothetical scenario: What would have happened, if the player had missed the tackle?

Measure the prevented expected points (PEP) by a tackle:
- Mapping from EOPY to expected points.
- Point prediction of mean EOPY lacks uncertainty quantification.
- Full conditional density estimate necessary to calculate expected points.

Gameplan

Estimate the full conditional density of the EOPY.
Estimate an expected points model as mapping of the EOPY.
Estimate a tackle value by comparing the real outcome with the hypothetical outcome:
- Obtain the average expected points from the hypothetical play based on the full conditional density estimate of the EOPY.
- Obtain the expected points from the real EOPY.
- Calculate the prevented expected points (PEP) as the difference of the above.

Data and preprocessing

Data

NFL Next Gen Stats player tracking data via NFL Big Data Bowl.
- \((x,y)\)-coordinates, speed, acceleration of each player and ball
- Data from weeks 1-9 of 2022 NFL season.
- Includes roughly 11900 plays with tackles.
Additionally:
- play description data of each play.
- tackle information data of each play
- player and game information data.

Data preprocessing

Transform each play such that direction of play is the same:
- redefine \(x\)-variable as the \(x\)-distance to the end zone (play directions from right to left and relevant end zone at zero).
- center \(y\)-variable (center of field at zero).
- changing direction variable (0 degrees \(\Rightarrow\) heading towards the end zone).
Feature engineering:
- \(x\),\(y\)-coordinates, speed, acceleration, distance covered and direction.
- euclidean distance, \(x\)-distance and \(y\)-distance to the ball carrier.
- absolute difference of defender’s direction and the angle of the shortest segment between defender and ball carrier.
- order all players by euclidean distance to the ball carrier.

Estimating the end-of-play yard line (EOPY)

Random forest conditional density estimation

Repeatedly grow different regression trees.
- Each tree partitions data set into subgroups based on simple decisions until a suitable stopping criterion is met.
Trees are build to decrease variance (decrease correlation between single trees) via randomization:
- trees are built on bootstrap samples of the data.
- best split is found on only a subset of all predictors at each node.
Usually: predictions obtained by averaging individual tree predictions.

Here: Use all individual tree predictions as estimator of the conditional density of the outcome.

Model training

Train random forest on preprocessed data:
- Outcome: yards to be gained, i.e. difference of \(x\)-position of the ball carrier to the EOPY.
- Covariates: all features engineered in previous steps (at each time point).
- Only include the ten closest defenders (necessary for hypothetical scenario!)
Train two models for evaluation of tackles:
- Use 1/2 of data for training, predict yards to be gained on the other half
Model evaluation:
- MAE = 3.212 yards (RMSE = 5.864).
- Similar to other related models (Yurko et al. (2020))

Expected points calculation

Expected Points (EP)

Expected points: Given the game situation, how many points do we expect the team to score?

\[ {\rm EP} = \mathbb{E}[Y|X] = \sum_{y} y \cdot \mathbb{P}(Y = y|X),\] \(Y \dots\) scoring outcomes, \(X \dots\) covariates describing the game state.

7 possible scoring outcomes¹: touchdown (7 points), field goal (3 points), safety (2 points), opponent safety (-2 points), opponent field goal (-3 points), opponent touchdown (-7 points), no score (0 points).

EP model

Estimate the probabilities \(\mathbb{P}(Y = y|X)\) for each scoring outcome.
Covariates \(X\):
- need to be extractable solely from the EOPY predictions of the random forest \(\Rightarrow\) limited feature space!
- yard line of the play (adjusted LOS), yards to go, down, quarter, a home team indicator, time outs remaining for each team.
XGBoost model for estimation of probabilities with careful tuning and evaluation.

Quantifying tackle value

EP from EOPY predictions

At the time of tackle: estimate the mean expected points from the EOPY predictions.

\[\mathbb{E}(g(Y) \mid x) = \int g(y) \: \hat{f} (y \mid x) \: dy,\]

\(g(y)\): Mapping from the yard line \(y\) to expected points (EP model).
\(\hat{f} (y \mid x)\): conditional density estimate from the random forest.

Calculation could be done two ways:
- Obtain kernel density estimate from individual tree predictions and integrate numerically \(\Rightarrow\) subjectivity in kernel and bandwidth choice.
- Preferred choice: Treat individual tree predictions as samples from the conditional density and use Monte Carlo estimate.

\[\frac{1}{1000} \sum_{i=1}^{1000} g(\hat{y}_i)\]

Prevented Expected Points (PEP)

\(\mathbb{E}(g(Y) \mid x)\) allows to analyze predicted yards gained on a EP scale.

Tackle evaluation:

Still interested in evaluating a hypothetical scenario \(\Rightarrow\) evaluate mean expected points for the tackle, when removing the tackler from play \(\mathbb{E}(g(Y) \mid x_{removed})\):
- Compute draws \(\hat y_{removed}\) from conditional density \(\hat{f} (y \mid x_{removed})\)
- Average over \(g(\hat y_{removed})\) to obtain obtain estimate for EP.
We know the true outcome \(y_0\) (the true EOPY) \(\Rightarrow\) compute EP \(g(y_0)\).
Prevented expected points:

\[\text{PEP} = \text{E}(g(Y) \mid x_{removed}) - g(y_0).\]

Evaluating players

Aggregation of PEP values

How can we evaluate a player’s tackling ability?

Sum up the PEP values of each tackle a player makes.
- Problem: Some positions tackle more than others \(\Rightarrow\) Linebackers and safeties accumulate more tackles (and thus also a higher PEP value).

Aggregation of PEP values

How can we evaluate a player’s tackling ability?

Sum up the PEP values of each tackle a player makes.
- Problem: Some positions tackle more than others \(\Rightarrow\) Linebackers and safeties accumulate more tackles (and thus also a higher PEP value).
Average the PEP values of players (if tackle reasonably often).
- Problem: Still some positions tackles more and some position tend to tackle more often in threatening situations \(\Rightarrow\) e.g.: defensive line on 4th and short.

Mixed effects models

How can we evaluate a player’s tackling ability?

Model the impact of players on PEP values for specific tackle.
- Treat tackles as random effects, i.e. \(T_i \sim \mathcal{N}(0,\sigma^2)\), \(i = 1,\dots,N_{tacklers}\).
- Account for tackle specific facts: position of tackler, rusher (also random effect), down, turnover, …

PEP not normally distributed (heavy tails). Use GAMLSS (R. A. Rigby and D. M. Stasinopoulos (2005)) to fit a flexible model: \[{\rm PEP} \sim lst(\mu,\tau^2,\nu),\] with \(g(\mu) = X \beta + Z\gamma\) (identity link on mean).

Uncertainty in the random effect estimate for players by bootstrapping (cleverly).

Player effects

More results (and where to find them)

Summary and final slide

PEP: quantifying tackle values on a relevant scale to the game.
Using conditional density estimates to propagate uncertainty from yards gained to EP scale.

Extensions:

Ball Carrier? Evaluate ball carriers ability to maintain EPs despite being tackled.
Missed tackles and assisted tackles? Measuring the impact of players close by on PEP.

Thank you for your attention!

References

Carl, Sebastian, and Ben Baldwin. 2023. NflfastR: Functions to Efficiently Access NFl Play by Play Data.

R. A. Rigby, and D. M. Stasinopoulos. 2005. “Generalized Additive Models for Location, Scale and Shape,(with Discussion).” Applied Statistics 54.3: 507–54.

Yurko, Ronald, Francesca Matano, Lee F. Richardson, Nicholas Granered, Taylor Pospisil, Konstantinos Pelechrinis, and Samuel L. Ventura. 2020. “Going Deep: Models for Continuous-Time Within-Play Valuation of Game Outcomes in American Football with Tracking Data.” Journal of Quantitative Analysis in Sports 16 (2): 163–82. https://doi.org/10.1515/jqas-2019-0056.

Yurko, Ronald, Samuel Ventura, and Maksim Horowitz. 2019. Journal of Quantitative Analysis in Sports 15 (3): 163–83. https://doi.org/doi:10.1515/jqas-2018-0010.