Machine learning based statistical inference in sports analytics

Robert Bajons | Joint work with Lucas Kook

Vienna University of Economics and Business

May 6, 2025

Motivation

How to identify important predictors of a sport-specific outcome?

Sports analytics focuses on identifying the factors that influence a game:
- Top players (Ranking)
- Key predictors for final score
- Key predictors for injuries
- …
Big data era:
- Novel high-resolution data formats (tracking and event stream data)
- Novel statistical toolbox:
  - Flexible machine learning models
  - Features derived by advanced statistical models (statistically enhanced learning)

Motivation

How to identify important predictors of a sport-specific outcome?

Sports analytics focuses on identifying the factors that influence a game:
- Top players (Ranking)
- Key predictors for final score
- Key predictors for injuries
- …
Big data era:
- Novel high-resolution data formats (tracking and event stream data)
- Novel statistical toolbox:
  - Flexible machine learning models
  - Features derived by advanced statistical models (statistically enhanced learning)

Motivation

How to identify important predictors of a sport-specific outcome?

Parametric models (linear/logistic regression):
- Advantages: easy to interpret, valid inference, simple to estimate
- Disadvantages: unrealistic assumptions (too simple), bad predictive performance
Machine learning models:
- Advantages: highly flexible, good predictive performance
- Disadvantages: difficult to interpret, no statistical inference, lack of uncertainty quantification

Example use case

Do HMM features help in defensive coverage prediction in the NFL?

Goal: predict coverage type (1: man coverage, 0: zone coverage) for a defense on a play based on pre-snap features (derived from tracking data)
3 types of features:
- Pre-motion: locations, distances to football, and convex hull of defensive and offensive players
- Naive post-motion: x-distance, y-distance, and total distances covered of players in motion
- Statistically derived post-motion features: Hidden Markov model (HMM) features for players in motion
Typical approach for answering the question:
- Fit models to data with only pre-motion, pre-motion + naive post-motion, all types of features (glmnet, xgboost)
- Variable importance measures
- Compare predictive performances

Example use case: Variable importance

Example use case: Predictive performance

Procedure:
- Split data in train (80 %) and test (20 %) set
- Carefully tune (cross validation) and train model on train set
- Evaluate predictive performance on test set

	glmnet			xgboost
metric	pre	post	HMM	pre	post	HMM
Accuracy	0.8345	0.8453	0.8501	0.8573	0.8837	0.8753
AUC	0.8282	0.8661	0.8703	0.8419	0.8907	0.8839
Logloss	0.4035	0.3647	0.3612	0.3834	0.3240	0.3335

	worst models		medium models		best models

Problems:
- Train-test-split and cross-validation split of data random
- Results dependent on randomly sampled \(\Rightarrow\) repeat procedure with different data splits

Example use case: Take aways

Do HMM features help in defensive coverage prediction in the NFL?

Variable importance:
- HMM features have at least “some” importance
- naive post motion features seem more important
Predictive performance:
- Flexible xgboost models perform better than glmnet
- HMM features seem to add to predictive performance (at least a bit)
Answer to the question: not clear
- Small dataset
- Tuning of parameters difficult
- No possibility for valid significance tests

Statistical inference in a semiparametric model

Setup

How can we identify important predictors of an outcome?

\(Y \in \{0, 1\}\) … binary outcome of of interest (e.g. coverage type, shot ends in a goal, etc.)
\(X \in \mathbb{R}^{d_X}\) … variable(s) of interest
\(Z \in \mathbb{R}^{d_Z}\) … potential confounding factors for variable of interest

Classical approach: logistic regression:
- \(\log\left(\frac{\pi(X,Z)}{1-\pi(X,Z)}\right) = X^{\top}\beta + Z^{\top}\gamma\), where \(\pi(X,Z) = P(Y=1 \mid X,Z)\)
- Well-understood inference on \(\beta\) but assumptions unrealistic

Desired: More flexible approach (machine learning) while still obtaining valid inference

A semiparametric model

Partially linear logistic regression model (PLLM)

\[ \log\left(\frac{\pi(X,Z)}{1-\pi(X,Z)}\right) = X^{\top}\beta + g(Z) \]

Under PLLM: Test for \(Y\) conditionally independent of \(X\) given \(Z\) (\(Y \perp\!\!\!\perp X \mid Z\)) \(\Leftrightarrow\) Test for \(H_0 : \beta = 0\)
- Under \(H_0\): No modelling assumptions on relationship between \(Y\) and \(Z\), and \(X\) and \(Z\)

Use Generalised Covariance Measure (GCM) test (Shah and Peters 2020) for inference

GCM test

Generalised Covariance Measure:

\[ \operatorname{GCM} = \mathbb{E}[\operatorname{Cov}(Y,X \mid Z)] =\mathbb{E}[(Y - \mathbb{E}[Y | Z])(X - \mathbb{E}[X | Z])]\]

Basis for GCM test:

\[Y \perp\!\!\!\perp X \mid Z \Rightarrow \mathbb{E}[\operatorname{Cov}(Y,X \mid Z)] = 0\]

GCM test in practice:
- Given i.i.d. data, use arbitrary machine learning models and
  - Regress \(Y\) on \(Z\) and obtain estimate \(\hat h\) for \(\mathbb{E}[Y | Z]\)
  - Regress \(X\) on \(Z\) and obtain estimate \(\hat f\) for \(\mathbb{E}[X | Z]\)
- Under mild rate conditions (similar to DML Chernozhukov et al. (2018)):
  - Sample version of GCM: \(\operatorname{\widehat{GCM}} : = \sum_{i = 1}^N (Y_i-\hat h(Z_i))(X_i - \hat f(Z_i)) \leadsto \mathcal{N}(0,\frac{\sigma^2}{N})\)

GCM test and PLLM

Interested in testing \(H_0 : \beta = 0\)

Proposition

Consider a PLLM and let \(X \in \mathbb{R}^{d_X}\) with \(\operatorname{Cov}(X,X \mid Z)\) a.s. positively semidefinit. Then \[\beta = 0 \Leftrightarrow \mathbb{E}[(\operatorname{Cov}(Y,X \mid Z)] = 0\]

Takeaways:
- GCM test allows directional testing: alternatives of the form \(H_1 : \beta > 0\)
- Non-parametric test can be used in semi-parametric model \(\rightarrow\) nicely interpretable

GCM test and PLLM

Interested in testing \(H_0 : \beta = 0\)

Proposition

Consider a PLLM and let \(X \in \mathbb{R}\) with \(\operatorname{Var}(X \mid Z) > 0\) a.s. Then \[\operatorname{sign}(\beta) = \operatorname{sign}(\mathbb{E}[\operatorname{Cov}(Y,X \mid Z)])\]

Takeaways:
- GCM test allows directional testing: alternatives of the form \(H_1 : \beta > 0\)
- Non-parametric test can be used in semi-parametric model \(\rightarrow\) nicely interpretable

GCM test and PLLM

Interested in testing \(H_0 : \beta = 0\)

Proposition

Consider a PLLM and let \(X \in \mathbb{R}\) with \(\operatorname{Var}(X \mid Z) > 0\) a.s. Then \[\operatorname{sign}(\beta) = \operatorname{sign}(\mathbb{E}[\operatorname{Cov}(Y,X \mid Z)])\]

Takeaways:
- GCM test allows directional testing: alternatives of the form \(H_1 : \beta > 0\)
- Non-parametric test can be used in semi-parametric model \(\rightarrow\) nicely interpretable

Key points of the proposed framework

Valid inference while still being flexible in the choice of models
Intuitive interpretation via partially linear model coefficient
Directional testing
GCM test results conveniently obtained via comets R package (Kook 2025)
Even more benefits by leveraging test statistics from GCM test (see application to GAX)

In this talk: binary outcome \(\rightarrow\) result generalizable to other cases
- Partially generalized linear models
- Extensions to partially linear cox-models (survival), partially linear beta regression model (proportions), etc.

Applications

Coverage detection in the NFL

Coverage prediction model:
- Binary outcome \(Y\): 1: man coverage, 0: zone coverage
- Confounding variables \(Z \in \mathbb{R}^{36}\): 30 pre-motion features, 6 naive post-motion features
- Variables of interest \(X \in \mathbb{R}^3\): 3 HMM features

Testing for predictive power of HMM features:
- Test for \(Y \perp X \mid Z\), i.e. omnibus test \(H_0 : \beta = 0\), where \(\beta \in \mathbb{R}^3\)
- Test for \(H_0^j : Y \perp X_j \mid (Z, X_{-j})\), \(j \in J := \{1, 2, 3\}\), where \(-j\) denotes removing feature \(j\)
- Test for \(H_0^j : Y \perp X_j \mid Z\), \(j \in J\), i.e. removing all HMM features as conditioning variables

Coverage detection in the NFL: p values

Omnibus test: \(H_0 : \beta = 0\), where \(\beta \in \mathbb{R}^3\)
Tests for: \(H_0^j : Y \perp X_j \mid (Z, X_{-j})\), \(j \in J\)

Coverage detection in the NFL: Test statistic

Test for: \(H_0^j : Y \perp X_j \mid Z\), \(j \in J\)

Shooting skills evaluation via RGAX

Popular measure for shooting skill of players: Goals above expectation (GAX)
- Difference between goals and xG for all shots of player \(i\): \(\operatorname{GAX}_i = \sum_{j=1}^{N_i} (Y_j - \hat h(Z_j))\)
- Problems:
  - Low stability (Baron et al. 2024)
  - High variance and no (direct) uncertainty quantification
  - Biases in xG models due to overrepresented players (Davis and Robberechts 2024)

Solution:
- Residualized GAX (RGAX): \(\operatorname{RGAX}_i = \sum_{j=1}^{N} (Y_j-\hat h(Z_j))(X_j^i - \hat f(Z_j))\)
- RGAX is sample version of (scaled) GCM
- Doubly robust score \(\Rightarrow\) valid inference
- Interpretation RGAX: additional regression accounts for whether a player would take the shot under the circumstances described by \(Z_i\)

Shooting skills evaluation via RGAX: results

player	GAX	RGAX	Rank GAX	Rank RGAX	Absolute Difference	Relative Difference
Cristiano Ronaldo	-2.02	2.68	288	34	254	7.47
Miralem Pjanić	1.83	3.27	65	20	45	2.25
Lorenzo Insigne	0.06	1.80	162	50	112	2.24
Edinson Cavani	-0.90	1.22	232	73	159	2.18
Alessandro Florenzi	2.27	3.59	47	15	32	2.13
Antoine Griezmann	5.84	6.42	3	1	2	2.00
Mohamed Salah	3.46	4.29	19	7	12	1.71
Roberto Firmino	1.92	3.00	58	23	35	1.52
Lionel Messi	2.49	3.44	42	17	25	1.47
Ivan Rakitić	1.00	2.13	103	44	59	1.34

Conclusion

Identifying important factors for a sport-specific outcome difficult:
- Well known parametric models often not suitable
- Machine learning models difficult to interpret and do not allow for statistical inference

Machine learning based statistical inference in a semiparametric model:
- Valid statistical inference combined with flexible machine learning models
- Well-understood interpretation of results
- Conveniently implemented in via R-package comets

Various use cases:
- Coverage type in the NFL
- Enhancing player evaluation metrics such as GAX (soccer), EPA (American football), SI (basketball), …
- Identifying drivers of injuries in survival setup

Thank you for your attention!

References

Baron, Ethan, Nathan Sandholtz, Devin Pleuler, and Timothy C. Y. Chan. 2024. Journal of Quantitative Analysis in Sports 20 (1): 37–50. https://doi.org/doi:10.1515/jqas-2022-0107.

Chernozhukov, Victor, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. 2018. “Double/Debiased Machine Learning for Treatment and Structural Parameters.” The Econometrics Journal 21 (1): C1–68. http://www.jstor.org/stable/45172267.

Davis, Jesse, and Pieter Robberechts. 2024. “Biases in Expected Goals Models Confound Finishing Ability.” https://arxiv.org/abs/2401.09940.

Kook, Lucas. 2025. comets: Covariance Measure Tests for Conditional Independence. https://doi.org/10.32614/CRAN.package.comets.

Shah, Rajen D., and Jonas Peters. 2020. “The hardness of conditional independence testing and the generalised covariance measure.” The Annals of Statistics 48 (3): 1514–38. https://doi.org/10.1214/19-AOS1857.