metric | pre | post | HMM |
|---|---|---|---|
Accuracy | 0.8573 | 0.8837 | 0.8753 |
AUC | 0.8419 | 0.8907 | 0.8839 |
Loglik | -0.3834 | -0.3240 | -0.3335 |
color scale | worst models | medium models | best models |
Machine learning based statistical inference in sports analytics
Robert Bajons | Joint work with Lucas Kook
Vienna University of Economics and Business
Jul 16, 2025
Motivation
Motivation
Sports analytics = Statistics + Sports Science:
- Advancements in data collection (tracking and event data)
- Multi-billion dollar industry: increasing demand for statistical solutions (Lolli et al. 2025)
How to identify important predictors of a sport-specific outcome?
Motivation
Sports analytics = Statistics + Sports Science:
- Advancements in data collection (tracking and event data)
- Multi-billion dollar industry: increasing demand for statistical solutions (Lolli et al. 2025)
How to identify important predictors of a sport-specific outcome?
Machine learning models vs parametric models (linear/logistic regression):
- Advantages: highly flexible, able to capture complex relationships, good predictive performance
- Disadvantages: difficult to interpret, no (valid) statistical inference, lack of uncertainty quantification
Example use case
Do HMM features help in defensive coverage prediction in the NFL?
Goal: predict coverage type (1: man coverage, 0: zone coverage) for a defense on a play based on pre-snap features (derived from tracking data)
3 types of features:
- Pre-motion: locations, distances to football, and convex hull of defensive and offensive players
- Naive post-motion: x-distance, y-distance, and total distances covered of players in motion
- Statistically derived post-motion features: Hidden Markov model (HMM) features for players in motion
Typical approach for answering the question:
- Fit models to data with only pre-motion, pre-motion + naive post-motion, all types of features (xgboost)
- Variable importance measures
- Compare predictive performances
Example use case: Variable importance
Example use case: Predictive performance
- Split data in train (80 %) and test (20 %) set
- Carefully tune (cross validation) and train model on train set
- Evaluate predictive performance on test set
Problems:
- Train-test-split and cross-validation split of data random
- Results dependent on randomly sampled \(\Rightarrow\) repeat procedure with different data splits
Statistical inference in a semiparametric model
Setup
How can we identify important predictors of an outcome?
- \(Y \in \{0, 1\}\) … binary outcome of of interest (e.g. coverage type, shot ends in a goal, etc.)
- \(X \in \mathbb{R}^{d_X}\) … variable(s) of interest
- \(Z \in \mathbb{R}^{d_Z}\) … potential confounding factors for variable of interest
Classical approach: logistic regression:
- \(\log\left(\frac{\pi(X,Z)}{1-\pi(X,Z)}\right) = X^{\top}\beta + Z^{\top}\gamma\), where \(\pi(X,Z) = P(Y=1 \mid X,Z)\)
- Well-understood inference on \(\beta\) but assumptions unrealistic
- Desired: More flexible approach (machine learning) while still obtaining valid inference
A semiparametric model
Partially linear logistic regression model (PLLM)
\[ \log\left(\frac{\pi(X,Z)}{1-\pi(X,Z)}\right) = X^{\top}\beta + g(Z) \]
Interested in testing \(H_0 : \beta = 0\)
- Under \(H_0\): No modelling assumptions on relationship between \(Y\) and \(Z\), and \(X\) and \(Z\)
- Test for \(Y\) conditionally independent of \(X\) given \(Z\) (\(Y \perp\!\!\!\perp X \mid Z\))
Use Generalised Covariance Measure (GCM) test (Shah and Peters 2020) for inference
- \(\operatorname{GCM} = \mathbb{E}[\operatorname{Cov}(Y,X \mid Z)] =\mathbb{E}[(Y - \mathbb{E}[Y | Z])(X - \mathbb{E}[X | Z])]\)
- Basis of the test: \(Y \perp\!\!\!\perp X \mid Z \Rightarrow \mathbb{E}[\operatorname{Cov}(Y,X \mid Z)] = 0\)
GCM test and PLLM
- Interested in testing \(H_0 : \beta = 0\)
Takeaways:
- GCM test allows directional testing: alternatives of the form \(H_1 : \beta > 0\)
- Non-parametric test can be used in semiparametric model \(\rightarrow\) nicely interpretable
Proposition
Consider a PLLM and let \(X \in \mathbb{R}^{d_X}\) with \(\operatorname{Var}(X \mid Z)\) a.s. positive semidefinite. Then \[\beta = 0 \Leftrightarrow \mathbb{E}[(\operatorname{Cov}(Y,X \mid Z)] = 0\]
GCM test and PLLM
- Interested in testing \(H_0 : \beta = 0\)
Takeaways:
- GCM test allows directional testing: alternatives of the form \(H_1 : \beta > 0\)
- Non-parametric test can be used in semiarametric model \(\rightarrow\) nicely interpretable
Proposition
Consider a PLLM and let \(X \in \mathbb{R}^{d_X}\) with \(\operatorname{Var}(X \mid Z)\) a.s. positive semidefinite. Then \[\beta = 0 \Leftrightarrow \mathbb{E}[(\operatorname{Cov}(Y,X \mid Z)] = 0\]
Proposition
Consider a PLLM and let \(X \in \mathbb{R}\) with \(\operatorname{Var}(X \mid Z) > 0\) a.s. Then \[\operatorname{sign}(\beta) = \operatorname{sign}(\mathbb{E}[\operatorname{Cov}(Y,X \mid Z)])\]
GCM test and PLLM
- Interested in testing \(H_0 : \beta = 0\)
Takeaways:
- GCM test allows directional testing: alternatives of the form \(H_1 : \beta > 0\)
- Non-parametric test can be used in semiparametric model \(\rightarrow\) nicely interpretable
Proposition
Consider a PLLM and let \(X \in \mathbb{R}^{d_X}\) with \(\operatorname{Var}(X \mid Z)\) a.s. positive semidefinite. Then \[\beta = 0 \Leftrightarrow \mathbb{E}[(\operatorname{Cov}(Y,X \mid Z)] = 0\]
Proposition
Consider a PLLM and let \(X \in \mathbb{R}\) with \(\operatorname{Var}(X \mid Z) > 0\) a.s. Then \[\operatorname{sign}(\beta) = \operatorname{sign}(\mathbb{E}[\operatorname{Cov}(Y,X \mid Z)])\]
Applications
Coverage detection in the NFL
Coverage prediction model:
- Binary outcome \(Y\): 1: man coverage, 0: zone coverage
- Confounding variables \(Z \in \mathbb{R}^{36}\): 30 pre-motion features, 6 naive post-motion features
- Variables of interest \(X \in \mathbb{R}^3\): 3 HMM features
Testing for predictive power of HMM features:
- Test for \(Y \perp\!\!\!\perp X \mid Z\), i.e. omnibus test \(H_0 : \beta = 0\), where \(\beta \in \mathbb{R}^3\)
- Test for \(H_0^j : Y \perp\!\!\!\perp X_j \mid (Z, X_{-j})\), \(j \in J := \{1, 2, 3\}\), where \(-j\) denotes removing feature \(j\)
Coverage detection in the NFL: p values
- Omnibus test: \(H_0 : \beta = 0\), where \(\beta \in \mathbb{R}^3\)
- Tests for: \(H_0^j : Y \perp\!\!\!\perp X_j \mid (Z, X_{-j})\), \(j \in J\)
Coverage detection in the NFL: Test statistic
- Tests for: \(H_0^j : Y \perp\!\!\!\perp X_j \mid (Z, X_{-j})\), \(j \in J\)
Conclusion
Identifying important factors for a sport-specific outcome is difficult:
- Well-known parametric models are often not suitable
- Machine learning models are difficult to interpret and do not allow for statistical inference
Machine learning based statistical inference in a semiparametric model:
- Valid statistical inference combined with flexible machine learning models
- Well-understood interpretation of results
- Conveniently implemented via
R-packagecomets(Kook 2025)
Various use cases:
- Coverage type in the NFL
- Enhancing player evaluation metrics such as GAX/GSAX (soccer, ice hockey), SI (basketball), …
- Identifying drivers of injuries in survival setup
Thank you for your attention!
References
Baron, Ethan, Nathan Sandholtz, Devin Pleuler, and Timothy C. Y. Chan. 2024. Journal of Quantitative Analysis in Sports 20 (1): 37–50. https://doi.org/doi:10.1515/jqas-2022-0107.
Chernozhukov, Victor, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. 2018. “Double/Debiased Machine Learning for Treatment and Structural Parameters.” The Econometrics Journal 21 (1): C1–68. http://www.jstor.org/stable/45172267.
Davis, Jesse, and Pieter Robberechts. 2024. “Biases in Expected Goals Models Confound Finishing Ability.” https://arxiv.org/abs/2401.09940.
Kook, Lucas. 2025. comets: Covariance Measure Tests for Conditional Independence. https://doi.org/10.32614/CRAN.package.comets.
Lolli, Lorenzo, Pascal Bauer, Callum Irving, Daniele Bonanno, Oliver Höner, Warren Gregson, and Valter Di Salvo. 2025. “Data Analytics in the Football Industry: A Survey Investigating Operational Frameworks and Practices in Professional Clubs and National Federations from Around the World.” Science and Medicine in Football 9 (2): 189–98. https://doi.org/10.1080/24733938.2024.2341837.
Shah, Rajen D., and Jonas Peters. 2020. “The hardness of conditional independence testing and the generalised covariance measure.” The Annals of Statistics 48 (3): 1514–38. https://doi.org/10.1214/19-AOS1857.
Appendix
GCM test details
Generalised Covariance Measure:
\[ \operatorname{GCM} = \mathbb{E}[\operatorname{Cov}(Y,X \mid Z)] =\mathbb{E}[(Y - \mathbb{E}[Y | Z])(X - \mathbb{E}[X | Z])]\]
Basis for GCM test:
\[Y \perp\!\!\!\perp X \mid Z \Rightarrow \mathbb{E}[\operatorname{Cov}(Y,X \mid Z)] = 0\]
GCM test in practice:
- Given i.i.d. data, use arbitrary machine learning models and
- Regress \(Y\) on \(Z\) and obtain estimate \(\hat h\) for \(\mathbb{E}[Y | Z]\)
- Regress \(X\) on \(Z\) and obtain estimate \(\hat f\) for \(\mathbb{E}[X | Z]\)
- Under mild rate conditions (similar to DML Chernozhukov et al. (2018)):
- Sample version of GCM: \(\operatorname{\widehat{GCM}} : = \sum_{i = 1}^N (Y_i-\hat h(Z_i))(X_i - \hat f(Z_i)) \leadsto \mathcal{N}(0,\frac{\sigma^2}{N})\)
- Given i.i.d. data, use arbitrary machine learning models and
Shooting skills evaluation via RGAX
Popular measure for shooting skill of players: Goals above expectation (GAX)
- Difference between goals and xG for all shots of player \(i\): \(\operatorname{GAX}_i = \sum_{j=1}^{N_i} (Y_j - \hat h(Z_j))\)
- Problems:
- Low stability (Baron et al. 2024)
- High variance and no (direct) uncertainty quantification
- Biases in xG models due to overrepresented players (Davis and Robberechts 2024)
Solution:
- Residualized GAX (RGAX): \(\operatorname{RGAX}_i = \sum_{j=1}^{N} (Y_j-\hat h(Z_j))(X_j^i - \hat f(Z_j))\)
- RGAX is sample version of (scaled) GCM
- Doubly robust score \(\Rightarrow\) valid inference
- Interpretation RGAX: additional regression accounts for whether a player would take the shot under the circumstances described by \(Z_i\)
Shooting skills evaluation: GAX vs. RGAX
Coverage prediction: Variable importance
Coverage prediction: Predictive performance
Coverage prediction: More tests on variables
- Test for: \(H_0^j : Y \perp\!\!\!\perp X_j \mid Z\), \(j \in J\) (removing all HMM features as conditioning variables)
