Where your models can be trusted:
evaluating spatial machine learning reliably

Jakub Nowosad, https://jakubnowosad.com/

Machine Learning for Earth Observation 2026, Exeter, UK

2026-06-23

Setup

Data and code

The data and code for this workshop are available at https://jakubnowosad.com/ml4eo2026workshop.

The main dataset, Germany, is a synthetic dataset based on real-world predictors and a simulated target variable, and three sampling designs.

Software

• https://hannameyer.github.io/CAST/
• https://cran.r-project.org/package=CAST
• Meyer et al. (2026)

Motivation

Spatial predictions are often evaluated as if the testing data were a good stand-in for the places where the model will later be used.

In practice, this assumption often fails:

• nearby observations can make random CV look too optimistic
• prediction locations may be environmentally unlike the training data
• a single accuracy number hides where a model is likely to be more or less trustworthy

This workshop focuses on how to practically apply prediction domain-adaptive evaluation to:

1. Ensure that the reported performance is realistic for the intended prediction scenario
2. Identify where in the prediction domain the model inputs are sufficiently represented by the training data
3. Interpret how the expected error varies across the prediction domain

Prediction domain and extrapolation continuum

Area of applicability (AoA) and local point density (LPD)

Identify areas where the environment is not well represented, making predictions less trustworthy (Area of Applicability – AoA, Meyer and Pebesma, 2021); also local point density (LPD, Schumacher et al., 2025)

See also: MESS, Elith et al., 2010, Jung et al., 2020, and Velazco et al., 2023 for other related approaches

Area of applicability for different validation strategies

Specific evaluation strategy

Adaptive evaluation (kNNDM)

k-fold Nearest Neighbor Distance Matching (kNNDM, Linnenbrink et al., 2024) matches folds to the prediction scenario using distance structure (either in geographic or predictor space).

See also: NNDM, Milà et al. (2022) and DA-CV, Wang et al. (2025) for other related approaches

Evaluation results for different validation strategies

Error profiles

Error profiles show how expected error (based on AoA or LPD) relates to true error. They can be used to identify where predictions are more or less trustworthy, and to adjust the AoA threshold or interpret LPD values.

Next: live code

Germany example script

Exercises

Optional hands-on exercises

Take-home messages

• Evaluation should reflect the prediction scenario, not just the sample at hand
• Random CV can be too optimistic for clustered samples, and spatial CV can be too pessimistic for random samples
• kNNDM CV can be a useful adaptive strategy to match the evaluation to the prediction scenario
• AoA and LPD help identify where predictions are more or less supported by the training data
• Expected performance is often spatially variable rather than uniform across a map (Error profiles)

There is still much more to explore, including feature-space distance matching, alternative adaptive CV strategies, uncertainty quantification, temporal transfer, etc.

Our current related work

• Linnenbrink et al. (2026) Moving beyond spatial and random cross-validation in environmental modelling: a call for prediction-domain adaptive evaluation
• Nowosad et al. (2026) Navigating challenges in spatial machine learning: Validation, uncertainty, algorithms, and reproducibility
• Spatio-Temporal Modelling Protocol (STeMP) – feedback welcome!

Contact and materials

Mastodon: fosstodon.org/@nowosad

Website: https://jakubnowosad.com

Workshop materials:

https://jakubnowosad.com/ml4eo2026workshop