Stability of impact stats in the playoffs (Patreon)

One of the major issues with plus-minus based metrics is that they are super noisy. Even single-year RAPMs, put through the mathematical magic of "regularization," are still kind of noisy for my taste.

For my recent video about star players improving or declining in the playoffs, I needed to call on metrics with smaller samples than a full season or two worth of games, which leads to the question: 

How much of a sample do we need in the playoffs to be confident in impact?

If you've followed me for a while, you know I often refer to playoff stats in 3-year stretches (or sometimes more), and this is because a few games in the postseason is insufficient to really ballpark a player's value. 

Box only is most stable

Box only metrics, like my version of BPM, are inherently more stable than metrics that incorporate plus-minus; it only takes a few hundred minutes of play to achieve strong stability for non-superstars in this stat. 

For instance, in December of this year, there were 48 players who logged between 400-600 minutes who would then go on to log another 750 minutes by the end of the season. On average, those players finished within 0.5 points of their December BPM. 46 of the 48 were within 1.6 points of their final BPM and 34 of 48 within a point. And some of that change is from organic player growth/decline. 

Higher-value players had slightly more budge, changing on average by 0.7 points at the end of the year from where they were 800-1000 minutes into the season. This includes rookies like Trae Young, who jumped 3.4 points by the end of the year in what was clearly improved play. (Young had by far the biggest jump from early season to end season of any player.)

Hybrids are less stable

When we introduce plus-minus, it takes a larger sample to achieve stability. Comparing PIPM from the same point back in December shows a weaker correlation to final-year PIPM values. Lower-minute players ended up about the same as the box only results above, but higher-minutes players were more volatile in PIPM than BPM. 

At 1000-1200 minutes into the season, the average 30-something minute player (who went on to log another 1000 minutes this year) ended up about 0.9 points from his final PIPM. It takes about 1500-1800 minutes for key players to see their PIPM come within 0.4-0.5 points of its final season value (among players who logged another 750 minutes or more). This is still slightly more volatile than where the same players were in BPM after 1200-1400 minutes played.

ESPN's RPM is actually less stable than PIPM. After 1100-1400 minutes among these top-minute players, the average player (who logged at least 1000 more minutes) was 0.8 points away from his final total. That's about half of a regular season, and this year, nearly one-third of those players didn't finish within a point of their final RPM value.

This wasn't rookies either. In the second half of the year, RPM for Damian Lillard, Jrue Holiday, Klay Thompson and Giannis Antetokounmpo changed by 1.5-2.0 points. Some of this is likely the effect of prior-season weighting, but that weighting is there precisely because of the volatility in small samples. 

Mid-to-late season PIPM was almost identical to final PIPM among most players, but RPM still had an average change of 0.6-0.7 points among lower-minute players 50 or 60 games into the season. 

Regressing play-by-play data is noisy, even with some priors. 

The playoffs -- how much is enough?

Hopefully this gives you a good feel for when variance is no longer an issue among these all-in-one stats. The noisiness of impact metrics can become a limiting factor in analyzing postseason play -- many of us judge players swiftly based on playoff-only numbers, without realizing how unstable some of those numbers are in small samples. 

A star player will usually play no more than 250 minutes in a 6-game series, and without making a deep playoff run, that's too small of a sample to reach many conclusions. So how much is enough?

 If I had to set some rough guidelines, I'd say: 

• It's very hard to use adjusted plus-minus (or a derivative) even across consecutive postseasons because of the small sample size. 
• Hybrids like PIPM (and Augmented Plus-Minus) are actually less noisy given the heftier box score component, but you still probably want something closer to 1500 minutes to feel really good them. Samples closer to 1000 minutes are pretty good, but have a little noise in them still. 
• Box-score-only stabilizes the fastest, and at around 1000-1200 minutes is really stable among high-minute players, and has a little noise after only 500-600 minutes played. 

Feel free to inject these concepts into any conversations about playoff stats.