Innovative or Just Very Lucky?

The role of random variation, sometimes labelled as luck, is increasingly being recognised in the interpretation of footballing stats. The record of a player's on-field actions over a single season will always be a combination of his true capabilities and a measure of randomness, that sometimes inflates his figures and at other times reduces them. The very good will often be not quite as good as one outstanding season among many merely good ones appears to indicate and a hugely disappointing year from a journeyman may prove to be partly down to bad luck and won't be precisely repeated if he is given the benefit of the doubt.

However, the temptation to always presume that any deviation from the expected level of performance is always down to solely random variation is to assume the existence a uniformity of approach and application of the talent available to a coach, that may not prevail throughout the league.

To celebrate London hosting another regular season NFL game this weekend, I'll draw an example from gridiron. Defended passes are more frequent occurrences than the more valuable full bloodied interception, although both disciplines require a similar skill set and are therefore, reasonably closely correlated. It is analogous to a situation in football, where the much more numerous final third touches in one season appear to better predict goals scored in a subsequent season.

Over the last ten completed years, an NFL side could have seen between 60 to 130 defended passes in a year and between 6 to 30 actual interception made by their defense. You can use the relationship between passes defended and interceptions to produce an expected number of interceptions in a single season. This derived interception total can then be used instead of a side's actual interception total in year N to predict interceptions made in year N+1. In around 65% of the cases the expected figure is a better predictor of future picks.

At the start of the 2012 season, passed defended from 2011 suggested that New England's defense would grab 17 regular season picks. They actually caught 20, so they exceeded the prediction and it is tempting to say they were slightly better than average (the average number of interceptions over the last decade is 16 per season) and lucky. In the previous season, the same thing happened, they beat the prediction from a model that, overall improves the reliability of simply using previous year totals across all 32 NFL teams. And the next....and the next....

How NWE Out Performed A Predictive Model for Defensive Interceptions.

Year	Actual Interceptions.	Predicted Interceptions	Lucky?
2012	20	17	Yes
2011	23	13	Yes
2010	25	17	Yes
2009	18	15	Yes
2008	14	13	Yes
2007	19	15	Yes
2006	22	15	Yes
2005	10	14	No
2004	20	15	Yes
2003	29	26	Yes

Over the ten year period, NWE outperformed the model (essentially a predictive regression) in nine years. The most likely number of seasons a side would expect to out perform the prediction is, unsurprisingly, five.
For a side to out perform nine times out of ten, if the regression models reality could happen by chance around once every 500 team seasons. We have looked at ten years for 32 teams, so to have found one team who went 9-1 against the model is certainly unusual.

However things get worse for the model, because Chicago beat predictions in eight of the ten seasons (about a 2% chance of happening by chance alone). So now we have two sides that recorded more interceptions than predicted by this model. 26 of the 32 teams over the ten seasons have over (or under) performing years that are within 2 season of the five season average. So the model works well for them.

But for NWE, Chicago, as well as Green Bay, Atlanta, Tennessee and (appropriately) Tampa it under estimates, consistently their intercepting prowess or alternatively, we have to assume these six teams were good and lucky (very, very lucky as a group). As ever, everyone is able to set their own level of confidence in each an every possible scenario or explanation.

An alternative solution is that although improving on a naive use of previous interception totals, this model still omits all possible causes for a side's intercepting abilities. Defensive scheme is one glaring omission (Tampa 2 type zone defenses invite interceptions to be thrown, whereas bounty hunting blitzes, recently favoured by New Orleans prize the opponent above the ball). It also omits the coaching input from defensive gurus, such as Belichick at NWE (who wasn't afraid to use wide receivers on the defensive side of the ball and wasn't above secretly taping their opponents (aka cheating), although this usually referred to the offensive side of the Pats game).

In short, models don't always capture everything and the missing bits may be what sets some teams, coaches and players above the rest or at least fails to identify those that may demonstrate a different tactical approach. A sides relationship to a measurement that can well describe the majority of the league may be branding them lucky, rather than recognising the flaws of a deficient model. Invoking luck to cover the unexpectedly different performance with undue haste should be resisted at least until we see if the "luck" is sustainable and therefore may have a causative agent.