Analytics Might Need to Get Lucky.

Let's try a thought experiment.

Analytics has taken root in the game and you've been tasked with improving the goalkeeping abilities of one of your two goalies. Both are equally talented at every aspect of the game, (you've developed a statistical method that strips away random variation to tell you this, but of course that must remain proprietary). The two keepers are identical in virtually every way, both on and off the field. They are the Fabio and Rafael of the goal keeping ranks.

But there is one facet of the game where a slight improvement may be possible. Namely, saving penalty kicks.

You load up a database with physical and locational data for every type of penalty kick. The pace of approach, the angle of run, the kicker's natural foot, even the flavour of ball being kicked. And the numbers allow you to give the keeper a slight advantage compared to his previous spot kick saving talent levels.

You pass the information onto just one of the keepers because the club has decided to sell the other (now slightly inferior) goalie for a hefty profit. Just to be sure, you trial the keepers over a million penalty kicks (it is a thought experiment, remember) and sure enough, keeper A saves kicks at a rate of 25% and keeper B is stuck at the league average of 22%, (we could have used our proprietary tool to strip out random variation, but I made that bit up).

Prior to this analytical tweaking, your robotic, talent flat lining keepers and their 22% ability to save penalties had faced 43 kicks spread over eight seasons and they had prevented 8 from entering the net. So they'd been a bit unlucky and saved 19%. Even for a true 22% keeper, you are going to save 8 or fewer about 34% of the time.

Keeper B leaves and keeper A, his talent constant, remains the starter for the next two seasons. Buoyed by the knowledge that their keeper can save 25% of the penalties compared to just 22% previously, the defence thinks nothing of fouling opponents in the box and they concede 17 penalties in two seasons....and all but two are scored.

Just 12% stay out of the net! It's a disaster.

Meanwhile, keeper B, by a happy coincidence, also faces 17 penalties and saves 4, nearly a 24% success rate. Inevitably, keeper B is repurchased for a lot more than he was sold and the analytics department closes to finance a couple of days worth of his now considerable salary.

So what went wrong?

The answer is probably nothing. Inferior teams out perform superior ones all the time over limited numbers of trials. The World Series springs to mind. If you pair a keeper who has a 25% chance of saving a kick with one that has an inferior 22% chance over 17 trials, the better keeper will save more kicks on just over half of the occasions. About 17% of the time they will save the same number. But in a not insignificant 32% of the paired trials, keeper B will own the bragging rights.

17% of the time keeper A will save 2 or fewer penalties and 40% of the time your new improved keeper won't beat the raw success rate of 8 from 43 posted over the previous eight seasons by the combined individual might of two 22% penalty deniers.

You've succeeded in your brief to improve a player, but that improvement isn't guaranteed to show itself in 17 repetitions. You still need to get a bit lucky to become the hero.

Moneyball, the movie played around with the chronology of some of the trades for dramatic effect, (once again stats take a back seat to the demands of narrative), but I'm assuming the sluggish start to the 2002 Oakland A's season, when an analytical approach was in full swing and the results on the field were not, is more faithful to the actual course of events.

Chance, might not always love a trier.