The Christmas Number One.

Sir Alex Ferguson often made the point that the race to win the Premiership title only really starts around the turn of the year, when about half of the 38 games have been completed and the true front runners have begun to put some distance between themselves the very best of the rest. A couple of media sources took a much more dogmatic reading of the post Christmas table, by pointing out that the last four table toppers on Christmas Day also occupied top spot in May.

This year's Christmas number one was Liverpool, whom clung on to the prized top spot by the finest of margins following Arsenal's dour 0-0 pre-Christmas Monday night draw with Chelsea. Only goal difference separated the top two, affluent Manchester City lay a point further back in third and just another single point further back found both Chelsea and Everton. So the simplistic predictive power of the Christmas number one, alluded to in the Daily Mail is apparently going to be sorely tested this season.

Being top of the league at any time outside of the very early weeks of the season is of course no mean feat. Liverpool are performing consistently at a higher level than they have managed in recent seasons, but the close proximity of a raft of talented challengers, allied to the upcoming transfer window and the ever present threat of injury and suspension, makes their potential crowning in May far from certain.

The flaws present in a naive statistic, such as the ultimate fate of Christmas table toppers is easily demonstrated. 17 matches can see front runners over perform and chasing sides fail to reap their just rewards. League position also tells you nothing about how dominant the leaders are in relation to their challengers and a trend that is quoted over just a four year timescale almost certainly failed to "predict" the league champions reliably in seasons previous to 2009-10.

Inevitably, if we ignore the chosen selective cutoff point and look back one more season to 2008-09, the leaders on Christmas Day that year, coincidentally, also Liverpool, only finished 2nd in May. Arsenal slipped to 3rd in 2007-08 and there then followed a run of three season where the leaders held on to completion. From  2003-04 to the inception of the 20 team Premiership, eight sides failed to maintain top spot, with only Manchester United in 2000-01 honouring this implied trend during the early years of the EPL.

Overall less than half of the sides that are top at Christmas (8/18) are also to in May. Four sides that were placed 2nd went on to lift the title, a similar number of 3rd placed teams and a single 4th placed side coupled with a single team placed 5th or lower before ultimately rising to the top completes the Christmas record.

So despite the inevitable temptation to create certainty and cast iron cause and effect through selectively chosen outcomes, there does appear to be some benefit from enjoying Christmas from the top of the tree, but this needs to be couched in terms of probability and expectation, rather than small sample certainty.

One way to add some context to league positions is to convert the rate at which sides have accumulated points into standard scores, essentially measuring how far from the league average a side's achievements have strayed.

Team.	Year.	Position at Christmas.	Points per Game as a Standard Score.	Final Position.
Liverpool.	2013-14	1	1.45	2
Manchester United.	2012-13	1	2.12	1
Manchester City.	2011-12	1	2.08	1
Manchester United.	2010-11	1	2.17	1
Chelsea.	2009-10	1	1.94	1
Liverpool.	2008-09	1	1.96	2
Arsenal	2007-08	1	1.78	3
Manchester United.	2006-07	1	2.24	1
Chelsea.	2005-06	1	2.30	1
Chelsea.	2004-05	1	2.08	1
Manchester United.	2003-04	1	2.12	3
Arsenal.	2002-03	1	1.75	2
Newcastle United.	2001-02	1	1.79	4
Manchester United.	2000-01	1	2.15	1
Leeds United.	1999-00	1	1.67	3
Aston Villa.	1998-99	1	1.77	6
Manchester United.	1997-98	1	2.10	2
Liverpool.	1996-97	1	1.53	4
Newcastle United.	1995-96	1	2.18	2

The larger the standard score, the more dominant a side has been when measured against the league as a whole. And while an impressive score doesn't guarantee that they will have also left most of their championship challengers well behind, it does make a raft of close pursuers less likely.

Liverpool's 2013-14 figure is the least impressive recorded in the history of a 20 team Premiership and visually from the table shown above, the lower the Christmas standard score achieved by the leaders, the less likely they appear to be able to carry their pre Christmas success into May. Liverpool last topped the mid season table with a similarly poor standard score back in 1996-97, when four teams, all with games in hand lay within six points of them and they ultimately fell back to finish fourth.

A more formal relationship between the standard score of the Christmas leaders and their final finishing position gave Liverpool around a 16% chance of lifting the title based solely on the historical precedents of previous EPL seasons. Well in line with the odds of 6.0 being quoted about them on Christmas night....as Sir Alex says and the bookmakers also recognise "the 2013-14 race starts now".

The Power Of Goals.

It is perhaps an indication of the rarity of goals that of the 100's of photographs I've taken at EPL games, I've only managed to record one goal (a penalty) and one disallowed effort. In comparison I've snapped countless pictures of final third passes, shots at goal and the ubiquitous touches.

Locally, where junior football goal counts climb steadily as the skill differential between sides also increases, I've been more successful at capturing some of the defining moments of a match. Uttoxeter's Oldfields Ground may fall well down in the pecking order of sporting venues, although the turn of the (second last) century pavilion did once host over three rain curtailed days, the 1909 touring Australian cricket team. Nowadays, when it stages local junior football, it is the place for goals and when sides are regularly playing matches will lots of goals, that important measure quickly reaches useful sample sizes.

Goal number six in a nine goal shootout.

As the title of this blog implies, goals and the ability of a side to cumulatively score the lion's share in their matches over a prolonged period of time is a good indicator of how successful a team will be. In addition, goal difference doggedly tracks points accumulated, so the proportion of total goals scored by a side would appear to be the ideal way to express different levels of team talent and likely league success.

However, the relative scarcity of goals in the highest ranks of the professional game, compared to its more lowly cousin, park football, has understandably led to a search for a more numerous match action to act as an improved proxy for this comparatively rare event.

In the opening 10 matches of 2011/12 for example, relegation bound Bolton, scored 10 times by their own efforts with generous opponents chipping in with a further three strikes, took 122 shots, including blocked efforts, completed 737 final third passes and made 1190 final third touches. By comparison, their opponents had scored 27 goals in reply, shot 188 times, completed 970 final third passes and touched the ball 1475 times in that area by the time game totals had reached double figures.

None of the figures gave cause to suppose that the 19th place Bolton occupied after 10 matches was anything but a fair reflection of the abilities they had shown in the opening matches. But did more clues to their future performance lie with their 27% goal share (32% in you included gifts from the opposition) or their 39% share of a tenfold numerically inflated shot count.

100+ shots attempted and around 200 shots conceded is immediately appealing because of the inflated sample size. However, shots and every other secondary statistic often come with unwanted baggage. In particular, as this post shows, such figures are prone to inflated or reduced levels due to the course a game takes. In short, shots, passes in certain pitch areas and touches are highly situational.

The path a team takes when attempting to impose their superiority over an opponent may depend on the order in which the goals arrive in a single game. Following an opening goal, the balance of offensive and defensive actions may shift, as illustrated by the large minority of matches where a winning team may find itself out-shot by a defeated opponent, especially if the winners sprinted to an early lead.

If a side wants to win a match, the only option is to outscore their opponent, but fluctuating scorelines on route to that win may allow a side greater flexibility in how they chose to try to guarantee that win.

So even though shots attempted greatly out number goals as match events, the degree to which a team dominates the other in this secondary statistical category may be highly dependent upon the course the game took. We cannot guarantee that matches for individual sides will follow similar patterns in the future. Therefore, shots, especially over smallish sample sizes may prove to be poorer indicators of future performance than that much rarer prize, the goal, over the same time scale.

To test this, I repeated the auto-correlation post here, but charted the correlation as measured by the r^2 values between the cumulative, proportional share of such game events as shots, touches and goals over the first 10 games of a season and each individual side's points total or goal difference over the subsequent 28 matches.

R^2 for Proportion of Game Events Recorded by Teams in First 10 Games and Points/GD in Subsequent Games. 2011/12.

Proportion of  Game Events For Team in First 10 Games.	r^2 Between Event and Points over Final 28 Games.	r^2 Between Event and Goal Difference over Final 28 Games.
Shots.	0.229	0.339
Shots+Blocked Shots.	0.272	0.374
Touches in Final 3rd.	0.287	0.396
Total Touches.	0.348	0.435
Goals.	0.531	0.546

When the individual proportions of match events accrued by teams from 2011/12 are used to try to predict subsequent performance for those same individual teams, shots languish in the table. Only 23% of the variance in team points over the subsequent 28 matches is explained by the variance in team shots over the first 10 games. Adding blocked efforts improves matters, in 2011/12 at least, as does moving to final 3rd touches and total touches all over the pitch in the first 10 games. However, the proportion of total match goals scored by each individual team, despite their rarity, prove far and away the best predictor of future individual team performance over the remainder of the 2011/12 season.

And the same plot, but this time using the proportion of total match goals scored by each side from the 2011/12 Premiership.

The strong tie between the proportion of shots that a side accrued and the unique situations that transpired within those first 10 matches during 2011/12 appears to seriously weaken their use as a predictive tool.

R^2 for Proportion of Total Goals Scored by Teams in First 10 Games and Points/GD in Subsequent Games. 2012/13 to 2004/05.

Season.	r^2 Between Goals & Points Accrued Over Final 28 Games.	r^2 Between Goals & GD Over Final 28 Games.
2012/13	0.499	0.432
2011/12	0.531	0.546
2010/11	0.400	0.390
2009/10	0.669	0.687
2008/09	0.572	0.579
2007/08	0.557	0.466
2006/07	0.420	0.401
2004/05	0.591	0.505

To see if the power of goals in 2011/12 was a fluke, I have also looked at the strength of the relationship between the proportion of goals each side scored after ten matches and their points haul and goal difference over the final 28 matches for 8 of the last 9 seasons. (2005/06 is omitted simply because the race to ten games was an abnormally spread out affair). In most of the years, the strength of the relationship seen in 2011/12 is confirmed and sometimes bettered.

To be usefully predictive of future performance a statistic should be capable of surviving changing context. But in the short term of the first 10 matches for each side during he 2011/12 season, shots, as a proportion of total shots, appear to be too dependent upon such external forces as game state and current score to pass that test.

World Cup Groups. The Numbers Sometimes Lie.

The 2014 FIFA World Cup becomes much more tangible and real on Friday when the draw for the group stages takes place.  Inevitably some sides will appear to be presented with a relatively easy passage to the later stages and others will fall foul of an imperfect seeding system and find themselves competing in the ritual group of death.

The FIFA rankings determine the seeded teams, continuing the influence they had over the prolonged qualifying stages. And while it is easy to pick flaws in both the rankings themselves and the manner of their use in deciding group make up, they do perform a reasonable job of sorting sides into a recognizable order of merit.

The non competitive nature of many of the friendly matches that contribute towards a side's FIFA ranking figure, along with the seemingly arbitrarily applied weightings to such games, can sometimes undermines the authority of the figures. Additionally, factors that are unique to international football, such as a continental advantage, akin to home field advantage in domestic games, also complicate their use as a predictive tool for future matches. As does the lack of meaningful, collateral form lines between the various FIFA confederations outside of the years of a World Cup.

In view of these issues, it is perhaps surprising that FIFA rankings for two sides can provide a decent indication of the likely match outcome when those sides do meet. Other systems of course exist to provide international team rankings, such as the many elo based ratings and these may be preferred by some.

So whether FIFA is your preferred starting point or not, much of the group analysis that will appear following the draw will rely on the use and interpretation of a rating figure for each of the four teams that will comprise the individual World Cup groups. Simulating the outcomes of the group games by use of a ratings differential based on historical outcomes of similar games is a well recognized way of evaluating the challenges faced by each side before a Brazuca is kicked in anger.

Ratings based analysis of individual matches in football bears a similarity to their use in horse racing, where it has long been realised that translating ratings to likely outcomes doesn't always follow a smooth and regular progression. A single rating figure may describe a weighted evaluation of a side's recent performances, but the expectation in a single upcoming match is likely to merely be centered around that figure. A team may perform better than their rating or they may perform worse. This scenario can be accounted for by randomly selecting figures to be used in simulations from values distributed around that mean.

This approach still requires assumptions to be made that may not accurately reflect reality. The more opportunity a team (or horse) has to truly demonstrate their ability, the more confident we may be that any future performance, independent of a multitude of other factors, such as venue (or going), will be close to that central number. Mature ratings are more likely to have readily identifiable up and downsides. However, in the cases of a lightly exposed horse, in particular, the upside and downside to their recorded rating may be considerably skewed in one direction or another, especially if that horse has shown itself capable of at least being competitive on a major stage.

"Potential for major improvement" may be a horse racing cliche, but a combination of increased opportunity to show their true worth, combined with increasing experience, often throws up cases of underrated talent as measured by traditional ratings.

Relatively few numbers of runs, combined with good, but not great ratings, often characterize horses with capabilities that may far outstrip a tight, normally distributed range of recent performances. Mining the extensive, commercially available racing databases can identify such cases where subsequent performance deviates markedly from the more usual progression. The FIFA ratings provide the footballing equivalent of the racing handicap, but it is less easy to define how "exposed" a team may be.

Horse Ratings Rarely Follow the Straight and Narrow.

One way is to look at the average number of caps gained by the current side. Player turnover can be relatively rapid in international sides. Only four of the starting England players who lost on penalties to Italy at Euro 2010, started the final World Cup qualifying win over Poland at Wembley three years later. So a rating forged over multiple seasons has partly been passed to the inexperienced likes of Andros Townsend, Chris Smalling and Daniel Sturridge. This fluctuating lineup doesn't guarantee improvement, but it may make any conclusion we draw from England's current rating prone to a distorted up or downside compared to a rating belonging to a more mature starting 11.

The number of caps gained by a starting eleven isn't readily available and there is a limit to the amount of data I'm prepared to collect, but below I've listed the average number of caps owned by the starting eleven for all of the European qualifying teams in their most impressive performance during the round of group matches.

Average No. of Caps Owned by Each Starting 11 in their Most Impressive WC Qualifying Game.

Team.	Mean Number of Caps Owned by Starting 11.	Median Number of Caps.
Netherlands.	32	13
England.	40	22
France.	33	26
Switzerland.	34	30
Bosnia & H.	37	32
Italy.	44	41
Belgium.	38	42
Germany.	48	44
Greece.	45	46
Russia.	49	47
Portugal.	58	57
Spain.	71	63
Croatia.	67	70

In comfortably beating playoff bound Romania, the Netherlands did so with a side that produced an impressive performance and did so with an under exposed side, by the standards of current international football. If a similar database existed in football as it does in horse racing, we could perhaps make a more informed prediction about the likely size and shape of any immediate upside for such a side. But in the absence of such data, when simulating the WC chances of the Dutch team, we should consider that their upside may be heavily skewed and inflated compared to their downside, especially if they persist with "proven inexperience".

Teams towards the lower end of the table, such as Spain and Portugal, will be fancied to do well as highly rated European teams, but their range of likely outcomes may not surprise.

Anecdotal evidence of a decidedly non linear progression for under exposed talent, will inevitably be tainted by survivor bias, but the handful of established players I have looked at do show an exponential increase in useful output, such as goals and assists as their cap count climbed into the 30's and 40's.

Equally non random in selection and therefore probably inadmissible as anything more than an interesting nugget of information, is the average cap count of sides that have shown performances at major tournaments that belied their more modest pre-tournament ratings.

2004 Euro champions and pre-tournament 250/1 outsiders, Greece, started the final with a side which had an average of just 30 caps per player. Senegal kick started their 2002 World Cup campaign with a 20 cap a man win over France. Caps largely gained in a partly isolated confederation. Republic of Ireland defeated Italy in a WC with a median of 26 caps and Bulgaria's run to the 1994 WC quarter finals was achieved with a 32 cap average.

Prediction can draw from many pots and while the potential for ratings to progress in a non linear manner for teams about which our information may be limited (even if that side's name is well known) is unlikely to be an over riding factor, it will add some degree of uncertainty. So an established higher ranked side that welcomes the likes of Bosnia and Hertzegovina should probably heed the example of an upwardly skewed Bulgaria from 1994.

For those who can't wait for Friday's draw, a nice primer can be found here at http://21stclub.footballfanalytics.co.uk

Free Kicks verses Shots From Open Play.

The spotlight continues to fall on the effectiveness of shooting from distance, but how does much of an advantage does a set piece shot from outside the box give to the shooter. In this guest post I try to quantify how much closer to goal you need to be before a shot from open play becomes roughly equivalent to a shot during regular play

The Scoreline Isn't Everything.

One of the major problems when attempting to add context to football statistics is that many of the recorded events are highly situational. It is relatively easy to produce a chain of events from touches to passes to key passes through to shots and ultimately onto goals. But the urgency and frequency at which a side attempts these actions and the commitment in effort and manpower that their opponents put into preventing a successful conclusion, depends greatly on the state of the game. Current score, time remaining and the relative abilities and expectation of each side are the three most obvious indicators of the current game state.

It is hugely tempting to collect every positive action performed on the field of play and relate those numbers to match outcome and often the results appear to confirm a connection. Shooting at goal is to be preferred to having to stop a similar effort from your opponent. Therefore, it isn't surprising to see that the winning teams in the EPL during 2011/12 out-shot their opponents by an average of 3.5 shots per match.

However, averages almost always fail to capture the full nuance of a situation. Although 180 of the 2011/12  matches where there was an outright winner saw the winning side out-shoot their opponent, a not insignificant 110 matches saw the loser out-shot the victor. Nearly 40% of result games went to the loser or tie in terms of shots.

Goalscoring, in a low event sport such as football contains a lot of random variation and out and out luck in a single match. If Stoke had filled in the corners of the south stand as proposed in 2011, thus creating a windbreak to the gale that habitually blows towards the Boothen End, if Shawcross hadn't won the toss against Southampton and spurned tradition by turning the sides around, if Southampton's defenders had heeded the mantra of "never let the ball bounce", then Begovic might not have scored the fastest goal by a keeper on a Saturday in November.

However you define it, luck or randomness may play a part in an out-shot side winning the game, but other factors, such as the tactical approach of each side under the influence of the underlying game state is an obvious additional candidate.

Many sports that incorporate game state analysis have either greater numbers of scoring events than football, making a draw at full time much less likely or positively balk at the thought of a draw and actively legislate against such an outcome by incorporating overtime. Therefore, the draw is firstly, uncommon in these sports at the end of regulation and often eliminated by extra playing time.

This isn't the case in football, the relative low scoring nature of the game results in around half of the playing time being spent with the scores level. Secondly, the draw at full time does represent a safe haven for a side that is content with a point. So, a tied scoreline in football is much more likely to still see widely differing intentions displayed by a side and their opponents compared to other sports, because it is a scoreline that can persist at the final whistle.

In dealing with game states, therefore, we must address the issue of the tied scoreline. Once the massive rump of time spent level is given a more team specific outlook, we can begin to see if the out-shot teams took advantage of a fair wind or actively adjusted to a renewed and diversified challenge from a ultimately defeated rival.

Above are the 290 games in 2011/12 that ended with a winning team, showing the influence of average game state on the shot differential experienced by the winning side. Game state is customized stat, so as a general indicator, zero along the horizontal axis indicates a side that was consistently behind expectations for large parts of the match. For example, a good side that couldn't break down weaker opponents until very late in the game or a side that achieved a "come from behind" victory. Values of 1.5 or more comprise sides that were generally well in charge of the game from a relatively early stage or weaker sides that held superior opponents and won late on.

Correlations using single matches as a data point are notoriously weak, but even if a few outlying games may have pulled the line of best fit downwards, the correlation appears clear. The longer an ultimate winner was behind or a stronger team was held by a weaker rival (denoted by the decreasing size of the number along the horizontal axis), the more likely they were to have out-shot their opponents on their way to victory.

On the other side of the game state mirror, an acceptable scoreline for the majority of the game, denoted by a positive value along the horizontal axis appears to indicate that in 2011/12, either by design or through necessity (or a combination of the two), a majority of such sides won despite being out-shot in the match. A side sitting on a lead and defending shots or an underdog defending from the outset and catching their more illustrious opponent on an effective, but rare counter, for example.

Rather than averaging out the detail and manner of victory, by including average game state, the typical manner and frequency of how EPL sides achieved their wins when faced with particular in game situations starts to become more apparent.

Clearances, for example follow the opposite trend. A side that either led relatively early or held a superior rival, before winning tended to account for the majority of the clearances seen in the match.

The opportunity to make clearances depends partly on the willingness of the opponent to proved such opportunities. In short, the kind of stats a side records is greatly dependent upon the average game states they experience either during a single game, a run of matches or an entire season.

The full-time result tells you very little about how a side arrived at such a favourable outcome. A team may lead from the first minute or overturn an early deficit with a couple of late strikes. Sides that win from prolonged bouts of poor in running scorelines (by their standards), also tend to win the majority of the corners, attempt more aerial crosses, make more forward passes and accrue more key passes. They also take the majority of longer range shots and see more efforts blocked than their immediate opponents. These trends are reversed, in general for teams that win matches where they were satisfied with their position for large parts of the game. Paradoxically therefore, a play-maker whose side creates a couple of early goals is less likely to accrue a large amount of key passes in such a match because the team priority may have switched to defence.

An approach that adds game context to the stats could be used to look at the kind of actions individual sides take to achieve an acceptable result, this time compared to their own usual average. Swansea are known for their possession based passing style, but their commitment to passing the ball in the final 3rd was strongly dependent upon game state as shown in all their matches from 2011/12.

For example, they traveled to Anfield as big outsiders with just a 10% chance of winning the game at kick off. Therefore, they were perfectly happy for the game to remain scoreless, because with every passing goalless minute, their points expectation from the game edged upwards from it's paltry beginnings towards the reality of a single point gained. So they rarely ventured into Liverpool's final third when compared to their usual way of playing. And they similarly reproduced these lowly final 3rd figures when they enjoyed the benefit of any early goal during their much less onerous trip to Villa Park. By contrast the Swans attempted almost 100 more final third passes than was usual for them when they trailed early at home to Newcastle.

Talent may dictate a side's ability to attempt final third passes, but game state is also a powerful driver of their desire and need to exhibit that talent.

You were more likely to see Swansea passing in the final 3rd when they were doing poorly 2011/12.

As a further example, Spurs under Redknapp became a much more aerially orientated attack in poor game states. Once again there is a strong trend, this time for more aerial crosses to be played into the box as they tried to turn around the game state. Spurs' red card assisted romp against Liverpool was their most comfortable game during 2011/12, an early Modric goal was quickly followed by Adam's dismissal and the number of crosses into the box throughout the game was well below their seasonal average. Unlike the Sunday game in mid December at Stoke, where a couple of first half Etherington goals had them chasing the comeback for much of the game and they bombarded the box with well above numbers of crosses.

Plots like these may more clearly illustrate what you can/could expect from a team in certain circumstances. If you got the upperhand on Swansea in 2011/12 you could expect to have to deal with more passes in more dangerous areas, while Spurs increased the amount of aerial balls they played into the box. Just as importantly, Swansea's output of long balls for instance, remained relatively constant regardless of game state, so defenders were unlikely to find themselves chasing back into the corners with anymore frequency if they were defending a good game position against Swansea.

Equally, the individual player statistics are strongly tied to the game states and the needs of their side in that game state. An apparent decline or upswing in an individual raw stats, such as headed clearances, assists or final 3rd pass attempts could have as much to do with the ebb and flow within recent matches, as it has any real decline or improvement from the player concerned.

In short, quantifying a player or team's abilities is often tied to fully appreciating their need to show off that ability to the full.

Pulisball. What to Expect at Palace.

Crystal Palace's long search for a new manager finally ended on Friday night, when one of the early front runners, Tony Pulis put pen to paper. Any lingering doubts about Pulis' at times ugly style of play was ultimately trumped by a managerial record that has never seen him relegated. Sacked yes, but not relegated.

Pulisball gained ridicule, contempt, grudging respect (and that was just from his own fans), but just enough points to guarantee perennial survival. With the unique ingredient provided by Delap's long throw it did prove an over achieving way of retaining Premiership status for a largely Championship quality team.

Long throws, long balls and set piece goals were the most visible face of a Pulis led Stoke, but it wasn't the only requirement if 40 points were to be achieved year on year. It also needed a defence. A side that relies on set pieces to provide a higher that average proportion of their goals are usually limited attacking sides. The correlation isn't overwhelming, but the trend exists for sides that score proportionally more goals from set plays, to also score relatively fewer goals over a season compared to sides that have a more balanced attacking approach.

Unsurprisingly, an inability to score goals also goes hand in hand with lower end of season points totals and the strength of the correlation is very strong. The r^2 values for success rate (a proxy for points) and goals scored since the start of the 2008 season is 0.77, so 77% of the variance in success rate comes about as a product of the variance in goals scored.

A slightly weaker correlation, but of a similar order of magnitude for r^2 also exists between goals allowed and seasonal success, which leads to the inevitable conclusion that goals scored and goals conceded by sides are themselves not independent. The negative co dependency between the number of goals a side scores and the number they concede is plotted below. The more goals a side scores in a season, the fewer they will tend to concede.

This correlation is probably driven, firstly by a general difference in relative ability between sides. Better sides are simply more adept at playing all aspects of football. Such sides score more and concede less, but there may also be a tactical aspect. It is difficult to score yourself when your opponents has the ball in your final third. In some cases, attack may also be an effective form of defence.

So the way a side either chooses or it forced to try to accumulate goals in the EPL can have an effect on the amount of goals you score. Stoke's reliance on set play goals was likely to result in few goals being scored. The poor goal totals would in turn lead to unimpressive seasonal success rates and that in turn often led to relegation at worst and bottom half finishes as the norm.

However, Stoke ended four of the five seasons under Pulis with higher seasonal success rates compared to the line of best fit for goals scored, denoted by the red points in the second plot. A clue to why this might have occurred can be seen in the plot above. In all years under Pulis, they allowed fewer goals than expected for a side that scored at the rate they did. Again Stoke's five seasons under Pulis are represented by the five red points in the plot above. The brutal simplicity of creating a set piece chance just by winning a throw inside the opponents half, often meant that Stoke didn't have to commit many players forward to do so. The tactical aspect of Stoke's chance creation left plenty of defensive resources intact to keep the score down in games that mattered.

We can't dismiss the possibility that Stoke's relative defensive excellence was just merely down to chance. If we look at enough sides, some are going to appear continually better than expected, but by pure chance. However, anyone who has watched Stoke, especially in the early EPL seasons can't have failed to notice the overtly defensive stance they took both at home and away from the Potteries.

Although they were correctly classified as a side that relied on set pieces for the majority of their goals, it would be a mistake to look at this aspect of their play in isolation. A team is a sum of their parts and the general outlook for set play sides is rather bleak. They are more likely to be relegated than a side that can score proportionally more often from open play. They finish in the bottom half of the table with greater frequency and they are highly unlikely to be capable of challenging for even a Europa spot.

Pulis' Stoke ticked most of the statistical boxes for a limited, set piece reliant side. Palace already score nearly half of their goals from that source, although raw numbers are small, so percentages will fluctuate. But the real wrinkle that Pulis introduced that propelled an approach with little upside or room for error to a relatively comfortable way of retaining your top flight existence, was his insistence on a defensive team effort.

Palace are already a side that find scoring difficult, so their limitations already provide half of the ingredients for Pulisball to make a Premiership return. To complete the formula, Pulis' secret sauce will be liberally applied to the defence in an effort to maintain his proud record.... but this time he'll have to do it already 12 games into a season and without a defence enhancing wildcard such as Rory Delap.

World Cup Qualification. The Talented and the Lucky.

The extended prequel to World Cup 2014, namely the qualification process finally closed for UEFA following the second legs of  the playoff matches on Tuesday night. Spaces are limited, so it is inevitable that there will be some notable absentees when the final draw takes place in Rio on December 6th. Apparently, parts of the media are inconsolable because the 20th World Cup will take place without Ibrahimovic following  Portugal's elimination of Sweden.

Seeding for the qualification groups was decided by the FIFA rankings on July 2011 and the majority of the top seeds were recognizable as the current, leading national teams. France provided the biggest point of interest, by falling into pot 2, although they had been outside the top 9 position that guaranteed a more favourable draw for over a year. While France failed to recover ground lost at a dismal WC 2010, sides such as Norway reeled off a consistent string of narrow wins in both friendlies and higher weighted competitive Euro 2012 qualifying matches to retain their place in pot 1.

As Simon Gleave points out here, tournament formats and the vagaries of chance can play a huge role in deciding major sporting events. And by falling into pot 2, the chances of France drawing a group containing a previous World Cup winner or the Netherlands was more likely than not. By being paired with Spain in the five team Group I, France weren't quite in the group of death, but they were odds on to need the playoffs to progress. So more the group of maximum inconvenience and by taking the playoff route to the finals, they were unlikely be blessed with a comfortable passage if they did make the finals in Brazil.

Five of the nine top seeds qualified as group winners, three made it through to the playoffs and only Norway belied their ranking by slipping out of the competition at the earliest possible stage. France, as expected trailed in behind Spain and Russia (from pot 2), with a resurgent Switzerland and Belgium (pot 3) completed the list of nine group winners. So class, as measured by recent FIFA rankings appeared to shine through with reasonable clarity.

If luck, in the most purest sense, decides the make up of each qualifying group, where danger occasionally lurks in pot's 2 or 3, as a new generation of starlets sweeps countries to levels beyond their recent station, it is small sample variation that can derail teams once the groups are fixed. Of the two heavyweights in Group I, Spain would be confident of confirming their supremacy over a France team that is ranked over a dozen places below them, but the eight match format afforded the current holders just two shots at their most likely challengers for automatic qualification. A strong team, possibly out of place at the wrong time, can cause unwelcome early challenges, even for the top seeded side.

The outcome of each of the nine UEFA qualifying groups can be viewed as a random, weighed draw comprised of the outcomes of each match played in the group once the relative abilities of each side is accounted for. Reducing World Cup qualification to a sterile number crunch lacks all of the tension of a wet night in Warsaw, but it does help to add context to the perceived achievements and failings that we have seen over the last year and a half. A side can perform well above expectations, but that may be partly due to improvement and partly to the randomness at which results cluster in small sample sizes. Malta do have a shot at defeating Italy, but it is a very longshot.

France could do nothing to influence their chances of avoiding Spain or a similarly talented seed, once their own performance/luck combination over 2010 had anchored them outside the best nine European sides, but raw ability and 94th minute equalizers, vied with random chance to decide the individual outcomes of the subsequent matches. The destination of the group honours was a combination of talent and luck, where the actual placings that transpired were just one of many possible combinations that could have occurred from that heady mix.

Below I've simulated the outcomes of 1,000's of iterations for all nine groups, using the bookmakers odds as a proxy for team talent on the day and the constant repetition to host the role of randomness.

The progression of a young Belgium side, as measured by their gradual elevation in the eyes of the oddsmakers over the qualifying period, gave them around a 50% chance of topping the group in the simulations. Their actual points total of 26 was above their simulated median score of 20. The FIFA rankings took this over achievement at face value and propelled them up to third in UEFA from a starting placing of 22nd at the start of the qualifying process.

..............................................................................................................................

Denmark could consider themselves unlucky to miss out on a play off berth as the ninth best runner up. The tie breaker saw the lowest points scoring runner up eliminated once the record against the group's worst side was expunged to account for Group I only having 5 teams. However, the average points total achieved in all simulations of Group B by the runner up was 19.4 with a standard deviation of nearly 2 points and Denmark's actual tally of 16 was therefore, nearly 2 standard deviations below the average, making them comfortably the most under performing second placed team over all nine groups.

........................................................................................................................

A comfortable qualification for Germany, where the group make up provided them with little danger from three inferior challengers who were likely to take points from each other. In gaining 28 and 20 points respectively, Germany and Sweden each gained a couple of points more than their median points totals across the simulations.

..........................................................................................................................

Romania's finishing position of 2nd appeared impressive because it got them into the playoffs. But it was only one spot above their most likely finishing spot of 3rd and they gained just two points more than their median in all simulations. Without an injury time equaliser in Budapest, Hungary could have swapped places with them at the death. Three teams were close together trailing runaway winners the Netherlands and Romania finished qualifying in a still relatively lowly 19th place in FIFA.
..................................................................................................................................

A case of Iceland performing well above their initial standing as a team drawn from the lowest pot or a rare occasion of a lowly rated side collecting a fortuitous sequence of unlikely results to propel them to unsustainably heady heights in the short term? They were rated by the oddsmakers throughout the campaign as the second worst side in the group, with a most likely finishing spot of 5th and around a 10% chance of snatching second place in simulations.

A very tight group on the field, with 18 of the 30 matches being either drawn or won by a single goal. Both winners Switzerland and 2nd placed Iceland outperformed their odds based median points total by 5 points, indicating a miss calculation by the bookmakers or fortuitous, short term set of results that saw, in the case of Iceland, a less fancied 10/1 shot beat two more deserving talents in Slovenia and Norway?
 ...............................................................................................................................

Russia topping the group should hardly be considered a surprise as they did so in over 30% of the simulations. Portugal's median points total was 24 (which they would have got had they not allowed Israel a late equaliser in Lisbon) and Russia's was 22, which was their actual total. The perception of Portugal shouldn't change because they required the playoffs to progress to Brazil.

................................................................................................................................

.............................................................................................................................

England's qualification campaign has already been covered in detail here.

.................................................................................................................................

France were unlikely to topple Spain as winners of the group, although they did so in nearly 20% of the simulations.They ultimately took their most likely road to Brazil, as runners up and then by way of the playoffs, although being France, the latter was not achieved without considerable drama and uncertainty.

With a couple of exceptions, the 17 UEFA representatives could broadly be predicted before a qualifying ball was kicked. The seeding process, coupled with the large talent gap between the best and the worst of European national football, invariably gives half of the sides in each group very little chance of scooping the top slot. But the truncated nature of the qualifying process does give middle ranking teams the opportunity to jump a place or two above their natural long term station. Rather like merely good sides occupying the elevated Champions League placings in the EPL after a dozen matches.

Sides that produce a couple of atypically good or bad results, especially in highly weighted matches, such as WC and Euro qualifiers can fall prey to FIFA's blunt rating system, where results are understandably considered interchangeable with true ability, with no room for random chance. 

Belgium's relatively poor WC and Euro results prior to July 2011, led to their placing in pot 3 for the 2014 draw, but their (possibly) small sample sized over performance in group A should see them set fair for upcoming future draws, baring a major meltdown in Brazil. 

Neither a placing in pot 3 in July 2011 nor a top three rating now, truly reflects Belgium's actual ability, anymore than France possibly deserve to be struggling to clamber back into the elite on a diet of lowly rated friendlies and two fewer qualifying matches. But the process has done a decent job of producing the cream of European team talent for Brazil 2014 (even if some individual players will miss out). 

At the very least a ratings system based on ranking points exchange, with too few matches for lucky streaks to be fully eradicated from the system before they are used to shape major competitions, merely adds to the uncertainty and excitement, both for the team that is out of place and the sides that have the unwelcome task of taking them on early in major tournaments.   

Old School Meets New School.

Evaluating the abilities of a football team can take many forms. From a purely numerical analysis, where their strengths and weaknesses are expressed as lines on a spreadsheet and future performance is cited in probability, to gut instinct formed through actually watching a side interact and play the sport in the flesh, complemented with a knowledge of their recent player acquisitions and departures.

Neither preference is guaranteed to capture the complete nuance of each side's true worth. So can an integrated approach that borrows from both sides of the divide provide a worthwhile collaboration?

In this guest post I try to combine the strengths of both approaches.

Conversion Percentage Inside the Box. Mind The Gap.

So I used 2011/12 seasonal shot data to look at all goal attempts to see if the rate of scoring was merely the result of random variation around a common mean or if their was likely to be a genuine difference between the best and the worst sides in terms of conversion efficiency.

I split the sample between shots inside the box and shots from everywhere else (including shots and goals scored from you own box.....Tim Howard, take a bow), in an attempt to maintain a decent sample size, but to also smooth out any positional shooting preferences among the teams.

The method sees if the distribution of goals scored by each side, given their relative shot totals, is substantially different from the range you may expect if every side is equally talented at converting similar types of chances.

I deliberately left in penalty kicks for shots from inside the box, because I wanted to see how the conclusions changed as certain types of readily identifiable and unevenly distributed shots were removed from the sample.

So the first run included every goal attempt from inside the box. The distribution of goals scored by the twenty sides did appear to differ markedly from the spread you might expect to see if Manchester City had had 450+ attempts and Stoke had 230+ with every other team contained somewhere between those shooting extremes, but all sides had striking talent that was equally adept at converting the chances that fell to them.

Next, I took out penalties, which tend over time to be given to those that do the most attacking and present a significantly higher chance of scoring that other, open play opportunities from inside the box.

Virtually the same result.

Compared to the sample with penalties, we do edge very slightly closer to a distribution of actual goals in 2011/12 that better resembles a random draw from an equally talented 20 team strike force being presented with varying numbers of opportunities. But we still can very safely say that our actual spread of goals from 2011/12 doesn't resemble a lucky dip with a universal  strike rate. About 2% of teams manged at least 60 goals from the distribution of shots actually attempted by teams during 2011/12 in simulations using a universal, average conversion rate. In reality during 2011/12, three teams out of 20 managed to surpass this target.

So I then took out headers.

Overall, headers present a poorer likelihood of success compared to shots and in 2011/12 headers comprised a heft chunk of the total goalmouth attempts for some teams, (no prizes for guessing Stoke).

With headers culled from the data, the difference between the actual distribution and the range you might expect from one drawn from a group of equally lethal strikes, plummeted to within touching distance of each other.

It is just one season, but once you take out penalties and headers, then the number of goals scored by all other means inside the box, still differs from what might expect to occur by random chance where there is no difference in the finishing talents of each forward line, but the gap is small....Very small.

Here's the regressed conversion rates for shots (with the feet) inside the box for sides from 2011/12 suggested by the above analysis.

EPL Side from 2011/12.	Regressed Conversion Rate for Foot Shots Inside the Box %.
Newcastle.	14.9
Arsenal.	14.8
Chelsea.	14.8
Manchester United.	14.7
Norwich.	14.6
Tottenham.	14.4
Wolves.	14.3
Manchester City.	14.2
Aston Villa.	14.1
QPR.	14.1
Stoke.	14.0
Sunderland.	14.0
Bolton.	14.0
Everton.	13.9
Blackburn.	13.9
Swansea.	13.9
Fulham.	13.8
WBA.	13.8
Wigan.	13.2
Liverpool.	13.1

To put these figures into perspective, the difference in conversion rates between top and bottom, given an average number of shots from inside the box (240) accounts for 4 extra goals and that represents about three league points.

If we ignore Newcastle at the top and Liverpool at the bottom, both of whom broke most statistical models during 2011/12, the actual top five from 2011/12 are to be found in the top seven for converting shots inside the box. And relegated Bolton and Blackburn are at least in the bottom half. So the ranking is fairly consistent with league position in May.

By attempting to produce a reasonably sized, homogeneous sample size, the gap between the degree by which real life conversion rates fall, at first slightly and then precipitously towards a random draw is seen. There's still evidence for a talent divide at the very top, but it is narrowing, throwing the importance of shot volume into the spotlight.

Ten years worth of shot data would be nice to see which side of the line shot conversion rates finally settle on!

Begovic Scores! We Draw!

All of the best photographers appear to have a sixth sense about when a worthwhile picture opportunity is about to arise. It was therefore no surprise that my camera had just been returned to my rucksack when Asmir Begovic (a goalkeeper) scored for Stoke (after 13 seconds) against Southampton at the Britannia Stadium on Saturday.

There were clues available that would have indicated that Mark Hughes had a plan. All teams have a preferred end to attack in the second half and there appears to be a tacit agreement between captains that if the visitor wins the coin toss, he will take the kick off, rather than earn the early game wrath of the home supporters by turning the teams around. Therefore, the undercurrent of discontent that accompanied the sides swapping ends after the coin toss on Saturday, turned to slight bemusement as Southampton lined up to take the kickoff.

Stoke had turned themselves around.

The geography of the Britannia Stadium makes it an ideal site for endurance training. The prevailing wind regularly blows in from Trentham, funnels itself through the two open corners at the south end of the ground and then struggles to exit at the single open corner to the left of the Boothen End, tipping a hat to the statue depicting the three ages of Sir Stan as it carries on towards the city.

Stoke now have a chequered history with near gale force winds. In the distant past it has removed the roof of the Butler Street Stand (along with our best striker to pay for the uninsured infrastructure), but the worst it has managed at the Britannia was the late postponement of a game against WBA. On Saturday it provided Stoke with the opening goal by way of partial payback.

In the subsequent press conference, Mark Hughes acknowledged the deliberate decision to play with the wind during the first half, citing the importance of scoring first, which is encouraging. However, it is (hopefully) unlikely that his keeper was considered the most likely scorer. Hughes' apparent encouragement for his sides to shoot from distance, so vividly demonstrated at QPR, must have some bounds.

Following the kick off, Southampton chose to attack Stoke with a series of intricate passes. This quickly broke down, the ball was rolled back to Begovic, who launched a wind assisted punt goalwards. Both Southampton defenders chose to ignore the golden rule of defending, namely "never, ever let the ball bounce" and Boruc was left embarrassed by a slick bounce on the wet surface.

Begovic is the fifth keeper to score in the Premiership and invariably such goals require additional help or unusual circumstances. Tim Howard's effort against Bolton was a replica of Begovic's goal, but Peter Schmeichel, when playing for Villa opened the goalkeeping Premeiership goal tally from the more advanced position of the opposing penalty area. So modelling the likelihood of a keeper netting is going to be hugely situational.

We may have more luck trying to quantify the quickfire timing of the goal.

Stoke v Southampton, two minutes 13 secs from history being made (not shown).

The chance of a goal being scored increases slowly, but inexorably as time elapses as caution and fitness, gives way to adventure and fatigue. Your chances of seeing a goal during the sixty seconds that comprise the 8th minute is only 80% of your chances of seeing one in the 80th.

However, three "sixty second" intervals are completely atypical compared to this gradual cranking up of goal expectation. The 45th is the second most goal laden "minute" followed by the 90th and the reasons are clear. Injury time extends both minutes well beyond sixty seconds leading to two big spikes. Therefore these increased rates are purely artificially created by traditional timing considerations. The second half starts with the first second of the 46th minute, even if the first half stretched well beyond 45 minutes of actual time.

The barren blip that comes with the first minute, by contrast is entirely real. If you chose any sixty second period in the first ten minutes, you are likely to see around 0.7 to 0.8 percent of the total goals scored during the match, on average. But if you plump for the first 60 seconds of a match, you will be lucky to see much more than half of the typical early minutes goal percentage.

Again, the reasons are fairly plain to see. Every game starts with a kickoff. The ball is about as far from either goal as it can possibly be and all eleven players are positioned between the ball and their goal and this formation of maximum protection for each goalmouth is guaranteed to occur during the first minute in every match. Hence the scoring is not only at its lowest because of the usual ebb of intent and desire to score, it is atypically lower because of the requirement to start the game with a kickoff.

So, pulling all the information together, around 0.5% of goals come in the first minute, the first 13 seconds are likely to see less than a pro rata division of this goal expectation because of the guaranteed safe starting position for the ball. A back of the envelop calculation using these figures and the average scoring expectation over the history of the Premiership gives an average goal expectation for the first 13 seconds of a Premiership match of around 0.0015 of a goal. The chances of scoring twice or more in 13 seconds is impossible, therefore a goal in the opening 13 seconds should, under these informed gu-estimations happen around once every 666 matches.

So, unlikely as Begovic's goal was purely from a timing perspective on that particular Saturday afternoon, we should expect to have seen around a dozen such goals scored before the 14th second has elapsed over the 8,326 game history of the Premiership.

Begovic's strike was the sixth such effort, so maybe the primeval order at kickoff takes slightly longer to descend into chaotic normality than I accounted for or Premiership audiences have just been slightly unlucky.

If you have to miss a minute of a match and you want to reduce your chances of missing a goal, chose the first minute, (although you may be really unlucky a miss club history in the making), but at least on Saturday any tardy spectator got to see a match featuring two keepers whom had both scored a career goal, (although Boruc's strike came from the altogether more likely source of the penalty spot).

Finishing and Hitting the Target in the MLS.

The first attempt I made at looking beyond the commonly available football stats of the day, namely goals, used shot and save data from the MLS. The quality of play may not have quite matched that seen in the Premiership of the day, but the amount of data available far outstripped that that was commonly available to the UK newsgroups that preceded blogging.

Attempting to tease the luck from the talent in the shot saving percentages seen in the likes of Kevin Hartman, Tony Meola, Joe Cannon and Tim Howard was a lot easier than trying to sensibly argue who England's current stopper should be. So a belated h/t to Big Soccer, where around half a dozen stat enthusiasts hung out in the dim distant past.

A recent tweet from the influential Steve Fenn, a must follow at @SoccerStatHunt, reminded me of the excellent work that is being done by the guys at http://americansocceranalysis.wordpress.com/ notably, Harrison Crow (@Harrison_Crow). They are collecting and also sharing shot data in the current MLS. So a major h/t to them, the first attribute is fairly common, but the second is extremely rare and most welcome!

The availability of data is the major bottleneck is blog based analysis. Methodologies are fairly standard, but weight and credence to any conclusions only comes with increased sample size. It is fairly easy to develop a novel methodology, but the limited data can still make you look dumb.

Back in the day, shot attempts and outcome was the limit of the data, but the the volume of the data, stretching over seasons and, in the case of keepers, their longevity, still made analysis possible, if with a slightly wider error bar attached. Increased shot volume, it was hoped would even out issues of shot and chance quality, that did not exist to such as degree in either the controlled pitcher/batter contest in baseball or the more restricted playing area of hockey.

 I don't have an MLS photo. Instead here's Clint Dempsey celebrating Sounders' Interest (and a Goal against Stoke).

Nowadays, the still flawed gold standard from blogging shot analysis is data with x,y co ordinates, but often devoid of even the tiniest hint of defensive pressure, except in the most dedicated of collectors. Which is why the MLS data dump at American Soccer Analysis is so welcome. It improves greatly on shot data of the past by partly bridging the gap to professionally collected and protected data with the subdivision of shots into zones. Usually, slicing and dicing sample size leads to noise and over fitting, but ASA's venture may sacrifice sample size, but greatly increase uniformity of events within those smaller samples.

Applying one of my shooting analysis methods to ASA's improved data was therefore both sensible and a nostalgic treat. Broadly, this method assumes that shot outcome is common to each MLS team and centered around the league average. Any apparent deviation in shot accuracy percentage or conversion (and there is bound to be some) is going to be down to random variation and a talent gap in performing these tasks between sides. Quality of opportunity is hopefully controlled by ASA's use of shooting zones. So if we see a wider range of outcomes in the attempts each side made, compared to a random draw using league averages, we can possibly conclude that random variation isn't the only factor at work in deciding the shooting pecking order.

The sectors used along with the data are all available at ASA's site, so I urge everyone to seek it out there, but for partial clarity the sector descriptions are sector's 1,2,4 and 5 are central to the goal and more distant with increasing number and sector 3 is wide within the area and sector 6 is wide to the flanks.

I have taken shooting data from the site for every game played by every side in 2013 and compared the spread in accuracy (in terms of shots that require a save), conversion rates (goals scored) and the undesirable ability to see shots blocked that was recorded by each side against the type of spread expected from those shot numbers if team talent was universally the same in each sector and variation of outcome was purely luck driven.

Do Sector Outcomes Suggest Factors Other Than Random Variation are at Play in the MLS?

Sector taken from American Soccer Analysis Site.	Does Accuracy Deviate from Random?	Does Conversion Rate Deviate from Random?	Does Avoiding Blocked Shots Deviate from Random?
1	Yes	Yes	Barely.
2	Strongly	V Strongly	Random.
3	Yes	Yes	Random
4	Yes	Random	Yes
5	Random	Random	Random
6	Yes	Random	Random

The results are tabulated above. Using shot data from 2013, there does appear to be some evidence that team conversion rates may show a talent differential when strikers are closest to goal. As attempts move further from goal (in the case of zones 4 and 5) and much wider out to the flanks (in case 6), that differential appears to disappear and outcomes become consistent with the average overall conversion rate for the  MLS. In short, skill may exist inside the box, but outside you're hoping to get lucky....in the MLS at least.

A talent for greater (or lesser) shooting accuracy as measured by an attempt requiring a save appears to survive to greater distances and angles or it may show a tactical approach whereby a side is required to "make the keeper work" in expectation of a follow up rebound....Or everything may be the result of insufficient detail contained in the current, admirable data.

I know very little about the specifics of the current MLS, other than Dallas produce technical adept players and Seattle has the coolest kit, but others may make sense of Philly being the best opportunity corrected finishers in sector 1( closet to the goal) and Portland the most efficient in sector 2.

Random variation is ever present in the data, but recourse to this concept as a catch all when a side over or under performs against the league norm, may be less (or more) than fair to player and coaches alike, especially in the absence of any evidence that the talent gap at the very top level has disappeared completely.

To reiterate here's the link to American Soccer Analysis.