Quantifying Which Team Is Happier With The Current Scoreline.

One vital ingredient you need to add to your analysis of a football match is the current score context. Teams adjust their playing style depending upon whether they are comfortably ahead or well behind, generally adopting a less attacking and more defensive stance in the former case and vice versa in the latter. These micro shifts in emphasis can in turn impact upon in game match events such as shots and saves. For example a team chasing a game may produce more goal attempts, but they are often from further out, instigated by a wider pool of players and subject to more defensive pressure. This kind a subtle difference in shot or save quality is often lost in aggregated data, but visible in more granular batches.

The general game state for each team in a match is fairly easy to qualify if one team has an advantage on the scoreboard, but actually quantifying these positions as well as the numerous occasions when the match is stalemated requires more effort. Arsenal and Sunderland were level after a hour at the Emirates on Saturday and the visitors from the North East would have been much more comfortable with the scoreline than were their hosts. The real question is how much happier were Sunderland ?

One way to add context to such games is to calculate the combined win and draw expectancies for each team in running and track the change in these values compared to where they stood at kick off.

Full Time Score. 0-0.

Arsenal were unsurprisingly large pregame favourites to beat Sunderland, they were in the region of 70% likely to win the game and a shade under 20% to draw it. Combined, these two figures suggest that Arsenal's long term success rate (wins + half draws divided by games played) from such a match up would average just under 0.8 making Sunderland's long term success rate just over 0.2. 

As the game progresses, goal expectacies for each team also decline and at the still stalemated hour mark Arsenal's win probability would have been around the region of 0.5 and their predicted long term success rate for this game position would have declined from just under 0.8 at kick off to just under 0.7 now. In raw terms The Gunners had lost 0.1 of their pre match predicted success rate by their failure to score. In running success rates of rivals are intimately entwined, they must always total one, so Sunderland had seen their pre game success rate climb by the same amount. 

In this situation using game win a draw probabilities allows the 60th minute to be contextualized for both Arsenal and Sunderland. In non numerical terms, Sunderland are very happy and Arsenal aren't and this will partly dictate how each team approaches the final third of the match.

A further example from yesterday illustrates the effect of  goals in a more evenly matched game such as Newcastle's entertaining of Tottenham.

Scorers.

1-0, D Ba, 55'

1-1, Defoe, 76'

2-1, H B Arfa (pen), 81'

Newcastle are probably inferior to Tottenham at the moment, but home advantage gave them a very slight match day edge. In contrast to the Arsenal/Sunderland game, while the game remained or became level, both teams were probably fairly happy with the scoreline. This is denoted by the closeness of each team's plots to the neutral zero line and tactical approaches from both sides are likely to mirror those used league wide in evenly matched contests. Tottenham found themselves twice in losing positions, one of which they managed to salvage and league wide, teams attempt this rescue operation by committing more to attack than defence. 

Utilizing time and score specific success rate movements such as these into more granular shot data will prevent erroneous conclusions regarding team shot conversion rates being formed and being incorporated into aggregated totals. Teams, even good ones may appear to have declining shot conversion rates from previous years or previous months, but often this is because they may have found themselves trailing or drawing more often and therefore had more frequently faced overtly defensively minded opponents. This variation in game position is to be expected for all teams across and during seasons. One year Arsenal will find themselves trailing or drawing more often than previously through random variation rather than a sea change in team quality.

Aggregated data can be very useful, but it can also mislead. 

Expected Points Graphs for EPL Games,November 19th 2011.

Norwich 1 Arsenal 2.

Scorers:
1-0,Morison,16'
1-1,van Persie,27'
1-2,van Persie,59'

Everton 2 Wolves 1.

Scorers:
0-1,Hunt(pen),37'
1-1,Jagielka,44'
2-1,Baines(pen),83'


Stoke 2 QPR 3.

Scorers:
1-0,Walters,8'
1-1,Helguson,22'
1-2,Young,44'
1-3,Helguson,54'
2-3,Shawcross,64'


Wigan 3 Blackburn 3.

Scorers:
0-1,Yakubu,2'
1-1,Gomez,8'
2-1,Caldwell,31'
Red Card,Dunn(Blackburn),48'
2-2,Hoilett,60'
3-2,Crusat,88'
3-3,Yakubu(pen),90+9'.

Promoted Teams Play it Safe at Home.

Having watched sub Premiership football for over two decades courtesy of Stoke City,from a fans perspective I can say that the jump in class to the EPL would appear to be a mighty one.Everything appeared to happen much,much quicker,the skill levels naturally were higher and mistakes were usually punished ruthlessly.Even mid table perennials stood head and shoulders above the everyday fare Stoke faced in the Championship.So you can only imagine how the players must have felt.

The cliche goes that there are no easy points in the EPL,but newly promoted teams do have the relative sanctuary of facing their fellow promoted teams four times in their first season.So it would be interesting to know how the promoted teams approach these match ups.It's very likely that fellow newboys are finding life in the EPL similarly tough,the average finishing position for promoted teams in their first EPL season is in the bottom quarter.So the teams involved must view these games as both an opportunity to pick up points and conversely a "must not lose" situation.

I've therefore taken the results of all six games involving the newly promoted sides to the EPL since 2001.The average league position of both the home and away teams when the game was played was 15th,which confirms the struggles these teams faced.

Results of games involving the teams promoted to the EPL 2000-2011.

Average Home Goals.	Average Away Goals.	Home Wins.	Draws.	Away Wins.
1.28	1.10	38%	33%	28%

The first point to make is that home advantage is greatly depressed in these games.This is illustrated in the smaller than usual difference between the average goals scored by the home and away sides.Home advantage is historically worth about 4 tenths of a goal in the EPL,however,in these games it is worth just 0.18 of a goal.On average the league position of the home team when these games were played was 15th,as was the away team's,therefore the average goal difference is solely down to home advantage.Similarly depressed is home wins 45-46% is the typical EPL figure for home teams,but just 38% in this case.Draws are higher than expected,a typical game between two teams hovering around 15th place would end stalemated 27% of the time instead of 33%.

So from the figures it would seem that the home side recognising the importance of these games adopts an over cautious approach resulting in less home wins,more draws and less match goals overall.The away side performs about as expected,they score about as many goals as a typical side of their quality would expect to score against a similar class of opponent and they win as often as expected.....but for the home side at least,losing is not an option to contemplate and potential home wins are turned into draws..

The EPL and The Championship Compared.

One very effective way to compare different leagues is to plot the monthly variations for each on the same graph.If we do this for the EPL and the Championship it quickly becomes evident that the leagues are fundamentally very similar.The shapes of the plots are mimic each ther quite closely for the majority of the season,differing only in the monthly values.

The EPL would appear to be a generally more attack orientated league.With the exception of the final month of the season,the EPL is more goal laden and certainly more shot laden.It's only in the truncated month of May that the Championship greatly outscores their superior rival.

The Championship commits less fouls in the first three months,but subsequently both the quantity and the variation matches that of the EPL very closely.By contrast the yellow card totals,although following a very similar track to those awards in the EPL,remain almost universally below the totals for the higher division.This implies that the EPL is officiated more strictly than is the Championship.

Monthly Variation in the EPL.

Monthly variation appears to be a well established effect in the EPL and it's most noticeable in the amount fouls committed per game.The number rises slightly after the first month of the season before showing a general downward trend for the rest of the campaign.There are occasional upward blips,most notably in December,possibly as a result of larger festive crowds.The most interesting question regarding foul counts is whether after an initial rise in number the teams self regulate and become more restrained or whether the competitiveness of the tackling remains broadly consistent and it is the officiating that varies. In tournament soccer referees do tend to be excesively harsh in their interpretation of what is a foul in the earlier exchanges before relaxing their stance as the competition progresses.

The reduction in fouls late in the season is possibly easier to understand.Players are either looking forward to the beach or important international tournaments.There will be games that have a huge bearing on relegation or the title,but these type of games will be outnumbered by other less vital matchups.A large downward spike in the number of red cards coupled with a decrease in fouls and increase in shots in April indicates the games becoming more benign later in the season.

Correlated varibles unsurprisingly tend to follow each other step by step.Goals peak relatively early in the season,level off mid year before peaking again towards the end of the campaign. Shots,shots on target and corners generally follow the trend set by goals.

The really interesting question is why the monthly variation happens.Is it down to coaching tendencies,is some of it driven by the officials or is it weather related?

Do Referees Subconsciously Influence Matches?

The Expected Points Model I use in this blog is a very potent weapon in analysis performance based questions in soccer.All things being equal the cumulative total pre game EP over a reasonably large number of games for individual teams almost always tallies up to be very close to the actual points a team achieves.If there is a major discrepancy between the actual and predicted points totals there is usually a readily discernible reason why.For example if we were to by pure chance select a series of games where the team was destined to have a player dismissed,the cumulative pre game EP for that team would be well above the actual points gained.However,the cause,namely the red cards would be readily apparent.

It is therefore possible to use an Expected Points approach to evaluate any influence a certain referee may
have had on the outcome of a batch of games.However,before I go any further I want to emphasis that because a referee shows up as an outlier in any analysis involving match decisions,it no way implies that there may be deliberate bias involved.People can respond subconsciously in different ways to very similar situations and if the majority react in a different way to you that difference can become apparent.It may present a problem in terms of consistency of decisions,but it doesn't automatically mean that there is deliberate bias at work.

Secondly,random events can converge to make a set of games appear unusual and the result of a common denominator such as the referee,when in fact they have just come about by random chance.I'll deal with these issues in a later post,but for now here's how games controlled by the major EPL ref's from the last seven seasons have panned out.

Recent and major decisions tend to be over emphasised in people's memories,so it's hardly surprising that every set of fans think that every official is out to get their team regardless of venue and regardless of their team's standing in the match in question.But it is an enduring opinion that refs favour either the home teams or the better teams in a game.This is reinforced because in general home sides and favourites will do more attacking,receive more free kicks and penalties and although this can be construed as favouritism from officialdom,it is really just down the better team getting it's just deserts.

These natural advantages enjoyed by home teams is already accounted for in the Expected Points calculation,so we can compare the cumulative pre game EP totals with the actual totals gained by the host and then sort them by referee to see if any patterns begin to emerge.All EPL games from the 2004-05 season until the present were included and any referee who officiated 30 games or less was excluded.The vast majority of the referees appear to officiate in games which on average produce actual cumulative points tallies that are extremely close to the pre game expectations.Mark Halsey for example officiated in 121 games and pre game estimates would have suggested that the home team would gain 198.5 points from those games.They actually gained 198 points.So from these figures we would have to conclude that Halsey was officiating in a way that corresponds very closely to the way officials have refereed in the past.Overall 13 of the 21 referees featured in the table saw their groups of matches produce home points that were within 4% or better of their expected value and the majority have been rewarded by being appointed to take charge of the FA Cup final.

Games officiated by Peter Walton saw the home side's actual points under perform the expected value by nearly 8% (two refs produced higher discrepancies,but their game tallies were each much less than Walton's total of 134 games).Again a caveat must apply.Firstly,these results could have arose through random chance or alternatively Walton may actually be one of the few officials who is able to resist the subconscious influence of the home support and it is his colleagues who are out of step.At the other end of the spectrum Mark Clattenburg is the ref with most games in the survey who saw an elevated points tally for the home teams he officiated.Home teams gained an average of 7% more points than expected when he was in charge.The most prestigious game either Clattenberg or Walton have taken charge of is the Community Shield.

Points accrued by home teams compared to pre game expectations sorted by referee.2005-11.

REFEREE.	PreGame Expected Points for Home Teams.	Actual Home Team Points.	% Difference.	Number of Games.
L Mason.	145.3	157	8.0	86
M Clattenburg.	210.1	224	6.6	124
L Probert.	126.0	134	6.3	72
K Friend.	56.9	60	5.4	33
U Rennie.	81.3	83	2.1	50
C Foy.	247.4	251	1.4	145
P Dowd.	251.5	255	1.4	152
M Dean.	274.1	276	0.7	169
M Halsey.	198.5	198	-0.3	121
M Riley.	178.7	178	-0.4	118
M Atkinson.	256.4	254	-0.9	151
A Marriner.	177.6	175	-1.5	109
H Webb.	294.2	289	-1.8	177
A Wiley.	244.0	239	-2.1	145
R Styles.	186.3	180	-3.4	111
G Poll.	104.6	101	-3.5	61
S Bennett	231.4	222	-4.1	139
P Walton.	231.7	214	-7.6	134
M Jones.	109.0	100	-8.3	60
S Atwell.	69.8	62	-11.2	36

* denotes a cup final referee.

If we repeat the exercise by tracing the points haul for all pregame favourites a similar pattern appears.14 of the listed refs see their games come in at within 4% of the expected points total.Game favourites are not always readily identifiable,for example when a relegation candidate entertains a mid table team.So in the majority of these games there's not a huge amount of subliminal pressure to ensure that the "right" team wins.Games overseen by Phil Dowd have coincided with the likes of Arsenal,Chelsea and ManU failing to fulfil their pre game expectations,while Lee Probert controlled matches has seen the favourites overachieve.

Points accrued by match favourites compared to pre game expectations sorted by referee.2005-11

REFEREE.	Pre Game Expected Points for Favourites.	Actual  Points won by Favourites.	% Difference.	Number of Games.
L Probert.	133.2	149	11.8	72
A Wiley.	278.4	296	6.3	145
C Foy.	274.1	290	5.8	145
R Styles.	213.2	222	4.1	111
K Friend.	60.6	63	4.0	118
M Riley.	191.1	199	4.0	33
M Halsey.	226.1	234	3.5	121
S Bennett.	268.8	276	2.7	139
G Poll.	111.1	113	1.7	61
U Rennie.	93.7	95	1.4	50
M Clattenburg.	235.5	236	0.2	124
D Gallagher.	63.2	63	-0.3	34
P Walton.	259.8	256	-1.5	134
M Dean.	313.6	309	-1.5	169
M Atkinson.	279.1	272	-2.6	151
A Marriner.	212.0	205	-3.3	109
L Mason	163.2	157	-3.8	86
M Jones.	113.3	109	-3.8	60
H Webb.	324.8	310	-4.6	177
S Atwell.	69.5	62	-10.8	36
P Dowd.	287.7	255	-11.4	152

From the previous table is is evident that the interpretation and implementation of the laws in the EPL are very largely consistent with previous seasons.There are very few refereeing performances that deviate greatly from the expected outcome over a large enough batch of games.The fewer the games the more likely it is we will see batches of games that appear atypical and even where outlying officials have controlled large numbers of games the possibility for freak sequences to appear is real and present.

Poor or contentious decisions are often seen as the result of incompetence or malice,but more often they result from a subconscious desire by the officials to please the majority ( in the case of home field teams) or satisfy the consensus opinion (in the case of favourites).Managers often criticise officials often unjustly,but always to gain a possible advantage in future games where the previous,often well publicised vent will subconsciously "encourage"referees to favour their side to atone for previous mistakes.

Judging by the above results EPL referees are largely immune to such pressures or if they do allow their judgement to be swayed it is within the bounds of historical limits and therefore relatively consistent between referees.

EPL Referees Appear To Show Disproportionately More Red Cards To the Away Side.

A glossary of terms is in the pipeline,but for the meantime.....The win probability for each team is tracked on a minute by minute basis throughout each game.At any point in a game a team will have a probability of winning the game outright and an associated probability of drawing the game.By multiplying the probability of winning the game by 3 and the probability of drawing by 1 and adding the results together you will get an expected long term average for the number of points that team will achieve from that position or Expected Points (EP) for short.

It should come as no surprise to discover that in games where one red card is issued,then that card is more likely to be shown to the visiting side.The home side more often will have the better chance of winning the game,therefore the away side will be forced to do more defending,sometimes against superior players,which will lead to more fouls,more yellow cards and ultimately more chance of a red.In the 300+ EPL games spread over the last 7 seasons where a red card was shown over 60% of the time it was shown to the away side.To further test our idea that poorer sides are more likely to attract red cards we first need a way of quantifying pre game team quality.One very good way to represent the difference in quality between teams prior to the game starting is to compare each side's Pre Game Expected Points tallies.And once again we find that the underdog,irrespective of venue receives over 60% of dismissals in games where a single red card is shown.The average Pre Game Expected Points for the team who receives the red card is 1.29 compared to 1.45 for the team that doesn't see red.

Expected Points Graphs for EPL Games,September 17th & 18th 2011.

The win probability for each team is tracked on a minute by minute basis throughout each EPL game.At any point in a game a team will have a probability of winning the game outright and an associated probability of drawing the game.By multiplying the probability of winning the game by 3 and the probability of drawing by 1 and adding the results together you will get an expected longterm average for the number of points that team will achieve from that position or Expected Points (EP) for short.

Blackburn 4-3 Arsenal.


The best game of the weekend from a neutral perspective.Arsenal owned the first half and Blackburn's manager,Steve Keane couldn't have been looking forward to his promised showdown with disgruntled fans.The Gunners then self destructed,this time on the defensive,rather than disciplinary front,but a they were a Paul Robinson injury time block away from a second 4-4 all draw within a year.Blackburn managed a goals to shots ratio of more than one after Arsenal's own goals.You get a 4-4 about once every 1,200 games.4-3's are more common,about 1 every 300 generally,but 1 in 500 when the much weaker team comes out the winner.

Scorers.
0-1,Gervinho,10'
1-1,Yakubu,25'
1-2,Arteta,34'
2-2,Song(og),50'
3-2,Yakubu,59'
4-2,Koscielny(og),68'
4-3,Chamakh,85'
................................................................................


Aston Villa 1-1 Newcastle.


Villa spent an hour hopeful of a three point haul,but not certain of it.Only home advantage made the Midlands team favoured,pre game and the decrease in EP upon Newcastle's equaliser emphasised how costly drawing from a winning position can be.





Scorers.
1-0,Agbonlahor,13'

1-1,Best,57'


....................................................................


Bolton 1-2 Norwich.  


Norwich gained the kind of reward some of their play this season has deserved.They replaced Bolton as EP favourites after the first strike and were greatly helped by the Klasnic red card.They remained much the most likely winners,even when Bolton halved their lead.Bolton still haven't recovered from their semi final mauling last April.

Scorers.
0-1,Pilkington,37'
0-2,Johnson,42'
Red Card Klasnic (Bolton),45'
1-2,Petrov(pen),64'
...................................................................


Everton 3-1 Wigan.


Wigan's opening goal was quickly cancelled out and both team's EP were trending towards 1 until Everton's two late strikes.



Scorers.
0-1,Di Santo,31'
1-1,Jagilka,33'
2-1,Vellios,84'
3-1,Drenthe,90'
..................................................................


Swansea 3-0 WBA.


A fairly unremarkable game statistically as the marginal favourites scored a couple of early goals to nudge the EP within touching distance of three and then comfortably ran the clock down.In reality a massive result at the Liberty against potential fellow relegation candidates.



Scorers.
1-0,Sinclair(pen),14'
2-0,Lita,24'
3-0,Dyer,49'
..................................................................


Wolves 0-3 QPR.


Another comfortable victory curve for the moneyed newcomers as Wolves' start becomes a distant memory and QPR's cash appears to have been well spent.

Scorers.
0-1,Barton,8'
0-2,Faurlin,10'
0-3,Campbell,87'
..................................................................


Tottenham 4-0 Liverpool.


Spurs' early goal split the two equally matched teams,but after that all the damage was self inflicted by Liverpool.Two red cards are fairly rare and by the time of the second one there was very little chance that Liverpool would be able to score and it was overwhelmingly likely that Tottenham would add to the Modric strike.Game over after an hour,even at just 1-0.

Scorers.
1-0,Modric,7'
Red Card,Adam(Livepool),28'
Red Card,Skrtel(Liverpool),63'
2-0,Defoe,66'
3-0,Adebayor,68'
4-0,Adebayor,90'
...................................................................


Sunderland 4-0 Stoke.


Another cakewalk for the side scoring first.A game of little interest after the two early goals unless you were a Sunderland fan.

Scorers.
1-0,Bramble,5'
2-0,Woodgate(og),11'
3-0,Gardner,28'
4-0,Larsson,59'
..................................................................


Fulham 2-2 Man City.


Comeback performance runner up of the week award goes to Fulham,who were very impressive in recovering from a two goal deficit against markedly superior opponents.The Cottagers' EP had hit the floor after almost an hour and that's where it was heading again until Murphy's late leveller.

Scorers.
0-1,Aguero,18'
0-2,Aguero,46'
1-2,Zamora,56'
2-2,Murphy,75'
.....................................................................


Man Utd 3-1 Chelsea.


A seemingly comfortable Man U win concealing a hand full of "what ifs".To take the most obvious one,if Torres had scored instead of missing an 83rd minute open goal,the score would have been 3-2 to United.About 12 minutes remained and Chelsea's EP would have increased over tenfold from 0.01 Expected Points to 0.13 EP.However, Man Utd would still score around 60% of any subsequent goals,it was odds on that there wouldn't be any anyway and the home side's EP would still have been 2.75.In a nutshell,if Torres had scored,Man Utd would still go on to win 9 times out of 10.Of course Sir Alex would argue that Chelsea should have been down to 10 men by then anyway.

Scorers.
1-0,Smalling,8'
2-0,Nani,37'
3-0,Rooney,45'
3-1,Torres,49'