9 ms·

Goldman Sachs model to predict World Cup game results didn’t come close

245 points by rodionos 8y ago

dmichulke 8y ago
I watched quite a few matches and among the things I saw in the matches but not in any statistics are: - motivation (Germany and Croatia were the two extremes here, no idea how to measure it) - team cohesion (number of articles in a few journals questioning the team cohesion, maybe also articles about individual players) - creativity in offense (maybe measurable via "target missed from close distance" + "ball passed front of the goal") - number of errors in defense that didn't lead to a goal - percentage of times ball possession was lost from own goal to enemy's area (England was really bad here against Croatia)
- smcl 8y ago
  These kinda show what makes predicting football particularly difficult. I like the ideas, and I think we (or more likely some ML algorithm) can come up with the set of conditions that showed why France prevailed against the specific opposition at this specific World Cup ... but I suspect that the conditions would be pretty unique and invalid for Euro 2020, WC 2022 etc. As you identified, motivation could be pretty hard to measure ... but even if we could it might be a pretty poor predictor anyway. France in the early stages didn't look very motivated, while England and Colombia looked pretty lively. Team cohesion - the German team were pretty consistent (not dazzling, but consistent) and we know how that ended. Again France didn't really impress until the latter stages of the WC. Creativity in offense - I guess it can indicate a sort of calm or confidence in front of goal but actually it can actually be seen as pretty negative. For example Arsenal a few years back came under fire for having plenty of possession in the 18 yard box but failing to convert. Spain's confident quick pass-and-move "tiki-taka" was ever-present and has in my eyes been impotent in the last few years (and more important as a neutral viewer - very frustrating to watch). Defensive errors that didn't lead to a goal could be a nice indicator of the ability of a defence to pick up after each others mistakes - but at the same time these errors that lead to goals (i.e. Croatia's second goal in the final) are relatively rare and a lack of a goal could just point to the opposing team's inability to convert due to a poorly organised or a lack of opportunism from their strikers. I'm not sure what you mean with the last one, but I think this could be a nice one - if you mean "times you lost possession in your own half". A profligate midfield and defence is bound to ship goals, I doubt there are many teams that can either fight back after trailing by a goal or two or score enough to maintain a reasonable buffer. I applaud the effort though - it takes more creativity and care to think of some new angles (like you did) than to think of some possible counter examples (like I did)!
  - dmichulke 8y ago
    Thank you for the warm words, I guess the reason is my occupation plus the fact that I just spend my last few weeks watching many games with family and friends. > I'm not sure what you mean with the last one, but I think this could be a nice one - if you mean "times you lost possession in your own half" Almost, England lost the ball frequently (> 50+x% with a large x AFAI could see) due to the keeper sending out long balls. I'd like to measure that somehow. Could be done via number of seconds in possession after a goal kick, an indicator whether a hypothetical 85% marker of the field was reached or measuring whether the ball was at least 5x successfully passed (or resulted in a goal).
    - smcl 8y ago
      Ahhhh I see. Actually this is something I've really been curious about myself - whether the better strategy overall for a keeper returning the ball into open play (from goal kick or from hand) is to just boot it as far up-field as possible or passing it short to one of the defence or midfielders sitting deep. Interestingly something like this is a tactic used in Rugby (https://www.youtube.com/watch?v=cbti6mLvSJs https://www.youtube.com/watch?v=cbti6mLvSJs). I used to play a lot of football when I was younger and at our level (waaaay down the scottish league pyramid) against tired, hungover or generally weak opposition, keeping them under pressure by dominating the territorial game but sacrificing possession was criminally underrated. Usually if you could keep hammering them for 60 minutes and had the legs to step up a gear in the last 30 or so you could grab a valuable goal or two :-)
- lagadu 8y ago
  > - creativity in offense (maybe measurable via "target missed from close distance" + "ball passed front of the goal") This one would benefit possession-based teams, so it would fail to give decent odds to the current world and european champions (France and Portugal respectively) which don't play possession. Of course it's possible they're outliers but we'll never know.
boomboomsubban 8y ago
The World Cup is about the worst sporting event for data led predictions like this, far too much can rely on a few events that are basically a coin flip. It would be interesting to see how the predictions went for something like the Premiere League tables.
- fwdpropaganda 8y ago
  > far too much can rely on a few events that are basically a coin flip Can you give us some examples?
  - gtr 8y ago
    As there are relatively few goals, anything that can turn a goal into not a goal or vice versa can have a massive impact on the game. For example the penalty decision against Croatia in the final. Another thing that adds to the randomness is the chance that a key player may be sent off or injured.
    - systoll 8y ago
      To make that be specific -- in 44 of 64 games, and in every single penalty shoot-out, turning one goal into not a goal or vice-versa would've changed the outcome.
      - ufo 8y ago
        It is not so simple because goals in football are not independent from each other. A team that scores first has the opportunity to play more cautiously and go for more counter attacks.
  - zaphirplane 8y ago
    Someone deciding to handball the ball out of the way, a tackle that goes harder in a temper flare. A penalty that is saved/missed
  - kryptiskt 8y ago
lowkeyokay 8y ago
If anything, this is a clear illustration of poor use of probabilistic prediction. When used for investments you have many outcomes. If the model is any good, you will most of them right. In the World Cup you have very few. Even if you count all games played. Definitely not excusing Goldman Sachs here, they should have known better than to try to predict this. There was only a tiny chance this could be great advertisement for their model.
- Ntrails 8y ago
  > they should have known better than to try to predict this. There's no downside, only free publicity. If they, by good fortune and a following wind, get it right - then the publicity is incredible. If it's wrong they laugh and say "well, better stick to predicting what we're good at!" and they still get a shitload of headlines and awareness of their product. This was not a mistake.
  - jamespo 8y ago
    Well, there's the downside of articles like this pointing out they've had 4 years to work on their models and they've got worse
blattimwind 8y ago
This site is a good counter-example for website optimization: While it uses many assets, so a CDN domain makes sense, it spreads them out thinly. It loads over 100 CSS files, most of which are below 1K. Similarly it loads approximately 30 JS scripts, most of which are just a few K each. This is mitigated to a large extent by using HTTP/2.0, which permits a few dozen or so parallel requests, but it still means that a repeated load of the page takes 2-3 seconds. (Without HTTP/2.0 this probably takes ages, since browsers open only a few connections to each origin at most). There is also almost no difference between reloading with and without the cache.
barrkel 8y ago
I put money on Belgium (12.0 decimal odds) and Croatia (15.0) after the group stages, where some form was visible, combined with knowledge that they had some of the world's best players. The odds shortened as the tournament progressed, I was able to hedge as the shortened odds made lay betting profitable. (High variance in football outcomes means there's no guarantee of profit, I don't bet big sums.)
- anoncoward111 8y ago
  This answer is very useful and contains proper strategy advice :) If someone were to bet during the round of 16, if someone were to bet $1 on the bottom 8 and $2 on the top 8, the strategy would most likely yield a small profit or a small loss, rather than a total loss.
geraldbauer 8y ago
PS: If you want to build or train your own model or make predications, you can find open (structured) data about all world cups at the football.db, see https://github.com/openfootball/world-cup https://github.com/openfootball/world-cup and https://github.com/openfootball/world-cup.json https://github.com/openfootball/world-cup.json Enjoy the beautiful game.
pelasaco 8y ago
"Of course, past data don’t always predict the future; Goldman Sachs never tells clients to make decisions solely on the basis of its models’ findings" in the real world Goldman Sachs would manipulate the games to make it looks like they got it right and their clients got richer.
denzil_correa 8y ago
The "Ludic Fallacy" strikes again [0]. > The ludic fallacy, identified by Nassim Nicholas Taleb in his 2007 book The Black Swan, is "the misuse of games to model real-life situations." ... > The alleged fallacy is a central argument in the book and a rebuttal of the predictive mathematical models used to predict the future – as well as an attack on the idea of applying naïve and simplified statistical models in complex domains. According to Taleb, statistics is applicable only in some domains, for instance casinos in which the odds are visible and defined. Both Taleb's books, "The Black Swan" and "Fooled by Randomness" are an interesting take for such models. Meanwhile, most economists know about "Knightian Uncertainty" [1] which talks about differentiation of risk and uncertainty. > "Uncertainty must be taken in a sense radically distinct from the familiar notion of Risk, from which it has never been properly separated.... The essential fact is that 'risk' means in some cases a quantity susceptible of measurement, while at other times it is something distinctly not of this character; and there are far-reaching and crucial differences in the bearings of the phenomena depending on which of the two is really present and operating.... It will appear that a measurable uncertainty, or 'risk' proper, as we shall use the term, is so far different from an unmeasurable one that it is not in effect an uncertainty at all." [0] https://en.wikipedia.org/wiki/Ludic_fallacy https://en.wikipedia.org/wiki/Ludic_fallacy [1] https://en.wikipedia.org/wiki/Knightian_uncertainty https://en.wikipedia.org/wiki/Knightian_uncertainty
- fwdpropaganda 8y ago
  Damn, do I disliked Nassim Taleb. I don't think I've ever heard him say anything deep. That wikipedia article is an excellent. In [0] you have the following: > The ludic fallacy, identified by Nassim Nicholas Taleb in his 2007 book The Black Swan, is "the misuse of games to model real-life situations." And he gives an example of this: > One example given in the book is the following thought experiment. Two people are involved: > Dr. John who is regarded as a man of science and logical thinking > Fat Tony who is regarded as a man who lives by his wits > A third party asks them to "assume that a coin is fair, i.e., has an equal probability of coming up heads or tails when flipped. I flip it ninety-nine times and get heads each time. What are the odds of my getting tails on my next throw?" > Dr. John says that the odds are not affected by the previous outcomes so the odds must still be 50:50. > Fat Tony says that the odds of the coin coming up heads 99 times in a row are so low that the initial assumption that the coin had a 50:50 chance of coming up heads is most likely incorrect. "The coin gotta be loaded. It can't be a fair game." > The ludic fallacy here is to assume that in real life the rules from the purely hypothetical model (where Dr. John is correct) apply. Would a reasonable person bet on black on a roulette table that has come up red 99 times in a row (especially as the reward for a correct guess is so low when compared with the probable odds that the game is fixed)? So Nassim Taleb wanted to discuss "using games to model real-life situations" and to demonstrate the pitfalls he uses two characters. He _portrays_ the characters as "man of logical thinking" vs "man who lives by his wits", but as we'll see he's missing one dimension to his characterization. The first problem here is that implicitely he's suggesting to the reader that the decisions of the "man of logical thinking" represent the pitfalls of "applying games to model real-life situations", whereas the the other guy's decision represent.... it's not specified, but clearly has a better outcome. The second problem, is that he conflates "applying something you read on some textbook to real life without thinking" with "modelling real-life". He suggests to the reader that those two people are actually "logical" vs "instinct", but they're not. They're a dumb guy who knows maths vs a smart guy who doesn't know math. _Obviously_ real-life is more complex than your textbook examples, and so the smart guy is going to win because his fuzzy heuristics beat the first guys decisions which are optimal within his flawed model. An actual smart and logical person would update his model based on new evidence (i.e. "I was told that this coin was 50-50 but actually the chance of what I just saw is so small that it's more likely that I was just lied to") and then use maths to make predictions and beat the guy who's smart but doesn't know math. So ironically, he wants to portray the dangers of using over-simplified models and to do that he uses an example where he obscured one dimension. Nassim Taleb is really good a rhetoric but light on substance. [0] https://en.wikipedia.org/wiki/Ludic_fallacy https://en.wikipedia.org/wiki/Ludic_fallacy
jasode 8y ago
Leonid Bershidsky and a lot of other journalists laughing at Goldman Sachs' incorrect predictions seem to miss the point. The World Cup predictions from Goldman Sachs (and also UBS) are a form of recreation and entertainment with machine learning. It's an expression of quant nerd humor. Analogous intellectual games would be engineers devising ridiculous Rube Goldberg contraptions[1] or programmers building "enterprise" FizzBuzz[2]. (I think it would add to the fun if GS uploaded their raw data and models to Github for others to play with.) >It certainly didn't predict the final opposing France and Croatia on Sunday. True, but it did predict France having better chance winning overall but was handicapped by a tougher draw. It also predicted France beating Croatia in round 16 instead of the final. The pdf says: >While Germany is more likely to get to the final, France has a marginally higher overall chance of winning the tournament, [1] https://en.wikipedia.org/wiki/Rube_Goldberg_Machine_Contest#Past_tasks https://en.wikipedia.org/wiki/Rube_Goldberg_Machine_Contest#... [2] https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...
- learnstats2 8y ago
  On the other hand, this is a predictive task that has defined outcomes and clear historical data - by my understanding, it is easier than commercial uses of machine learning [at least, easier to measure the effectiveness]. It's also Goldman Sachs and UBS choosing to attach their names to these and stake some reputation on these predictions. If they had hit the bullseye, they would be lauding these results.
  - sgt101 8y ago
    Yup, the worst thing is that if they had got it right it would have been more or less due to pure chance, and it would have led to business flowing their way!
  - CoryG89 8y ago
    It may be easier to measure the effectiveness (give a confidence level for the prediction), but just because there is clear historical data and defined outcomes, that does not mean you will be able to predict a particular outcome with any high level of certainty. For example, imagine a tournament with a large number of participants, where the winner is picked simply by fairly choosing a single random participant. If I then gave you all the perfect historical data going back decades, you could do statistical analysis and determine that the winner is completely random and therefore the probability of success, for any particular participant, is p~=(1/n), where n is the number of participants. Your confidence in correctly predicting any particular outcome will drop as n rises. Not everything can be easily predicted just because you have enough data.
raverbashing 8y ago
People conflate statistics with actual results more often than not and I think those reporting on such stories and maybe even the original authors might fall for this. It was not wrong to say Hillary had a 95% chance of winning the presidential election, but the confidence was low and that value still allowed for the opposite result to happen. Also football has a lot of variance concerning team capability and end results. The better team might (and does) lose often, especially when going to penalty shoots. With basketball, the stronger team will be easily scoring more in most cases.
- kgwgk 8y ago
  > had a 95% chance [...] but the confidence was low So she had 95% chance of winning with 50% probability or what?
  - raverbashing 8y ago
    it is captured in the 95% but that was probably a bit overestimated (and there are always unknown biases and improbable events can happen) What's wrong is thinking 95% chance of winning means they will win
    - kgwgk 8y ago
      So it was not captured in the (overestimated) 95% :-) But I agree, the most likely thing, even if the probability was perfectly known, is not always what happens.
  - frockington 8y ago
    60% of the time, it works every time - Brian Fantana
  - fny 8y ago
    So this is something that people don't seem to grok quite well, and it really depends on the type of statistical analysis used. Say you make the assumption that the quantity being estimated is truly fixed: that there's some true value for the force of gravity or some true value for the number of people that vote for X or Y. The second assumption that comes along is that the stochasticity observed comes from your perspective of observation, and not from the ground truth. To be more blunt, you know that of all the observations you make 95% of them have the probability of yielding the result observed... but the ground truth is still fixed. Gravity has a fixed quantity, despite your experimental error, and you may have been lucky enough to observe it in your sample. Predicting elections with frequentist methods has this same characteristic, except the observed quantity itself shapeshifts and even lies... so then there are other complications that need to be dealt with. This is where that 50% feeling comes from. There are two outcomes, one will be true. You're data analysis just tells you that if you repeat your procedure, you'd expect 95% of those result to give you the outcome you observed.
tirumaraiselvan 8y ago
It's a fools errand to predict high variance events like football games.
- pbhjpbhj 8y ago
  Only predict events that are easy to predict, never fail!
anonu 8y ago
People love to beat up on these companies because of this stupid world cup prediction. Yes, Goldman is a giant vampire squid wrapped around the face of humanity (Matt Taibi quote). But it turns out it's really just great marketing for their research teams. Also, I've seen some people say (not in this forum) that banks now look stupid because they're in the business of making predictions and they can't even get the world cup right. Guess what? Banks make no money on predictions. They make money on flows and taking spreads on trades they do with clients. Any research or prediction is meant to be a catalyst for that trade.
- chopin 8y ago
  I am pretty sure Banks make money on predictions if they get people on following them.
  - anonu 8y ago
    Yes... This is less a prediction and more a legal form of front running.
    - throwawaymath 8y ago
      In what way is this a legal form of front running?
      - tedunangst 8y ago
        Everything Goldman Sachs does is front running and it's legal because they bribed all the regulators. QED.
- jasode 8y ago
  >Banks make no money on predictions. They make money on flows and taking spreads on trades they do with clients. You're mostly right but to further clarify, an investment bank like Goldman Sachs has revenue from mostly "market making" spreads but it does also have activities that depend on predictions such as their proprietary trading (before the Volcker Rule shut them down) and their GSAM (Goldman Sachs Asset Management) fund. The GSAM is basically a hedge fund for their wealthy clients' money. They will run predictions on macro trends on data like interest rates, commodities, indexes, etc to help them pick stocks for their portfolio. As the pdf noted, the World Cup data models and simulations came from Adam Atkins of GSAM.
yk 8y ago
> And in any case, the model only generated probabilities of winning a game and advancing, and no team was given more than an 18.5 percent chance of winning the World Cup. > [...] > But Goldman Sach’s misfire is perhaps the most curious. The model said, that there is a lot of uncertainty, and as it happens, it was entirely correct. A World Cup chance of 18.5 percent means, that 4 out of 5 times the team will not win, and that that is the highest chance does not say much about the model. And in general this is one instance of the well practiced journalistic technique to wait for results first and then define a bar afterwards to criticize the results according to standards that did not exist when the performance happened. (I guess in this case it is even worse, we could construct a reasonable test of the model performed, I have the suspicion that that was in the original paper and that the journalist either did not understand it, or, more likely, choose to ignore it in favor of writing a better story.)
- pbhjpbhj 8y ago
  Uncertainty is a truism; that's why people want to use a prediction algo. Did the system so better on results it was more certain about? Predicting the result of an A or B contest the bar is already defined. Either the system gets it right or doesn't, if it gets it right more often than not then (despite this being poor grounds mathematically, on a small result pool) popular press will report it as successful. IMO if matches become easy to predict then rules will change to reduce that predictability.
  - LeifCarrotson 8y ago
    > Predicting the result of an A or B contest the bar is already defined. I disagree: If team A has a 10-30% chance of winning, and A pulls off the upset, the correct answer was not "A Wins" it was "B has a 70-90% chance of winning". For Goldman Sachs' investments, the bar is not to predict that A wins or that B wins, it's to predict the probability and variance regarding which team will win. Of course, from a single upset game, it's impossible to tell whether these estimates are correct. You'd need to see the success or failure of many trials.
    - 8y ago
kulu2002 8y ago
Good... There was this discussion thread few days back on HN https://news.ycombinator.com/item?id=17509407 https://news.ycombinator.com/item?id=17509407 Did this investment bank use same set of algorithms that they use for financial predictions? ...And then I remember there was this Octopus[1] who used to predict winners with 85% accuracy [1]https://en.wikipedia.org/wiki/Paul_the_Octopus https://en.wikipedia.org/wiki/Paul_the_Octopus
tomelders 8y ago
While I agree that it's somewhat silly to try and predict a word cup winner like this (and I suspect it was just a bit of fun anyway), there is one other reason that could explain why all these attempts got it so wrong. Cheating. Before people start booing, let's not forget where this tournament is being held, and all the other nefarious things that country has been up to recently.
- teamk 8y ago
  FIFA has been corrupt for decades. Although supposedly its been cleaned up since Blatter was removed, it is doubtful the institutional corruption has been eliminated completely. The only question is how pervasive it is.
msravi 8y ago
Duh. Looks like there's a fundamental misunderstanding of how statistics works all around. The probability of an event does NOT predict a particular outcome. Ever. It only says that if the experiment is performed again and again and again, like a few thousand times, then X% of those will match that probability. If I toss a fair coin you cannot predict the next outcome. You can only say that if I toss the coin a 1000 times, then close to 500 are going to turn up heads, and another 500 are going to turn up tails. It was stupid of Goldman Sachs or whoever to predict an outcome. It was stupid of anyone else to lend credence to that prediction. Hopefully, Goldman Sachs is not relying on prediction of singular outcomes to make their investment decisions. I don't think they are. Probably just marketing brouhaha to ride the soccer wave. Although I'm not sure if that worked as expected.
- pbhjpbhj 8y ago
  I agree completely with your opening remarks. >"You can only say that if I toss the coin a 1000 times, then close to 500 are going to turn up heads, and another 500 are going to turn up tails." Sometimes you can do that and every single flip will be heads. It's unlikely, and across zillions of universes you'd only find it once - but we don't have a pool of universes that we can sample statistically.
- Sean1708 8y ago
  > It was stupid of Goldman Sachs or whoever to predict an outcome. If you read the actual report they did[0], they never claimed that any single outcome was more than 18.5% likely. [0]: http://www.goldmansachs.com/our-thinking/pages/world-cup-2018/multimedia/report.pdf http://www.goldmansachs.com/our-thinking/pages/world-cup-201...
rdlecler1 8y ago
In the world of models increasing precision for not necessarily increase accuracy.
cascom 8y ago
Isn’t this a little like flipping a coin four times - getting heads four times in a row, and looking at your friend and saying “but you told me the odds were 50/50 each flip?!”
- thousandautumns 8y ago
  Yes, it is.
crispyambulance 8y ago
I am somewhat shocked that GS would jump into the prediction business of the World Cup, even as joke. The risk of people getting the wrong idea about the prediction and GS itself is too great, even with a perfectly defensible model. This is an enterprise for bookies, not Goldman Sachs.
- TuringNYC 8y ago
  FYI - I worked at Goldman Sachs and then a hedge fund for a decade. On the Capital Markets / Trading side, you are literally a bookie. In fact the nomenclature is "you have a book." You are setting trading spreads based on where you think things will go. Depending on the market, your work may be more or less statistical and you're trying to gain a statistical advantage.
  - Maro 8y ago
    Off-topic: were you able to retire after that decade?
    - TuringNYC 8y ago
      Short Answer: No, but very comfortable. Long Answer: Full retirement is hard, Healthcare is a pain in the US. You cant really "save" for it in the US, it can swallow all your savings, so you'll always need some job or another to cover healthcare and catastrophic needs. That said, you can very easily down-shift once you have a house, savings, etc. Longer Answer: Could have, if I wanted to -- but you always give something up in exchange. These jobs will take everything you give them (time, health, life) and give back a decent percentage (income.) But you cannot easily dial up or down the work, it comes in chunks and you have to complete it. My life was increasingly unhinged at 27 and I decided to jump off the treadmill after seeing a colleague continue to work through his mother's terminal illness and death. Inertia and greed are a toxic combination. Numerous colleagues were on drugs, uppers, anti-depressants, etc. One died from stress (heart attack in his 30s.) I chose to get married, have two kids. I switched to a pure tech job (now an ML product owner at a Series A pure tech firm.) We have dinner together almost every single day. Weekends are completely ours. We go to the park most warm days. We take 3 to 4 vacations a year, many with my mom as well. There is a decent amount of work but I can choose when to do it (unlike Wall St.) and the work is longer term and I can dial it up/down as family requires. I sit outside and read during lunch. I turn off the markets when i step out of work. Many of my colleagues were easy millionaires by ~30 and multi-mullionaires if they stuck till their mid 30s and were focused. Many others blew through their bonuses (or snorted it away) and ended up with nothing and just live bonus to bonus. It also depends on the job (business/deal side vs quant side vs tech -- the money is a waterfall across the 3 sections.) As with all industries, you get ripped off if you dont fight for your share of the pie. Plenty of people avoid conflict and life comfortable lives and nothing more. I also saw several C++ programmer/manager earn double digit millions of dollars over several years, one earned over 100MM USD over the course of his time at the hedge fund (public records, check out AIG-FP https://en.wikipedia.org/wiki/AIG_bonus_payments_controversy https://en.wikipedia.org/wiki/AIG_bonus_payments_controversy) I think I did well and hopefully dont have to worry about poverty anymore. You either get lucky (early FB employee, hot product at xyz.com) or you have to give up something. I havent seen someone truthfully say they got both money and family and happiness all together.
known 8y ago
GarbageIn = ML = GarbageOut
known 8y ago
I worked in GS; Soccer/football prediction is not their forte
iainmerrick 8y ago
Thanks to the use of more granular data, made possible by AI, this year’s model should have worked better than the 2014 one. If anything, it worked worse. "If anything"? All the results are available, so it would be easy to put a precise number on this. Measure the Bayesian regret, or just report the winnings if you had used the GS model to bet on the outcomes. Unless it reports some concrete numbers, this article is garbage. It doesn't report any concrete numbers.
Keyframe 8y ago
It's as good time as any to plug in EA's simulation results: https://www.easports.com/fifa/news/2018/ea-sports-predicts-world-cup-fifa-18 https://www.easports.com/fifa/news/2018/ea-sports-predicts-w...
Sean1708 8y ago
In case anyone was interested here is a table of how likely the model thought each team was to make it through any particular stage[0] along with the stage that that team went out in and the probability that the model gave for that particular outcome (i.e. [probability of making it through the final stage they made it through] - [probability of making it through the stage they went out in]). Groups Round_16 Quarters Semis Finals Out_In Probability Brazil 87.5% 60.8% 42.0% 27.9% 18.5% Quarters 18.8% France 81.4% 58.4% 36.6% 19.9% 11.3% Won 11.3% Germany 80.5% 49.5% 30.5% 18.8% 10.7% Groups 19.5% Portugal 75.2% 52.8% 32.2% 17.3% 9.4% Round_16 22.4% Belgium 78.5% 51.1% 27.7% 15.8% 8.2% Semis 11.9% Spain 72.3% 50.1% 28.8% 15.4% 7.8% Round_16 22.2% England 73.1% 46.6% 24.4% 13.4% 6.5% Semis 11.0% Argentina 79.7% 44.2% 24.1% 11.8% 5.7% Round_16 35.5% Colombia 74.9% 37.3% 17.0% 8.5% 3.7% Round_16 37.6% Uruguay 74.4% 34.6% 17.2% 7.2% 3.2% Quarters 17.4% Poland 68.5% 30.5% 12.8% 5.8% 2.3% Groups 31.5% Denmark 47.8% 26.3% 12.4% 5.2% 2.0% Round_16 21.5% Mexico 52.0% 23.2% 10.5% 4.9% 1.9% Round_16 28.8% Sweden 45.9% 19.4% 8.3% 3.7% 1.3% Quarters 11.1% Iran 35.4% 18.1% 7.2% 2.6% 0.8% Groups 64.6% Peru 37.3% 17.2% 6.8% 2.5% 0.8% Groups 62.7% Australia 33.5% 15.4% 6.3% 2.3% 0.7% Groups 66.5% Russia 47.9% 16.3% 6.0% 2.0% 0.7% Quarters 10.3% Croatia 49.8% 16.9% 6.3% 2.1% 0.6% Finals 4.2% Switzerland 52.8% 15.9% 6.1% 2.0% 0.6% Round_16 36.9% Iceland 45.2% 15.1% 5.6% 1.8% 0.5% Groups 54.8% Costa_Rica 36.8% 13.3% 4.7% 1.6% 0.5% Groups 63.2% Serbia 32.9% 12.1% 4.5% 1.5% 0.5% Groups 67.1% Japan 36.5% 12.8% 3.8% 1.3% 0.4% Round_16 23.7% Saudi_Arabia 43.4% 12.7% 4.2% 1.3% 0.4% Groups 56.6% Tunisia 35.2% 13.3% 4.1% 1.3% 0.4% Groups 64.8% Egypt 34.4% 8.7% 2.5% 0.7% 0.2% Groups 65.6% South_Korea 21.6% 5.9% 7.1% 0.5% 0.2% Groups 78.4% Morocco 17.1% 6.8% 1.8% 0.5% 0.1% Groups 82.9% Nigeria 25.2% 6.5% 1.7% 0.4% 0.0% Groups 74.8% Senegal 20.1% 4.9% 1.2% 0.3% 0.0% Groups 79.9% Panama 13.2% 3.3% 0.5% 0.1% 0.0% Groups 86.8% [0]: Exhibit 2 in http://www.goldmansachs.com/our-thinking/pages/world-cup-2018/multimedia/report.pdf http://www.goldmansachs.com/our-thinking/pages/world-cup-201... Edit: Fix copy-paste errors and atrocious maths.
kgwgk 8y ago
The predictions were not so bad. At least one of the favourites won in the end. GS had France winning with 11.3% probability, second to Brazil with 18.5%. UBS was less fortunate, they had Germany (24%), Brazil (19.8%), Spain (16.1%) and England (8.5%) before France (7.3%). I compared the logloss for their predictions with the "uniform" benchmark (giving each team 1/32 probability of winning, 1/16 probability of getting to the finals, etc) and the results are the following (if I transcribed the data properly): Getting to second round: GS: 0.495 UBS: 0.495 bench: 0.693 Getting to quarter-finals: GS: 0.463 UBS: 0.459 bench: 0.562 Getting to semi-finals: GS: 0.310 UBS: 0.327 bench: 0.377 Getting to final: GS: 0.231 UBS: 0.269 bench: 0.234 World-cap winner: GS: 0.097 UBS: 0.113 bench: 0.139 The performance of the models was ok until Croatia got to the finals. This hurt specially UBS, who predicted less than 0.9% probability of such an event (compared to 2.1% in Goldman's model). Edit: these would have been the "best case" scores (if the high-probabilty teams had classified to each round, ignoring that this may be impossible due to the structure of the tournament): GS: 0.432 0.302 0.220 0.141 0.079 UBS: 0.365 0.251 0.176 0.111 0.070 UBS could potentially achive lower logloss metrics because it had more extreme predictions.
corpMaverick 8y ago
Soccer is a sport with a big random component. This is probably why it is so exciting. An average team can beat a better team. The reason is easy to see. The game can be decided by one, two or three key plays. Compare that to basket ball. To win a game you have to consistently score more and defend better. Rarely the game is decided by one or two plays. That only happens when the game is already very tight.
patagonia 8y ago
Financial modeling is about risk adjust return. Because GS knows they can not determine with certainty the outcome of a given investment, they diversify and hedge. Most of all, GS is a market maker, the equivalent of a bookie. To say that GS’s models “didn’t come close” is to ignore all the ways in which such a grading scheme is different than GS’s actual business model. If their WC prediction efforts acted as anything more than a fun spirited PR project, it was likely that GS wanted to somehow keep its employees engaged and adding business value during the WC which they otherwise would have been certainly watched all month.
gesman 8y ago
If GS would need to bet money - their actual business model would likely be to sell a bit of each higher probability losers (less risk) vs. buy big on a projected winners (higher risk).
deleted 8y ago
[deleted]
rcdmd 8y ago
This article didn't compare the Goldman Sachs model to any other models-- why not compare it with sports betting odds? Would Goldman have made or lost money betting their model was better than the crowd?
- sunstone 8y ago
  Or compare it with the fivethirtyeight blog predictions.
rossdavidh 8y ago
In addition to the many other problems with this article, I would like to point out that if, somehow, Goldman Sachs had managed to create a model that could accurately predict the results, the game of soccer would have to be changed to make it more unpredictable somehow. It is intrinsic to the nature of sport that, in order to be entertaining, there has to be a realistic chance for more than one team to win. Not many people (even from the winning country) would bother watching if it were accurately predictable.
vl 8y ago
>Soccer, with the many factors that affect game outcomes — players’ injuries and intra-team conflicts, the refereeing, the weather, coaches’ errors and moments of inspiration — remains only a tightly-regulated game involving a few dozen people. The behavior and performance of big corporations, entire industries and nations is arguably even more difficult to model based on data about the past. Author misses the way models work entirely, the larger the entity, the more statistics and averages kick in, and as a result, better model can be built.
- Donald 8y ago
  Depends on the complexity of the interactions between variables. There are plenty of examples where we have excellent local models, but make (comparatively) worse prediction at scale. A pretty classic example is biology - we have excellent knowledge about how genotypes work and their interactions in cells, but models of phenotypes are typically expensive, error-prone, or non-existent.
IkmoIkmo 8y ago
You'd have to run this world cup thousands of times by simulation, running it a single time and determining the results are not in line with the model is meaningless and silly. It's as silly as saying my claim for the odds of nearly perfectly modelling a coin toss (approximately 50/50%) is wrong because a series of 10 coin tosses show different results from my model. The model is not any less correct.
hsienmaneja 8y ago
They don’t have an edge like they do in their bread and butter markets, combined with a small sample set == high probability of a single year of sports predictions falling over like this

245 points by rodionos 8y ago

I watched quite a few matches and among the things I saw in the matches but not in any statistics are: - motivation (Germany and Croatia were the two extremes here, no idea how to measure it) - team cohesion (number of articles in a few journals questioning the team cohesion, maybe also articles about individual players) - creativity in offense (maybe measurable via "target missed from close distance" + "ball passed front of the goal") - number of errors in defense that didn't lead to a goal - percentage of times ball possession was lost from own goal to enemy's area (England was really bad here against Croatia)

smcl 8y ago
These kinda show what makes predicting football particularly difficult. I like the ideas, and I think we (or more likely some ML algorithm) can come up with the set of conditions that showed why France prevailed against the specific opposition at this specific World Cup ... but I suspect that the conditions would be pretty unique and invalid for Euro 2020, WC 2022 etc. As you identified, motivation could be pretty hard to measure ... but even if we could it might be a pretty poor predictor anyway. France in the early stages didn't look very motivated, while England and Colombia looked pretty lively. Team cohesion - the German team were pretty consistent (not dazzling, but consistent) and we know how that ended. Again France didn't really impress until the latter stages of the WC. Creativity in offense - I guess it can indicate a sort of calm or confidence in front of goal but actually it can actually be seen as pretty negative. For example Arsenal a few years back came under fire for having plenty of possession in the 18 yard box but failing to convert. Spain's confident quick pass-and-move "tiki-taka" was ever-present and has in my eyes been impotent in the last few years (and more important as a neutral viewer - very frustrating to watch). Defensive errors that didn't lead to a goal could be a nice indicator of the ability of a defence to pick up after each others mistakes - but at the same time these errors that lead to goals (i.e. Croatia's second goal in the final) are relatively rare and a lack of a goal could just point to the opposing team's inability to convert due to a poorly organised or a lack of opportunism from their strikers. I'm not sure what you mean with the last one, but I think this could be a nice one - if you mean "times you lost possession in your own half". A profligate midfield and defence is bound to ship goals, I doubt there are many teams that can either fight back after trailing by a goal or two or score enough to maintain a reasonable buffer. I applaud the effort though - it takes more creativity and care to think of some new angles (like you did) than to think of some possible counter examples (like I did)!
- dmichulke 8y ago
  Thank you for the warm words, I guess the reason is my occupation plus the fact that I just spend my last few weeks watching many games with family and friends. > I'm not sure what you mean with the last one, but I think this could be a nice one - if you mean "times you lost possession in your own half" Almost, England lost the ball frequently (> 50+x% with a large x AFAI could see) due to the keeper sending out long balls. I'd like to measure that somehow. Could be done via number of seconds in possession after a goal kick, an indicator whether a hypothetical 85% marker of the field was reached or measuring whether the ball was at least 5x successfully passed (or resulted in a goal).
  - smcl 8y ago
    Ahhhh I see. Actually this is something I've really been curious about myself - whether the better strategy overall for a keeper returning the ball into open play (from goal kick or from hand) is to just boot it as far up-field as possible or passing it short to one of the defence or midfielders sitting deep. Interestingly something like this is a tactic used in Rugby (https://www.youtube.com/watch?v=cbti6mLvSJs https://www.youtube.com/watch?v=cbti6mLvSJs). I used to play a lot of football when I was younger and at our level (waaaay down the scottish league pyramid) against tired, hungover or generally weak opposition, keeping them under pressure by dominating the territorial game but sacrificing possession was criminally underrated. Usually if you could keep hammering them for 60 minutes and had the legs to step up a gear in the last 30 or so you could grab a valuable goal or two :-)
lagadu 8y ago
> - creativity in offense (maybe measurable via "target missed from close distance" + "ball passed front of the goal") This one would benefit possession-based teams, so it would fail to give decent odds to the current world and european champions (France and Portugal respectively) which don't play possession. Of course it's possible they're outliers but we'll never know.

boomboomsubban 8y ago

The World Cup is about the worst sporting event for data led predictions like this, far too much can rely on a few events that are basically a coin flip. It would be interesting to see how the predictions went for something like the Premiere League tables.

fwdpropaganda 8y ago
> far too much can rely on a few events that are basically a coin flip Can you give us some examples?
- gtr 8y ago
  As there are relatively few goals, anything that can turn a goal into not a goal or vice versa can have a massive impact on the game. For example the penalty decision against Croatia in the final. Another thing that adds to the randomness is the chance that a key player may be sent off or injured.
  - systoll 8y ago
    To make that be specific -- in 44 of 64 games, and in every single penalty shoot-out, turning one goal into not a goal or vice-versa would've changed the outcome.
    - ufo 8y ago
      It is not so simple because goals in football are not independent from each other. A team that scores first has the opportunity to play more cautiously and go for more counter attacks.
- zaphirplane 8y ago
  Someone deciding to handball the ball out of the way, a tackle that goes harder in a temper flare. A penalty that is saved/missed
- kryptiskt 8y ago

lowkeyokay 8y ago

If anything, this is a clear illustration of poor use of probabilistic prediction. When used for investments you have many outcomes. If the model is any good, you will most of them right. In the World Cup you have very few. Even if you count all games played. Definitely not excusing Goldman Sachs here, they should have known better than to try to predict this. There was only a tiny chance this could be great advertisement for their model.

Ntrails 8y ago
> they should have known better than to try to predict this. There's no downside, only free publicity. If they, by good fortune and a following wind, get it right - then the publicity is incredible. If it's wrong they laugh and say "well, better stick to predicting what we're good at!" and they still get a shitload of headlines and awareness of their product. This was not a mistake.
- jamespo 8y ago
  Well, there's the downside of articles like this pointing out they've had 4 years to work on their models and they've got worse

blattimwind 8y ago

This site is a good counter-example for website optimization: While it uses many assets, so a CDN domain makes sense, it spreads them out thinly. It loads over 100 CSS files, most of which are below 1K. Similarly it loads approximately 30 JS scripts, most of which are just a few K each. This is mitigated to a large extent by using HTTP/2.0, which permits a few dozen or so parallel requests, but it still means that a repeated load of the page takes 2-3 seconds. (Without HTTP/2.0 this probably takes ages, since browsers open only a few connections to each origin at most). There is also almost no difference between reloading with and without the cache.

barrkel 8y ago

I put money on Belgium (12.0 decimal odds) and Croatia (15.0) after the group stages, where some form was visible, combined with knowledge that they had some of the world's best players. The odds shortened as the tournament progressed, I was able to hedge as the shortened odds made lay betting profitable. (High variance in football outcomes means there's no guarantee of profit, I don't bet big sums.)

anoncoward111 8y ago
This answer is very useful and contains proper strategy advice :) If someone were to bet during the round of 16, if someone were to bet $1 on the bottom 8 and $2 on the top 8, the strategy would most likely yield a small profit or a small loss, rather than a total loss.

geraldbauer 8y ago

PS: If you want to build or train your own model or make predications, you can find open (structured) data about all world cups at the football.db, see https://github.com/openfootball/world-cup https://github.com/openfootball/world-cup and https://github.com/openfootball/world-cup.json https://github.com/openfootball/world-cup.json Enjoy the beautiful game.

pelasaco 8y ago

"Of course, past data don’t always predict the future; Goldman Sachs never tells clients to make decisions solely on the basis of its models’ findings" in the real world Goldman Sachs would manipulate the games to make it looks like they got it right and their clients got richer.

denzil_correa 8y ago

The "Ludic Fallacy" strikes again [0]. > The ludic fallacy, identified by Nassim Nicholas Taleb in his 2007 book The Black Swan, is "the misuse of games to model real-life situations." ... > The alleged fallacy is a central argument in the book and a rebuttal of the predictive mathematical models used to predict the future – as well as an attack on the idea of applying naïve and simplified statistical models in complex domains. According to Taleb, statistics is applicable only in some domains, for instance casinos in which the odds are visible and defined. Both Taleb's books, "The Black Swan" and "Fooled by Randomness" are an interesting take for such models. Meanwhile, most economists know about "Knightian Uncertainty" [1] which talks about differentiation of risk and uncertainty. > "Uncertainty must be taken in a sense radically distinct from the familiar notion of Risk, from which it has never been properly separated.... The essential fact is that 'risk' means in some cases a quantity susceptible of measurement, while at other times it is something distinctly not of this character; and there are far-reaching and crucial differences in the bearings of the phenomena depending on which of the two is really present and operating.... It will appear that a measurable uncertainty, or 'risk' proper, as we shall use the term, is so far different from an unmeasurable one that it is not in effect an uncertainty at all." [0] https://en.wikipedia.org/wiki/Ludic_fallacy https://en.wikipedia.org/wiki/Ludic_fallacy [1] https://en.wikipedia.org/wiki/Knightian_uncertainty https://en.wikipedia.org/wiki/Knightian_uncertainty

fwdpropaganda 8y ago
Damn, do I disliked Nassim Taleb. I don't think I've ever heard him say anything deep. That wikipedia article is an excellent. In [0] you have the following: > The ludic fallacy, identified by Nassim Nicholas Taleb in his 2007 book The Black Swan, is "the misuse of games to model real-life situations." And he gives an example of this: > One example given in the book is the following thought experiment. Two people are involved: > Dr. John who is regarded as a man of science and logical thinking > Fat Tony who is regarded as a man who lives by his wits > A third party asks them to "assume that a coin is fair, i.e., has an equal probability of coming up heads or tails when flipped. I flip it ninety-nine times and get heads each time. What are the odds of my getting tails on my next throw?" > Dr. John says that the odds are not affected by the previous outcomes so the odds must still be 50:50. > Fat Tony says that the odds of the coin coming up heads 99 times in a row are so low that the initial assumption that the coin had a 50:50 chance of coming up heads is most likely incorrect. "The coin gotta be loaded. It can't be a fair game." > The ludic fallacy here is to assume that in real life the rules from the purely hypothetical model (where Dr. John is correct) apply. Would a reasonable person bet on black on a roulette table that has come up red 99 times in a row (especially as the reward for a correct guess is so low when compared with the probable odds that the game is fixed)? So Nassim Taleb wanted to discuss "using games to model real-life situations" and to demonstrate the pitfalls he uses two characters. He _portrays_ the characters as "man of logical thinking" vs "man who lives by his wits", but as we'll see he's missing one dimension to his characterization. The first problem here is that implicitely he's suggesting to the reader that the decisions of the "man of logical thinking" represent the pitfalls of "applying games to model real-life situations", whereas the the other guy's decision represent.... it's not specified, but clearly has a better outcome. The second problem, is that he conflates "applying something you read on some textbook to real life without thinking" with "modelling real-life". He suggests to the reader that those two people are actually "logical" vs "instinct", but they're not. They're a dumb guy who knows maths vs a smart guy who doesn't know math. _Obviously_ real-life is more complex than your textbook examples, and so the smart guy is going to win because his fuzzy heuristics beat the first guys decisions which are optimal within his flawed model. An actual smart and logical person would update his model based on new evidence (i.e. "I was told that this coin was 50-50 but actually the chance of what I just saw is so small that it's more likely that I was just lied to") and then use maths to make predictions and beat the guy who's smart but doesn't know math. So ironically, he wants to portray the dangers of using over-simplified models and to do that he uses an example where he obscured one dimension. Nassim Taleb is really good a rhetoric but light on substance. [0] https://en.wikipedia.org/wiki/Ludic_fallacy https://en.wikipedia.org/wiki/Ludic_fallacy

jasode 8y ago

Leonid Bershidsky and a lot of other journalists laughing at Goldman Sachs' incorrect predictions seem to miss the point. The World Cup predictions from Goldman Sachs (and also UBS) are a form of recreation and entertainment with machine learning. It's an expression of quant nerd humor. Analogous intellectual games would be engineers devising ridiculous Rube Goldberg contraptions[1] or programmers building "enterprise" FizzBuzz[2]. (I think it would add to the fun if GS uploaded their raw data and models to Github for others to play with.) >It certainly didn't predict the final opposing France and Croatia on Sunday. True, but it did predict France having better chance winning overall but was handicapped by a tougher draw. It also predicted France beating Croatia in round 16 instead of the final. The pdf says: >While Germany is more likely to get to the final, France has a marginally higher overall chance of winning the tournament, [1] https://en.wikipedia.org/wiki/Rube_Goldberg_Machine_Contest#Past_tasks https://en.wikipedia.org/wiki/Rube_Goldberg_Machine_Contest#... [2] https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...

learnstats2 8y ago
On the other hand, this is a predictive task that has defined outcomes and clear historical data - by my understanding, it is easier than commercial uses of machine learning [at least, easier to measure the effectiveness]. It's also Goldman Sachs and UBS choosing to attach their names to these and stake some reputation on these predictions. If they had hit the bullseye, they would be lauding these results.
- sgt101 8y ago
  Yup, the worst thing is that if they had got it right it would have been more or less due to pure chance, and it would have led to business flowing their way!
- CoryG89 8y ago
  It may be easier to measure the effectiveness (give a confidence level for the prediction), but just because there is clear historical data and defined outcomes, that does not mean you will be able to predict a particular outcome with any high level of certainty. For example, imagine a tournament with a large number of participants, where the winner is picked simply by fairly choosing a single random participant. If I then gave you all the perfect historical data going back decades, you could do statistical analysis and determine that the winner is completely random and therefore the probability of success, for any particular participant, is p~=(1/n), where n is the number of participants. Your confidence in correctly predicting any particular outcome will drop as n rises. Not everything can be easily predicted just because you have enough data.

raverbashing 8y ago

People conflate statistics with actual results more often than not and I think those reporting on such stories and maybe even the original authors might fall for this. It was not wrong to say Hillary had a 95% chance of winning the presidential election, but the confidence was low and that value still allowed for the opposite result to happen. Also football has a lot of variance concerning team capability and end results. The better team might (and does) lose often, especially when going to penalty shoots. With basketball, the stronger team will be easily scoring more in most cases.

kgwgk 8y ago
> had a 95% chance [...] but the confidence was low So she had 95% chance of winning with 50% probability or what?
- raverbashing 8y ago
  it is captured in the 95% but that was probably a bit overestimated (and there are always unknown biases and improbable events can happen) What's wrong is thinking 95% chance of winning means they will win
  - kgwgk 8y ago
    So it was not captured in the (overestimated) 95% :-) But I agree, the most likely thing, even if the probability was perfectly known, is not always what happens.
- frockington 8y ago
  60% of the time, it works every time - Brian Fantana
- fny 8y ago
  So this is something that people don't seem to grok quite well, and it really depends on the type of statistical analysis used. Say you make the assumption that the quantity being estimated is truly fixed: that there's some true value for the force of gravity or some true value for the number of people that vote for X or Y. The second assumption that comes along is that the stochasticity observed comes from your perspective of observation, and not from the ground truth. To be more blunt, you know that of all the observations you make 95% of them have the probability of yielding the result observed... but the ground truth is still fixed. Gravity has a fixed quantity, despite your experimental error, and you may have been lucky enough to observe it in your sample. Predicting elections with frequentist methods has this same characteristic, except the observed quantity itself shapeshifts and even lies... so then there are other complications that need to be dealt with. This is where that 50% feeling comes from. There are two outcomes, one will be true. You're data analysis just tells you that if you repeat your procedure, you'd expect 95% of those result to give you the outcome you observed.

tirumaraiselvan 8y ago

It's a fools errand to predict high variance events like football games.

pbhjpbhj 8y ago
Only predict events that are easy to predict, never fail!

anonu 8y ago

People love to beat up on these companies because of this stupid world cup prediction. Yes, Goldman is a giant vampire squid wrapped around the face of humanity (Matt Taibi quote). But it turns out it's really just great marketing for their research teams. Also, I've seen some people say (not in this forum) that banks now look stupid because they're in the business of making predictions and they can't even get the world cup right. Guess what? Banks make no money on predictions. They make money on flows and taking spreads on trades they do with clients. Any research or prediction is meant to be a catalyst for that trade.

chopin 8y ago
I am pretty sure Banks make money on predictions if they get people on following them.
- anonu 8y ago
  Yes... This is less a prediction and more a legal form of front running.
  - throwawaymath 8y ago
    In what way is this a legal form of front running?
    - tedunangst 8y ago
      Everything Goldman Sachs does is front running and it's legal because they bribed all the regulators. QED.
jasode 8y ago
>Banks make no money on predictions. They make money on flows and taking spreads on trades they do with clients. You're mostly right but to further clarify, an investment bank like Goldman Sachs has revenue from mostly "market making" spreads but it does also have activities that depend on predictions such as their proprietary trading (before the Volcker Rule shut them down) and their GSAM (Goldman Sachs Asset Management) fund. The GSAM is basically a hedge fund for their wealthy clients' money. They will run predictions on macro trends on data like interest rates, commodities, indexes, etc to help them pick stocks for their portfolio. As the pdf noted, the World Cup data models and simulations came from Adam Atkins of GSAM.

yk 8y ago

> And in any case, the model only generated probabilities of winning a game and advancing, and no team was given more than an 18.5 percent chance of winning the World Cup. > [...] > But Goldman Sach’s misfire is perhaps the most curious. The model said, that there is a lot of uncertainty, and as it happens, it was entirely correct. A World Cup chance of 18.5 percent means, that 4 out of 5 times the team will not win, and that that is the highest chance does not say much about the model. And in general this is one instance of the well practiced journalistic technique to wait for results first and then define a bar afterwards to criticize the results according to standards that did not exist when the performance happened. (I guess in this case it is even worse, we could construct a reasonable test of the model performed, I have the suspicion that that was in the original paper and that the journalist either did not understand it, or, more likely, choose to ignore it in favor of writing a better story.)

pbhjpbhj 8y ago
Uncertainty is a truism; that's why people want to use a prediction algo. Did the system so better on results it was more certain about? Predicting the result of an A or B contest the bar is already defined. Either the system gets it right or doesn't, if it gets it right more often than not then (despite this being poor grounds mathematically, on a small result pool) popular press will report it as successful. IMO if matches become easy to predict then rules will change to reduce that predictability.
- LeifCarrotson 8y ago
  > Predicting the result of an A or B contest the bar is already defined. I disagree: If team A has a 10-30% chance of winning, and A pulls off the upset, the correct answer was not "A Wins" it was "B has a 70-90% chance of winning". For Goldman Sachs' investments, the bar is not to predict that A wins or that B wins, it's to predict the probability and variance regarding which team will win. Of course, from a single upset game, it's impossible to tell whether these estimates are correct. You'd need to see the success or failure of many trials.
  - 8y ago

kulu2002 8y ago

Good... There was this discussion thread few days back on HN https://news.ycombinator.com/item?id=17509407 https://news.ycombinator.com/item?id=17509407 Did this investment bank use same set of algorithms that they use for financial predictions? ...And then I remember there was this Octopus[1] who used to predict winners with 85% accuracy [1]https://en.wikipedia.org/wiki/Paul_the_Octopus https://en.wikipedia.org/wiki/Paul_the_Octopus

tomelders 8y ago

While I agree that it's somewhat silly to try and predict a word cup winner like this (and I suspect it was just a bit of fun anyway), there is one other reason that could explain why all these attempts got it so wrong. Cheating. Before people start booing, let's not forget where this tournament is being held, and all the other nefarious things that country has been up to recently.

teamk 8y ago
FIFA has been corrupt for decades. Although supposedly its been cleaned up since Blatter was removed, it is doubtful the institutional corruption has been eliminated completely. The only question is how pervasive it is.

msravi 8y ago

Duh. Looks like there's a fundamental misunderstanding of how statistics works all around. The probability of an event does NOT predict a particular outcome. Ever. It only says that if the experiment is performed again and again and again, like a few thousand times, then X% of those will match that probability. If I toss a fair coin you cannot predict the next outcome. You can only say that if I toss the coin a 1000 times, then close to 500 are going to turn up heads, and another 500 are going to turn up tails. It was stupid of Goldman Sachs or whoever to predict an outcome. It was stupid of anyone else to lend credence to that prediction. Hopefully, Goldman Sachs is not relying on prediction of singular outcomes to make their investment decisions. I don't think they are. Probably just marketing brouhaha to ride the soccer wave. Although I'm not sure if that worked as expected.

pbhjpbhj 8y ago
I agree completely with your opening remarks. >"You can only say that if I toss the coin a 1000 times, then close to 500 are going to turn up heads, and another 500 are going to turn up tails." Sometimes you can do that and every single flip will be heads. It's unlikely, and across zillions of universes you'd only find it once - but we don't have a pool of universes that we can sample statistically.
Sean1708 8y ago
> It was stupid of Goldman Sachs or whoever to predict an outcome. If you read the actual report they did[0], they never claimed that any single outcome was more than 18.5% likely. [0]: http://www.goldmansachs.com/our-thinking/pages/world-cup-2018/multimedia/report.pdf http://www.goldmansachs.com/our-thinking/pages/world-cup-201...

rdlecler1 8y ago

In the world of models increasing precision for not necessarily increase accuracy.

cascom 8y ago

Isn’t this a little like flipping a coin four times - getting heads four times in a row, and looking at your friend and saying “but you told me the odds were 50/50 each flip?!”

thousandautumns 8y ago
Yes, it is.

crispyambulance 8y ago

I am somewhat shocked that GS would jump into the prediction business of the World Cup, even as joke. The risk of people getting the wrong idea about the prediction and GS itself is too great, even with a perfectly defensible model. This is an enterprise for bookies, not Goldman Sachs.

TuringNYC 8y ago
FYI - I worked at Goldman Sachs and then a hedge fund for a decade. On the Capital Markets / Trading side, you are literally a bookie. In fact the nomenclature is "you have a book." You are setting trading spreads based on where you think things will go. Depending on the market, your work may be more or less statistical and you're trying to gain a statistical advantage.
- Maro 8y ago
  Off-topic: were you able to retire after that decade?
  - TuringNYC 8y ago
    Short Answer: No, but very comfortable. Long Answer: Full retirement is hard, Healthcare is a pain in the US. You cant really "save" for it in the US, it can swallow all your savings, so you'll always need some job or another to cover healthcare and catastrophic needs. That said, you can very easily down-shift once you have a house, savings, etc. Longer Answer: Could have, if I wanted to -- but you always give something up in exchange. These jobs will take everything you give them (time, health, life) and give back a decent percentage (income.) But you cannot easily dial up or down the work, it comes in chunks and you have to complete it. My life was increasingly unhinged at 27 and I decided to jump off the treadmill after seeing a colleague continue to work through his mother's terminal illness and death. Inertia and greed are a toxic combination. Numerous colleagues were on drugs, uppers, anti-depressants, etc. One died from stress (heart attack in his 30s.) I chose to get married, have two kids. I switched to a pure tech job (now an ML product owner at a Series A pure tech firm.) We have dinner together almost every single day. Weekends are completely ours. We go to the park most warm days. We take 3 to 4 vacations a year, many with my mom as well. There is a decent amount of work but I can choose when to do it (unlike Wall St.) and the work is longer term and I can dial it up/down as family requires. I sit outside and read during lunch. I turn off the markets when i step out of work. Many of my colleagues were easy millionaires by ~30 and multi-mullionaires if they stuck till their mid 30s and were focused. Many others blew through their bonuses (or snorted it away) and ended up with nothing and just live bonus to bonus. It also depends on the job (business/deal side vs quant side vs tech -- the money is a waterfall across the 3 sections.) As with all industries, you get ripped off if you dont fight for your share of the pie. Plenty of people avoid conflict and life comfortable lives and nothing more. I also saw several C++ programmer/manager earn double digit millions of dollars over several years, one earned over 100MM USD over the course of his time at the hedge fund (public records, check out AIG-FP https://en.wikipedia.org/wiki/AIG_bonus_payments_controversy https://en.wikipedia.org/wiki/AIG_bonus_payments_controversy) I think I did well and hopefully dont have to worry about poverty anymore. You either get lucky (early FB employee, hot product at xyz.com) or you have to give up something. I havent seen someone truthfully say they got both money and family and happiness all together.

known 8y ago

GarbageIn = ML = GarbageOut

known 8y ago

I worked in GS; Soccer/football prediction is not their forte

iainmerrick 8y ago

Thanks to the use of more granular data, made possible by AI, this year’s model should have worked better than the 2014 one. If anything, it worked worse. "If anything"? All the results are available, so it would be easy to put a precise number on this. Measure the Bayesian regret, or just report the winnings if you had used the GS model to bet on the outcomes. Unless it reports some concrete numbers, this article is garbage. It doesn't report any concrete numbers.

Keyframe 8y ago

It's as good time as any to plug in EA's simulation results: https://www.easports.com/fifa/news/2018/ea-sports-predicts-world-cup-fifa-18 https://www.easports.com/fifa/news/2018/ea-sports-predicts-w...

Sean1708 8y ago

In case anyone was interested here is a table of how likely the model thought each team was to make it through any particular stage[0] along with the stage that that team went out in and the probability that the model gave for that particular outcome (i.e. [probability of making it through the final stage they made it through] - [probability of making it through the stage they went out in]). Groups Round_16 Quarters Semis Finals Out_In Probability Brazil 87.5% 60.8% 42.0% 27.9% 18.5% Quarters 18.8% France 81.4% 58.4% 36.6% 19.9% 11.3% Won 11.3% Germany 80.5% 49.5% 30.5% 18.8% 10.7% Groups 19.5% Portugal 75.2% 52.8% 32.2% 17.3% 9.4% Round_16 22.4% Belgium 78.5% 51.1% 27.7% 15.8% 8.2% Semis 11.9% Spain 72.3% 50.1% 28.8% 15.4% 7.8% Round_16 22.2% England 73.1% 46.6% 24.4% 13.4% 6.5% Semis 11.0% Argentina 79.7% 44.2% 24.1% 11.8% 5.7% Round_16 35.5% Colombia 74.9% 37.3% 17.0% 8.5% 3.7% Round_16 37.6% Uruguay 74.4% 34.6% 17.2% 7.2% 3.2% Quarters 17.4% Poland 68.5% 30.5% 12.8% 5.8% 2.3% Groups 31.5% Denmark 47.8% 26.3% 12.4% 5.2% 2.0% Round_16 21.5% Mexico 52.0% 23.2% 10.5% 4.9% 1.9% Round_16 28.8% Sweden 45.9% 19.4% 8.3% 3.7% 1.3% Quarters 11.1% Iran 35.4% 18.1% 7.2% 2.6% 0.8% Groups 64.6% Peru 37.3% 17.2% 6.8% 2.5% 0.8% Groups 62.7% Australia 33.5% 15.4% 6.3% 2.3% 0.7% Groups 66.5% Russia 47.9% 16.3% 6.0% 2.0% 0.7% Quarters 10.3% Croatia 49.8% 16.9% 6.3% 2.1% 0.6% Finals 4.2% Switzerland 52.8% 15.9% 6.1% 2.0% 0.6% Round_16 36.9% Iceland 45.2% 15.1% 5.6% 1.8% 0.5% Groups 54.8% Costa_Rica 36.8% 13.3% 4.7% 1.6% 0.5% Groups 63.2% Serbia 32.9% 12.1% 4.5% 1.5% 0.5% Groups 67.1% Japan 36.5% 12.8% 3.8% 1.3% 0.4% Round_16 23.7% Saudi_Arabia 43.4% 12.7% 4.2% 1.3% 0.4% Groups 56.6% Tunisia 35.2% 13.3% 4.1% 1.3% 0.4% Groups 64.8% Egypt 34.4% 8.7% 2.5% 0.7% 0.2% Groups 65.6% South_Korea 21.6% 5.9% 7.1% 0.5% 0.2% Groups 78.4% Morocco 17.1% 6.8% 1.8% 0.5% 0.1% Groups 82.9% Nigeria 25.2% 6.5% 1.7% 0.4% 0.0% Groups 74.8% Senegal 20.1% 4.9% 1.2% 0.3% 0.0% Groups 79.9% Panama 13.2% 3.3% 0.5% 0.1% 0.0% Groups 86.8% [0]: Exhibit 2 in http://www.goldmansachs.com/our-thinking/pages/world-cup-2018/multimedia/report.pdf http://www.goldmansachs.com/our-thinking/pages/world-cup-201... Edit: Fix copy-paste errors and atrocious maths.

kgwgk 8y ago

The predictions were not so bad. At least one of the favourites won in the end. GS had France winning with 11.3% probability, second to Brazil with 18.5%. UBS was less fortunate, they had Germany (24%), Brazil (19.8%), Spain (16.1%) and England (8.5%) before France (7.3%). I compared the logloss for their predictions with the "uniform" benchmark (giving each team 1/32 probability of winning, 1/16 probability of getting to the finals, etc) and the results are the following (if I transcribed the data properly): Getting to second round: GS: 0.495 UBS: 0.495 bench: 0.693 Getting to quarter-finals: GS: 0.463 UBS: 0.459 bench: 0.562 Getting to semi-finals: GS: 0.310 UBS: 0.327 bench: 0.377 Getting to final: GS: 0.231 UBS: 0.269 bench: 0.234 World-cap winner: GS: 0.097 UBS: 0.113 bench: 0.139 The performance of the models was ok until Croatia got to the finals. This hurt specially UBS, who predicted less than 0.9% probability of such an event (compared to 2.1% in Goldman's model). Edit: these would have been the "best case" scores (if the high-probabilty teams had classified to each round, ignoring that this may be impossible due to the structure of the tournament): GS: 0.432 0.302 0.220 0.141 0.079 UBS: 0.365 0.251 0.176 0.111 0.070 UBS could potentially achive lower logloss metrics because it had more extreme predictions.

corpMaverick 8y ago

Soccer is a sport with a big random component. This is probably why it is so exciting. An average team can beat a better team. The reason is easy to see. The game can be decided by one, two or three key plays. Compare that to basket ball. To win a game you have to consistently score more and defend better. Rarely the game is decided by one or two plays. That only happens when the game is already very tight.

patagonia 8y ago

Financial modeling is about risk adjust return. Because GS knows they can not determine with certainty the outcome of a given investment, they diversify and hedge. Most of all, GS is a market maker, the equivalent of a bookie. To say that GS’s models “didn’t come close” is to ignore all the ways in which such a grading scheme is different than GS’s actual business model. If their WC prediction efforts acted as anything more than a fun spirited PR project, it was likely that GS wanted to somehow keep its employees engaged and adding business value during the WC which they otherwise would have been certainly watched all month.

gesman 8y ago

If GS would need to bet money - their actual business model would likely be to sell a bit of each higher probability losers (less risk) vs. buy big on a projected winners (higher risk).

deleted 8y ago

[deleted]

rcdmd 8y ago

This article didn't compare the Goldman Sachs model to any other models-- why not compare it with sports betting odds? Would Goldman have made or lost money betting their model was better than the crowd?

sunstone 8y ago
Or compare it with the fivethirtyeight blog predictions.

rossdavidh 8y ago

In addition to the many other problems with this article, I would like to point out that if, somehow, Goldman Sachs had managed to create a model that could accurately predict the results, the game of soccer would have to be changed to make it more unpredictable somehow. It is intrinsic to the nature of sport that, in order to be entertaining, there has to be a realistic chance for more than one team to win. Not many people (even from the winning country) would bother watching if it were accurately predictable.

vl 8y ago

>Soccer, with the many factors that affect game outcomes — players’ injuries and intra-team conflicts, the refereeing, the weather, coaches’ errors and moments of inspiration — remains only a tightly-regulated game involving a few dozen people. The behavior and performance of big corporations, entire industries and nations is arguably even more difficult to model based on data about the past. Author misses the way models work entirely, the larger the entity, the more statistics and averages kick in, and as a result, better model can be built.

Donald 8y ago
Depends on the complexity of the interactions between variables. There are plenty of examples where we have excellent local models, but make (comparatively) worse prediction at scale. A pretty classic example is biology - we have excellent knowledge about how genotypes work and their interactions in cells, but models of phenotypes are typically expensive, error-prone, or non-existent.

IkmoIkmo 8y ago

You'd have to run this world cup thousands of times by simulation, running it a single time and determining the results are not in line with the model is meaningless and silly. It's as silly as saying my claim for the odds of nearly perfectly modelling a coin toss (approximately 50/50%) is wrong because a series of 10 coin tosses show different results from my model. The model is not any less correct.

hsienmaneja 8y ago

They don’t have an edge like they do in their bread and butter markets, combined with a small sample set == high probability of a single year of sports predictions falling over like this