Socratic Form Microscopy

Grading My 2018 Predictions

by Zach Jacobi in Falsifiable

In what is becoming a New Year’s tradition, let’s look at how accurately I predicted the future a year ago.


  1. Liberals remain ahead in the CBC Poll Tracker seat projection – 70%
  2. Trudeau has a higher net favorability rating than Andrew Scheer according to the CBC Leader Meter on January 1, 2019 – 80%
  3. Marijuana is legalized in time for Canada Day – 60%
  4. Marijuana is legalized in 2018 – 90%
  5. At least one court finds the assisted dying bill isn't in line with Carter v Canada – 70%
  6. Ontario PC party wins the election – 60%
  7. The Ontario election results in a minority government – 80%
  8. The Quebec election results in a minority government – 80%
  9. No BC snap election in 2018 – 90%
  10. No terrorist attack in Canada that kills > 10 Canadians in 2018 – 90%
  11. More Canadian opioid poisoning deaths in 2018 than in 2017 – 60%
  12. Canada does better at the 2018 Winter Olympics (in both gold medals and total medals) than in 2014 – 90%
  13. Canada does not win a gold medal in men's hockey at the 2018 Olympics – 70%
  14. Canada does win a gold medal in women's hockey at the 2018 Olympics – 51%


  1. Trump announces that the US is pulling out of NAFTA and begins the process of putting the US withdrawal into motion – 51%
  2.  Less than 100km of concrete wall on the border with Mexico will be constructed – 90%
  3. No registry of Muslims created – 90%
  4. Congress doesn't take action to extend DACA – 80%
  5. No department of the Federal Government is eliminated – 90%
  6. There isn't a government shutdown before the midterm elections – 60%
  7. Democrats take back the house in the 2018 midterm elections – 80%
  8. Democrats take back the senate in the 2018 midterm elections – 60%
  9. Mueller's investigation finishes in 2018 – 60%
  10. Impeachment proceedings aimed at Trump are not started in 2018 – 80%
  11. Trump is still president at the end of 2018 – 90%
  12. No terrorist attack in America that kills > 10 Americans – 70%
  13. No terrorist attack in America that kills > 100 Americans – 90%
  14. Susan Collins doesn't get the Obamacare stabilization measures she was promised – 70%
  15. More US opioid poisoning deaths in 2018 than in 2017 – 80%

South America

  1. FARC peace deal remains in place on January 1, 2019 – 80%
  2. The black market exchange rate for Venezuelan Bolivars is above 110,000 to the US dollar on January 1, 2019 (as measured by DolarToday) – 80%
  3. Inflation in Venezuela is above 100% for the year of 2018 (as measured by DolarToday) – 90%
  4. United Socialist party retains control of the Venezuelan presidency in 2018 – 90%
  5. Protests (and the official response to those protests) result in more than 100 fatalities in Venezuela in 2018 – 60%
  6. Protests (and the official response to those protests) do not result in more than 1000 fatalities in Venezuela in 2018 – 70%
  7. Major Venezuelan opposition groups do not enter any sort of power sharing agreement with the Venezuelan regime in 2018 – 80%

Middle East

  1. No Israeli politician is indicted by the ICC over settlement activity in 2018 – 90%
  2. There isn't an election in Israel in 2018 – 80%
  3. US does not physically relocate its embassy to Jerusalem in 2018 – 90%
  4. No Palestinian led Intifada in Israel that results in the deaths of >1000 combined attackers, security forces, and civilians (this is a conflict characterized by suicide bombing and police responses) – 70%
  5. No Israeli led operation in the West Bank or Gaza that results in the deaths of >1000 combined soldiers, civilians, and militants (this is a conflict characterized by rocket fire and military strikes) – 70%
  6. Fatah and Hamas do not meaningfully reconcile in 2018 (e.g. Fatah still doesn't control Gaza by January 1, 2019) – 51%
  7. No significant resurgence in ISIL in 2018 (e.g. it does not gain territory over the next year) – 80%
  8. Fewer casualties in the Syrian Civil War in 2018 than in 2017 – 70%
  9. No power sharing agreement or durable ceasefire (typified by the three months following the agreement each having less than 500 fatalities) in Syria in 2018 – 80%
  10. Bashar Al Assad is still President of Syria on January 1, 2019 – 90%
  11. Protests in Iran do not result in more than 1000 fatalities by the end of 2018 – 70%
  12. Protests in Iran do not result in more than 100 fatalities by the end of 2018 – 51%
  13. Hassan Rouhani is still President of Iran on January 1, 2019 – 90%
  14. No new international sanctions against Iran (does not include adding new organizations or individuals to old categories and requires coordinated participation of at least two countries) – 80%
  15. No new US sanctions against Iran (does not include adding new organizations or individuals to old categories) – 51%
  16. No attack on the Iranian nuclear program by Israel – 90%
  17. Iran does not withdraw from the deal limiting its nuclear program – 90%
  18. Conditional on Iran remaining in the nuclear deal, inspectors find no evidence of violations after the deal began – 90%
  19. Yemen Civil War continues – 60%
  20. Saudi Arabia pulls troops out of Yemen – 51%
  21. Mohammed bin Salman either remains as crown prince of Saudi Arabia, or becomes king (i.e. no coup or succession shake-up) – 80%
  22. Rockets fired from Yemen cause casualties in another country – 51%
  23. No resolution or lifting of embargo in the Qatar crisis – 80%
  24. OPEC production cuts continue through to the end of 2018 – 60%


  1. No power sharing between ZANU-PF and the opposition will happen in Zimbabwe before the elections (if they occur) in 2018 – 80%
  2. Zimbabwe will hold election in 2018 – 70%
  3. No peace deal ends South Sudan fighting – 70%
  4. Libya still has two rival governments on January 1, 2019 – 70%
  5. No protests, riots, or rebellion in Egypt that kills >100 people in a one week period – 80%
  6. No protests, riots, or rebellion in Tunisian kills >50 people in a one week period – 90%
  7. No terrorist attack in Tunisia kills >20 people – 80%
  8. Zuma is not impeached in 2018 – 51%


  1.  Inflation rate in Japan still remains below 1% in 2018 – 70%
  2. Japanese constitutional reform (removing pacifism) does not occur in 2018 – 51%
  3. China will not deploy its military against Taiwan or Hong Kong in 2018 – 90%
  4. North Korea will test a submarine launched ballistic missile in 2018 – 70%
  5. North Korea will not test nuclear weapons or launch any missiles during the 2018 Olympics – 80%
  6. North Korea will test a nuclear weapon in 2018 – 51%
  7. No country will attempt to shoot down a North Korean missile test in 2018 – 80%
  8. If there is an attempt, it will succeed – 51%
  9. North Korea tests a missile that is judged by experts at 38 North as likely able to carry a plausible North Korean nuclear weapon to the United States – 60%
  10. No current member of China's Politburo Standing Committee visits North Korea in 2018 – 70%
  11. No meeting between Kim Jung-un and Moon Jae-in in 2018 – 90%


  1. No resolution to the crisis in Ukraine – 80%
  2. Russian GDP growth is less than 3% – 80%
  3. No gain of greater than 20% in the value of the ruble vs. the dollar – 70%
  4. Sanctions against Russia are not significantly rolled back (e.g. sanctions remain in place against Rosneft, Novate, Gazprombank and Vnesheconombank by all members of the G7 remain in place at the end of 2018) – 90%
  5. Angela Merkel remains chancellor of Germany – 60%
  6. Germany holds another election before a government can be formed – 51%
  7. No date set for another Scottish referendum in 2018 – 80%
  8. Teresa May remains prime minister of the United Kingdom – 70%
  9. The UK does not terminate the process of Brexit in 2018 – 90%
  10. There is no final Brexit withdrawal deal reached in 2019 (Donald Tusk wishes to have one by October) – 51%
  11. No snap election/vote of no-confidence in the UK in 2018 – 80%
  12. Poland's EU voting rights aren't suspended – 90%
  13. Poland and Hungary continue to refuse to accept migrant quotas – 90%


  • I originally worried that I'd missed the mark with my Venezuelan devaluation predictions (South America #2), as DolarToday gave an exchange rate of 730.29 BsS. to the dollar. But it turns out the BsS. is a new currency, with each one equivalent to 100,000 of the old bolivars. This means the current exchange rate is 73,029,000 old bolivars to the USD. Inflation was 4305%. It is now an unfortunate fact that some online video game currencies are worth more than the old Venezuelan bolivar.
  • For Middle East #3, I got conflicting information from Wikipedia as to whether the embassy was officially moved, or just moved in name pending the ambassador finding a place to live in Jerusalem. I decided that it should count as moved for all practical purposes.
  • With respect to Middle East #15, I believe that the US withdrawing from the P5+1 deal and restarting sanctions should probably count as new sanctions.
  • I think Africa #8 is technically correct, because Zuma wasn't impeached, but forced out by his party. Still, I think I will aim to make broader predictions about embattled leaders leaving in the future.
  • I calculate Japan's inflation rate at around 1.05% based off the most recent statistics I could find, although this may need to be adjusted once December's numbers are out and further adjusted once we have the year-end numbers. I got Asia #1 wrong, but not by much!
  • I was worried that no member of the standing committee had visited North Korea, potentially making Asia #10 a technical success, despite it obviously being falsified in spirit (by the three meetings in China by Xi Jinping and Kim Jong-un). Luckily, Li Zhanshu, a Standing Committee member, visited to commemorate the founding of North Korea.
  • I'm a bit unsure how I should treat Europe #10; the UK negotiated a Brexit deal, but no one actually believes in can pass parliament. I'm going to tentatively mark this as correct, but will revise if parliament passes this deal unaltered.
  • EDITED TO ADD: I originally didn't mark America #6 as failed because I totally forgot about the one day government shutdown in January. I've since updated everything to add this in.


The whole point of having predictions with binned probabilities (here, 51%, 60%, 70%, 80%, 90%) is that you can then check your accuracy; an individual prediction can be right or wrong, but when you pool several predictions you were equally sure of, you can tell if you were right or wrong to be that sure. Here’s how I did:

  • Of my predictions at a 51%confidence level, I got right and 6 wrong (50%).
  • Of my predictions at a 60%confidence level, I got 4 right and 7 wrong (36%).
  • Of my predictions at a 70%confidence level, I got 13 right and 5 wrong (72%).
  • Of my predictions at an 80%confidence level, I got 19 right and 5 wrong (79%).
  • Of my predictions at a 90%confidence level, I got 23 right and 2 wrong (92%).
  • Two predictions were invalid, because they were predicated on a North Korean missile test that never occurred.

Some people prefer to look at this graphically. Here I’ve plotted my answers versus what you’d expect if I was perfectly calibrated (something I say is 70% likely to happen happens 70% of the time). Whenever the blue line labelled “Mine” is below the red line labelled “Perfect”, I was overconfident. Whenever it’s above the red line, I was under-confident.

Like last year, I’m pleased with these results. Most of my probability bins were very well calibrated, although I seem to have become rather overconfident at the 60% level. I was under-confident at that level last year by a significant amount, so this might just be normal variance. I’ll keep an eye on this for next year.

In terms of specific classes of failed predictions, I can identify two.

First, I tend to overestimate how quickly bureaucratic processes will finish (see: 3, 5, 23). Second, I tend to assume the future will look much like the present (see: 11, 29, 72, 74-79). This might be because of the availability heuristic; the present is more available than alternative futures, so I might be viewing it as more likely than I should.

I hope to account for these biases in my soon to be released predictions for 2019. Stay tuned!

Tags: predictions