Socratic Form Microscopy

Grading my 2017 Predictions

by Zach Jacobi in Falsifiable

Now is the big reveal. Just how did I do in 2017?

Canada

  1. Trudeau ends the year with a lower approval rating than he started – 60%
  2. No bill introduced that changes the electoral system away from first past the post in 2019 – 50%
  3. No referendum scheduled on changing the electoral system away from first past the post before 2019 – 70%
  4. A bill legalizing marijuana is passed by the House of Commons – 90%
  5. The senate doesn’t block attempts to legalize marijuana – 80%
  6. At least one court finds the assisted dying bill isn’t in line with Carter v Canada – 60%
  7. Ontario Liberal Approval rating remains below 30% – 80%
  8. Patrick Brown “unsure” rating remains above 40% – 70%
  9. Kellie Leitch is not the next CPC leader – 80%
  10. Michael Chong is not the next CPC leader – 70%
  11. Maxine Bernier is not the next CPC leader – 90%
  12. No terrorist attack that kills >10 Canadians – 70%
  13. No terrorist attack that kills >100 Canadians – 90%
  14. At least one large technology company (valuation >$10 billion and >1,000 employees) will open a Waterloo office in 2017 – 80%

 

America

  1. Trump will veto at least 1 bill passed by the House and Senate – 70%
  2. Changes to NAFTA will not significantly affect Canada (e.g. introduce tariffs, eliminate visas, etc) – 80%
  3. Less than 100km of concrete wall on the border with Mexico will be constructed – 80%
  4. Unemployment rate changes by less than 0.5% in 2017 – 90%
  5. Bay Area housing prices increase in 2017 – 90%
  6. Protests (in America) on Trump’s inauguration day draw at least 1 million people – 80%
  7. Protests (in America) on Trump’s inauguration day draw at least 5 million people – 50%
  8. Protests (in America) on Trump’s inauguration day draw less than 10 million people – 70%
  9. Protests outside of America on Trump’s inauguration day draw at least 1 million people – 60%
  10. Terrorist attack in America that kills at least 10 Americans – 70%
  11. No terrorist attack in America that kills at least 100 Americans – 70%
  12. No registry of Muslims created in America – 90%
  13. New Supreme Court Justice is named to the USSC – 90%
  14. No repeal of any of: the individual mandate, the prohibition on denying coverage for pre-existing conditions, children remaining on their parents insurance plans until they are 25 – 70%
  15. gov is taken offline or otherwise rendered inoperative by the new administration – 80%
  16. No Federal Department is eliminated – 80%

South America

  1. No setback to the FARC peace deal significant enough to cause >1000 rebels to rearm – 70%
  2. On the black market, the exchange rate for Venezuelan Bolivars to US Dollars remains above 3000 bolivars per dollar. (As measured by DolarToday) – 80%
  3. Inflation in Venezuela for 2017 is higher than 100% (As measured by DolarToday) – 90%
  4. United Socialist party retains control of the Venezuelan presidency – 70%
  5. No uprising in Venezuela leading to >1000 combined civilian and soldier deaths – 70%

Middle East

  1. The “Regulation” Bill, legalizing many illegal settlements, is passed in Israel – 60%
  2. No Israeli politician is indicted by the ICC over settlement activity in 2017 – 80%
  3. The US moves its embassy to Jerusalem – 50%
  4. OPEC agreement fails (as evidenced by Saudi Arabia increasing oil production to >10.058 million BPD) – 50%
  5. Iraq takes back Mosul – 90%
  6. Mosul Dam does not fail – 70%
  7. Fewer casualties in Syrian Civil War in 2017 than in 2016 – 60%
  8. No new international sanctions against Iran – 80%
  9. No new US sanctions against Iran – 50%
  10. No attack on the Iranian nuclear program by Israel – 80%
  11. Iran does not withdraw from the deal limiting its nuclear program – 80%
  12. Conditional on Iran remaining in the nuclear deal, inspectors find no evidence of violations after the deal began – 90%
  13. Yemen Civil War continues – 60%

Africa

  1. Power transition in The Gambia requires ECOWAS troops – 50%
  2. Power transition occurs in The Gambia – 70%
  3. No peace deal ends South Sudan fighting – 50%
  4. IS or affiliated groups do not hold more territory in Africa at the end of 2017 than at the beginning – 90%
  5. Libya has a single government by the end of 2017 – 50%
  6. No protests, riots, or rebellion in Egypt that kills >100 people in a one week period – 80%
  7. No protests, riots, or rebellion in Tunisian kills >50 people in a one week period – 90%
  8. At least one terrorist attack kills >50 people in Tunisian – 50%

Asia

  1. Inflation rate in Japan remains below 1% in 2017 – 70%
  2. No Japanese snap election in 2017 – 90%
  3. Scandal involving Thailand’s new king makes its way to a major Western Newspaper – 50%
  4. Saenuri Party loses in the 2017 South Korean election – 80%
  5. China will send at least one diplomatic “insult” to the US (e.g. expelling an ambassador or consul or closing on of its embassies or consulates) – 60%
  6. By the end of 2017, none of the young lawmakers associated with the Umbrella Revolution will be in the Hong Kong parliament – 60%
  7. The Hong Kong lawmakers who are appealing their ban from parliament will have their final appeals denied – 80%
  8. China will not deploy its military against either Hong Kong or Taiwan in 2017 – 90%
  9. North Korea detonates a nuclear weapon – 70%
  10. North Korea does not demonstrate a completed weapon system (e.g. miniaturized bomb and ICBM capable of threatening the continental United States) – 90%

Europe

  1. No resolution to the crisis in Ukraine – 70%
  2. Crimea remains part of Russia – 90%
  3. Russian GDP growth is less than 2% – 80%
  4. No gain of greater than 15% in the value of the ruble vs the dollar – 60%
  5. Angela Merkel remains Chancellor of Germany – 60%
  6. Marie Le Pen does not become President of France – 70%
  7. Geert Wilders does not become Prime Minister of the Netherlands – 70%
  8. UK invokes Article 50 – 60%
  9. Conditional on the UK invoking article 50, this occurs behind schedule – 70%
  10. Conditional on the UK leaving the EU, Scotland prepares for another referendum – 80%
  11. No snap election called in the UK – 80%
  12. No regional independence movement (e.g. Scotland, Catalan) achieves success in Europe in 2017 – 90%
  13. Sanctions against Russia are not significantly rolled back (e.g. sanctions remain in place against Rosneft, Novate, Gazprombank and Vnesheconombank by all members of the G7 remain in place at the end of 2017) – 60%

Personal

  1. I will not break up with anyone I am currently dating – 90%
  2. I will buy a car – 50%
  3. I will still be working at my current job at the end of 2017 – 80%
  4. I will not move to another city in 2017 – 90%
  5. Conditional on remaining in my current city, I will not move to a different apartment in 2017 – 80%
  6. I will read at least 40 books this year – 80%
  7. I will read at least 10 non-fiction books this year – 50%
  8. I will start reading (and read at least 50 pages) of at least 10 books people recommended to me this year – 60%
  9. I will write at least 200,000 words this year – 80%
  10. I will post at least 15 blog posts or short stories – 80%
  11. I will post at least 25 blog posts or short stories – 50%
  12. I will be >15% over or under-confident for at least 2 confidence levels in these predictions (before taking into account this prediction) – 80%

Notes

I thought that making my predictions mostly numerical would make them easy to grade. This mostly worked, but there were a few edge cases, judgement calls, and other amusing things that I want to explicitly mention:

  • Throughout my predictions I used the word "remains". I regret this, because it is ambiguous. I think I intended it to mean "on January 1st, 2018, X remains true", but there's an alternative parsing that is "during all of 2017, X will be true". I feel it's most accurate to grade these according to my intent. For my 2018 predictions, I will use clearly language.
  • For 8, the two most recent polls I could find were both from November. In one, Patrick Brown had a "don't know" rating of 50%. In the other, it was 34%. Polls were found by Googling 'ontario leader popularity' and 'ontario leader popularity politics'; November was the last month in which I could find polls, so I only used November polls. I'm averaging these two and considering the prediction successful. The lack of good aggregation of Ontario political information is part of why I would like to create a website tracking the Ontario election this year.
  • While I was correct as to the who wasn't the next leader of the Conservative party, I definitely got emotionally involved such that I was severely miscalibrated. Bernier came far closer to winning it then Leitch or Chong and both of those two fringe candidates had much lower chances that it felt like they did.
  • WRT 24 and 25, I'm not counting the Las Vegas shooting as a terrorist attack because it lacked a political motive (as far as we currently know). I think I overestimated the risk of terrorist attack because of the availability heuristic (the last two years had seen a higher than normal amount of successful and dangerous terrorist attacks). A proper estimation would have focused more on the base rate.
  • I don't think Trump's cancellation of advertising comes anywhere close to fulfilling 29, so I'm marking it as failed.
  • 31 is borderline, with many former FARC fighters "joining criminal gangs or a dissident FARC movement that has about 1,000 fighters nationwide". Given that this still implies less than 1000 members rearmed to continue the fight as FARC, I think the prediction holds.
  • 38 is also borderline, but ultimately, I think there is a difference between an announcement of intent to move and an actual movement. Since I was going to mark 28 as a success if Trump hadn't signed the tax bill by now, it's only fair that I mark 38 as a failure.
  • I was way under-confident in the stability of the Mosul dam (41). Compare my probability with the chance on the Good Judgement Project and you'll see I really overstated the risk compared to the consensus.
  • WRT to 43 and 44, the US Treasury added new groups to existing designations, but these are neither new sanctions, nor international sanctions
  • For 59, I think this Daily Mail headline counts: "Thailand's colourful new King brought 'his mistress AND his former air stewardess wife' to his father's lavish cremation ceremony with both marching in bearskin hats". I think I want to stay away from prediction "scandals" in the future though, because it's a very fuzzy concept.
  • While North Korea claims to have a complete, miniaturized ICBM, it looks like them actually realizing this with a weapon able to hit the US mainland is about one year away. Therefore 66 is a success.
  • 69 is only provisionally true and needs to be revised when more GDP data is available.
  • I am apparently rubbish at predicting snap elections, given that I got both 77 and 58 wrong, while being highly confident in my wrongness.
  • Out of all of my failed predictions, the one that surprised me the most was the OPEC deal holding. I really thought that it would fall apart.

A complete list of the sources I used when grading all non-personal predictions is available here.

Calibration

The whole point of having predictions with few allowed probabilities (for me it was 50%, 60%, 70%, 80% and 90%) is that you can then check how accurate these were by pooling your answers. Here’s how I did:

Of my predictions at a 50% confidence level, I got 7 right and 6 wrong (54%). Of my predictions at a 60% confidence level, I got 9 right and 4 wrong (69%). Of my predictions at a 70% confidence level, I got 16 right and 4 wrong (80%). Of my predictions at an 80% confidence level, I got 20 right and 6 wrong (77%). Of my predictions at a 90% confidence level, I got 17 right and 2 wrong (89%).

If you prefer graphs, here’s the results on a graph. The red line shows what I would get if I was a perfect judge of probability. The blue line is actual me. Whenever the red line is below my results, I was under-confident. Whenever it’s above them, I was overconfident.

I’m pleased that in general (excepting 70% vs. 80%), things I thought were more likely were in fact more likely. I appear to be fairly under-confident at lower probability levels (50% through 70%), and fairly good at higher confidence levels (80% and 90%), although of course this is just one year and some of this could be due to chance and luck.

My meta-calibration was quite poor. I was never more than 10% off from perfect calibration, despite my worries that I would frequently be up to 15% from it.


Tags: predictions