I recently started reading Prof. J. S. Armstrong’s Principles of Forecasting: A Handbook for Researchers and Practitioners after reading Nate Silver’s glowing recommendation of it in his recent book The Signal and the Noise (Kindle). I’ve only read a little of the beginning, and I’m already hooked. Principles of Forecasting is an incredible resource on the art of prediction and decision making. This is my new desert island book as it’s going to take me years to read, re-read, and apply in entrepreneurship and investing.
The first non-introductory chapter starts off with a bang: cases where running role-played simulations beat expert opinions in predicting conflict outcomes. Fascinating. In the chapter, Armstrong asserts with ample evidence that when two parties are in conflict, specific forms of role-playing are more effective at predicting outcomes than simply asking the experts. He lists five scenarios, a drug company board handling a recall, an appliance manufacturer negotiating sales with a supermarket chain, a nation handling an artist protest, journal editors renegotiating their contract with their publisher, and the NFL negotiating with the players association.
All of these examples were real events, and expert opinion as well as multiple role-playing simulations per scenario were run before the outcomes were determined. Ultimately, role-playing out performed expert opinion by a wide margin. On average, expert opinion was right 16% of the time while role playing was right 56% of the time, and strictly chance was correct 25% of the time, so expert opinion was actually worse than chance on average. Makes you rethink listening to political or financial news.
Armstrong’s chapter in Principles of Forecasting goes into much more detail than I do here and is well worth reading. It’s apparent that even role-playing is by no means an accurate predictor even in the best of scenarios, but it clearly is better than expert opinion in predicting how parties resolve a conflict. I can’t help but hypothesize that a cognitive bias is at play here, the empathy gap bias. Most decisions are ultimately made based on emotions, and unless you attempt to simulate the emotional state of the decision makers, you will be missing key information in making a prediction. The best way to empathize with someone is to put yourself in their shoes, and Armstrong notes the importance of realistically simulating the decision environment with all the key decision makers as much as possible.
Where does this seems directly applicable? Merger arbitrage was the first that came to mind for me. But I’m sure anyone playing with prediction markets could have some fun as well. I suspect that markets often react more to expert opinion than well run simulations, so there should be cases where markets will get it wrong, but your simulations may just get it right a little more often. The next time you want to predict if two companies will come together, and there’s substantial money on the line, think about hiring some actors and running a few simulations as opposed to listening to that well paid consultant who knows the industry “inside and out.”
Photo Credit: tinmachine / photocase.com