All Large Calculations Are Wrong

One of the surprises about working in a large investment bank has been that most of the interesting, the truly interesting lessons I have learned there, have been sociological rather than mathematical or financial. Political rather than quantitative. Finance is business, distilled to its pure essence, in concentrated form. And business (in its better and worse moments) is about people, and agendas, and glad-handing, and personality and charisma, and luck, and salesmanship, and deal-making, and emotional intelligence, and listening, and talking. Very few of those things are “quantitative” in any real sense. When they are, the “quantitative” is often window dressing, frills, tricks used to make the pitch-book more appealing and authoritative-looking. There’s no mystery about the fact that the real quantity that matters is the profit taken from the table. And profit trumps your equations.

Many mathy finance types get out of school having this naive image of “quantitative finance” that it is this sort of post-graduate mathematical utopia where all the best and brightest mathematical geniuses get together every day and engage in neverending stochastic-calculus-circle-jerks to come up with great trade ideas with their genius and so forth, and everyone profits and everyone lives happily ever after. Most of these people (I was no exception) will gladly flock to whatever “finance” job they get offered, not quite knowing what they are getting into, not even knowing what the different roles entail. This is presumably how the back office risk and IT support groups end up getting seemingly overstuffed with the overeducated – particle physics PhDs making daily PDF risk reports absolutely no one reads, or debugging VBA code so a 24-year-old junior trader with a bachelor’s and a dad with connections won’t yell at them for making them late for their flight to their Florida summer house. This is what many, many quantitative types in finance get stuck doing. (And believe me: there but for the grace of God go I.)

But because of this, because “finance” has attracted such large numbers of these people even for the most menial and thankless roles, it is a decent real-world experiment for the potential that can be unleashed when you get scores of highly, highly educated and mathematically-inclined people together and charge them, in a relatively well-compensated setting, with collaborating on doing large calculations of actual real-world import. And I’m here to report that the result is: garbage. Not total garbage necessarily, but pretty smelly.

If you read Taleb’s book on the Black Swans or whatever it was called, or if (like me) you refused to read it but are pretty sure you know what its basic thesis was, then you know that banks all have to compute something called VaR (Value at Risk). This is a number that supposed to represent something like: “you’re 99% sure this is the most you could lose in the next however-many days”. Now Taleb’s claim (I gather) was that banks all manage their risk by looking at VaR, and since VaR is flawed (can’t capture “black swans”), that’s what helped cause the crisis. I call bullshit. Nobody manages to VaR, not really. Nevertheless, regulators do force all banks to compute this bogus number, and various other related bogus numbers that depend on it, so everyone can CYA and tell themselves they’re doing a good job regulating and deserve their salaries and everything is ok. So banks, as a result, typically have a group of people whose job – whose only job, mind you – is to generate this number – this single number (e.g. “VaR = $150 million”) on a regular basis.

And the one thing I can say with confidence about this number is that it is wrong.

Always wrong, every day. The one constant about this calculation – remember, a calculation that a global team of six-figure quants is charged with producing – is that if you dig into the calculations behind the number, truly dug deeply enough, you can always unfailingly find something wrong with it. It may not always be hugely wrong or materially wrong, or there will be various wrongs that kinda cancel each other out, but there will be something undeniably wrong. Whether it’s the inputs or the assumptions or the historical time-series or the way things are simulated or the way asset types are bucketed. Even if you correct some issue that you find today, another one will creep into the calculation tomorrow. And you won’t find all issues.

The point of this preface is that large calculations are hard and in particular large calculations are wrong.

And I want to stress here that this is not because the people doing the calculations are dumb. The people who compute VaR are for the most part (although often front-office types would disagree, a bit unfairly) highly intelligent bright people with their math-finance and financial-math and physics and computer science degrees up the wazoo. It’s because the calculations themselves are intrinsically hard to coordinate in such a way as to get right, no matter how “smart” the people doing it. Keep in mind that by large calculation I don’t mean something like a single number-crunching program to calculate the zillionth digit of pi or to visually rotate a detailed scene from a Toy Story movie – sure those are large calculations but at root they pure mathematics with definite answers. I’m talking about a problem rooted in the real world, that relies on a large amount of real-world inputs, inputs that can have errors or be fuzzy or have timing or data or interpretation issues, inputs that have to be assembled from different sources in a collaboration, then assimilated together in a large calculation, a calculation based on a model that makes assumptions and approximations and simplifications, assumptions that may/may not actually hold true, or may not still hold true.

It’s large calculations like that which are wrong. For the record, and in case it wasn’t clear where I was heading with all this, here are some calculations that fit the above description:

• what will the future climate of the earth be like
• how many jobs will exist in the economy if you do this or that
• what’s the probability that a drilling site will spring an oil leak under the ocean
• what will such and such policy mean for whether Social Security is solvent 30 years from now
• how many troops should you send somewhere to win the war, or will you win the war with the current number
• how should health-care be apportioned: who should get what procedures, and how much to charge for them

Those are all large calculations (or rely on them). And anyone who claims they have The Answer to any of them is lying. Because any a priori calculated answer to any of those questions is wrong.

It won’t have escaped your notice that these and more large calculations are all related to political questions to which the political left regularly claims to know The Answers. This is because the left is in thrall to the cult of Smart People. That is, the left is constitutionally unable to imagine a question which cannot be answered by getting together a sufficient number of Smart People and having them work on it. Just have Smart People do the calculation, tell us their answer, and that’s what we should do. And Smart People, at least some of them, do claim to know the answers to the above: the earth will definitely get warmer; we can’t win a war in Iraq or Afghanistan; Social Security is fine; we can lower health care costs using smart medicine; stimulus will increase the number of jobs. And when things go wrong or answers seem to fail – as in the financial crisis, or the oil leak – the left’s answer is always and inevitably: We needed more regulation and intervention. Not enough Smart People were in charge, not enough Smart ideas acted upon. This is why, for example, the fact that the “stimulus” didn’t create jobs will not ever cause them to disbelieve in “stimulus”. It will only strengthen their conviction that more “stimulus” was needed.

As a corollary, the left operates from an a priori assumption that whatever the Smart People seem to be saying, is automatically correct and should be acted upon. Well let me revise that slightly. In sober moments most of the left would probably admit they don’t necessarily think that the Smart Peoples’ answer to everything is actually correct. Their claim would more likely be: it may not be perfect but it’s the best we have, so we should act upon it. This may sound reasonable enough, but when one is talking about a large calculation – global warming, say – it takes on a rather different meaning. Because if large calculations are wrong (and I think that they are), that we should ignore their flaws and must act on them anyway, because the people who did them are Smart and we have no other or better choice, has the characteristics of a religious statement. It is a call for a scientific theocracy.

It may be answered: how do I know the climate calculations are wrong? Now I can say because probably unlike you, I’ve actually worked on climate models and know what they are and what goes into them. Or I can do as I’m doing here and point to the fact that giant teams of highly-paid quants can’t even get a single-number calculation – VaR – correct. One single number! So why would we expect, like, some grad student (‘overseen’, i.e. pointed towards some literature and a data set, by his professor) working in virtual ivory-tower isolation on sloppy C code for a summer to be able to generate any sort of accurate model of the fricking entire future climate of the entire fricking earth? Why would we look at the latter’s paper and go Oooh and Aaah and declare we should base policy on it? (And if you don’t think that’s what climate modeling is, then what exactly do you think it is?)

The point is just that some humility is in order when interpreting and using all these models and calculations. That goes for climate science, for Keynesian ‘stimulus’ claims, for claims of centralized medicine as a cure-all: humility, please. Large calculations are wrong.

This is hard for Smart People (and their cheerleaders) to admit, for obvious reasons. There has been built up quite a racket, a parasitic meritocracy, of Smart People all involved in politically agitating for the empowerment and betterment and enrichment of Smart People – i.e., of each other. Give him a job and he’ll give me a job. Just ask them: Smart People should be put in charge of the climate. They should be put in charge of medicine. They should be put in charge of the economy. All the Smartest People say so!

With salaries and lifestyles to match that responsibility, obviously. Or if there is no such responsibility, create one. Create a ‘carbon trading’ market: voila! More jobs for traders! Create a ‘health care exchange’. Voila! More jobs for actuaries! There’s no real end to how many phony made-up responsibilities Smart People can invent for each other.

So it’s got to be hard for someone invested in such a stance to admit let alone confront the fact that there are some problems that are inherently thorny, calculations that simply can’t be done correctly, no matter how ‘smart’ you are. But if we don’t, we’ll never get out of thrall to the Smart People cult.

One antidote might be: the private industry. Contact with private industry, with business, is quite a good remedy for an attitude that Smart People know all the answers. Indeed, the first people in an investment bank who’ll tell you that VaR is wrong are the people who calculate VaR. Again many of these people will have come from an academic background, but now they are slaving away to calculate a number they know is wrong, and receive daily firsthand evidence that nobody trusts it, and this is a highly effective humility-builder. Take a vaunted academic climate modeler, put them in a risk group and tell them to use their skills to calculate VaR, and I guarantee you’ll see some instant, quick (and needed) ego puncturing.

Smart People in academics and government, unfortunately, have no such corrective because they have no real contact with business or the private industry or the real world. So they never have to confront the fact that their assumptions aren’t necessarily accurate, their models not complete, their data not error-free. Their calculations are wrong. But who will tell them this? They clearly don’t want to hear it. Indeed, most of them seem to have an ideology pointed towards abolishing the real world: making the private industry as small as possible, and subsuming it entirely under the government. If large calculations can’t be right, let’s at least abolish all possible sources of evidence that they are wrong.