Most things are wrong (and it doesn’t matter) January 13, 2012 at 5:40 pm
For my sins, perhaps in a past life, I used to manage a model verification group. We looked at derivatives pricing models and checked their accuracy. Many of the ones we looked at were somewhat wrong, and some of these we passed anyway. Why?
- A model is only designed to be used with a domain of applicability. Provided that their are controls in place to make sure it is not used outside that domain, it doesn’t matter that it is wrong there.
- Moreover all models are a simplifications. They will always break if you stress them enough.
- Time to market sometimes beats correctness. Being first, even with a slightly wrong model, is sometimes better than being seventh with a more correct one.
In other words, modelling is like crossing a river on lily pads. It isn’t a question of whether things are secure – you know that they are not – it is a question of having sufficiently good judgement that you avoid taking a bath.
It does not surprise me, then, to learn that many research results may be false. People doing complicated things make mistakes, even without bias. Having open data (so that others can build their own model) and open models (so that they can see where yours breaks) helps, but mistakes are still going to slip through. Science, like finance, isn’t ‘correct'; the best it can aim for is ‘not obviously false’, and it might not hit that bar some of the time.
Indeed, ‘correctness’ is a really unhelpful idea in most modelling. Few models are absolutely correct, and certainly very few interesting ones. ‘Close enough, enough of the time’ is much more apposite, and ‘open enough that you can figure that out’ is a good way of helping to get there.