The main reason why almost all econometric models are wrong
Download the WEA commentaries issue ›
By Lars Syll
Since econometrics does not content itself with only making optimal predictions, but also aspires to explain things in terms of causes and effects, econometricians need loads of assumptions — most important of these are additivity and linearity. Important, simply because if they are not true, your model is invalid and descriptively incorrect. And when the model is wrong — well, then it is wrong.
Limiting model assumptions in economic science always have to be closely examined since if we are going to be able to show that the mechanisms or causes that we isolate and handle in our models are stable in the sense that they do not change when we ‘export’ them to our ‘target systems,’ we have to be able to show that they do not only hold under ceteris paribus conditions and a fortiori only are of limited value to our understanding, explanations or predictions of real economic systems.
Our admiration for technical virtuosity should not blind us to the fact that we have to have a cautious attitude towards probabilistic inferences in economic contexts. We should look out for causal relations, but econometrics can never be more than a starting point in that endeavour since econometric (statistical) explanations are not explanations in terms of mechanisms, powers, capacities or causes. Firmly stuck in an empiricist tradition, econometrics is only concerned with the measurable aspects of reality. But there is always the possibility that there are other variables – of vital importance and although perhaps unobservable and non-additive, not necessarily epistemologically inaccessible – that were not considered for the model. Those which were can hence never be guaranteed to be more than potential causes, and not real causes. A rigorous application of econometric methods in economics really presupposes that the phenomena of our real world economies are ruled by stable causal relations between variables. A perusal of the leading econom(etr)ic journals shows that most econometricians still concentrate on fixed parameter models and that parameter-values estimated in specific spatio-temporal contexts are presupposed to be exportable to totally different contexts. To warrant this assumption one, however, has to convincingly establish that the targeted acting causes are stable and invariant so that they maintain their parametric status after the bridging. The endemic lack of predictive success of the econometric project indicates that this hope of finding fixed parameters is a hope for which there really is no other ground than hope itself.
Real-world social systems are not governed by stable causal mechanisms or capacities. The kinds of ‘laws’ and relations that econometrics has established, are laws and relations about entities in models that presuppose causal mechanisms being atomistic and additive. When causal mechanisms operate in real-world systems they only do it in ever-changing and unstable combinations where the whole is more than a mechanical sum of parts. If economic regularities obtain they do it (as a rule) only because we engineered them for that purpose. Outside man-made ‘nomological machines’ they are rare, or even non-existent. Unfortunately, that also makes most of the achievements of econometrics – as most of the contemporary endeavours of mainstream economic theoretical modelling — rather useless.
Even in statistics, the researcher has many degrees of freedom. In statistics — as in economics and econometrics — the results we get depend on the assumptions we make in our models. Changing those assumptions — playing a more important role than the data we feed into our models — leads to far-reaching changes in our conclusions. Using statistics is no guarantee we get at any ‘objective truth.’
On the limits of ‘statistical causality’
Causality in social sciences — and economics — can never solely be a question of statistical inference. Causality entails more than predictability, and to really in depth explain social phenomena requires theory. Analysis of variation — the foundation of all econometrics — can never in itself reveal how these variations are brought about. First, when we are able to tie actions, processes or structures to the statistical relations detected, can we say that we are getting at relevant explanations of causation?
Most facts have many different, possible, alternative explanations, but we want to find the best of all contrastive (since all real explanation takes place relative to a set of alternatives) explanations. So which is the best explanation? Many scientists, influenced by statistical reasoning, think that the likeliest explanation is the best explanation. But the likelihood of x is not in itself a strong argument for thinking it explains y. I would rather argue that what makes one explanation better than another are things like aiming for and finding powerful, deep, causal, features and mechanisms that we have warranted and justified reasons to believe in. Statistical — especially the variety based on a Bayesian epistemology — reasoning generally has no room for these kinds of explanatory considerations. The only thing that matters is the probabilistic relation between evidence and hypothesis. That is also one of the main reasons I find abduction — inference to the best explanation — a better description and account of what constitute actual scientific reasoning and inferences.
In the social sciences … regression is used to discover relationships or to disentangle cause and effect. However, investigators have only vague ideas as to the relevant variables and their causal order … I see no cases in which regression equations, let alone the more complex methods, have succeeded as engines for discovering causal relationships.
David Freedman (1997:60)
Since statisticians and econometricians have not been able to convincingly warrant their assumptions of homogeneity, stability, invariance, independence, additivity as being ontologically isomorphic to real-world economic systems, there are still strong reasons to be critical of the econometric project. There are deep epistemological and ontological problems of applying statistical methods to a basically unpredictable, uncertain, complex, unstable, interdependent, and ever-changing social reality. Methods designed to analyze repeated sampling in controlled experiments under fixed conditions are not easily extended to an organic and non-atomistic world where time and history play decisive roles.
If contributions made by statisticians to the understanding of causation are to be taken over with advantage in any specific field of inquiry, then what is crucial is that the right relationship should exist between statistical and subject-matter concerns … The idea of causation as consequential manipulation is apt to research that can be undertaken primarily through experimental methods … However, the extension of the manipulative approach into sociology would not appear promising, other than in rather special circumstances … The more fundamental difficulty is that under the — highly anthropocentric — principle of ‘no causation without manipulation,’ the recognition that can be given to the action of individuals as having causal force is in fact peculiarly limited.
John H Goldthorpe (2000:159)
Why statistics and econometrics are not very helpful for understanding economies
As social researchers, we should never equate science with mathematics and statistical calculation. All science entail human judgement, and using mathematical and statistical models do not relieve us of that necessity. They are no substitutes for thinking and doing real science.
Most work in econometrics is made on the assumption that the researcher has a theoretical model that is ‘true.’ But — to think that we are being able to construct a model where all relevant variables are included and correctly specify the functional relationships that exist between them, is not only a belief without support, it is a belief impossible to support.
The theories we work with when building our econometric regression models are insufficient. No matter what we study, there are always some variables missing, and we do not know the correct way to functionally specify the relationships between the variables.
Every econometric model constructed is miss-specified. There is always an endless list of possible variables to include, and endless possible ways to specify the relationships between them. So every applied econometrician comes up with his own specification and ‘parameter’ estimates. The econometric Holy Grail of consistent and stable parameter-values is nothing but a dream.
The theoretical conditions that have to be fulfilled for econometrics to really work are nowhere even closely met in reality. Making outlandish statistical assumptions do not provide a solid ground for doing relevant social science and economics. Although econometrics has become the most used quantitative method in economics today, it is still a fact that the inferences made are as a rule invalid.
Econometrics is basically a deductive method. Given the assumptions, it delivers deductive inferences. The problem, of course, is that we will never completely know when the assumptions are right. Conclusions can only be as certain as their premises — and that also applies to econometrics.
On randomness and probability
Modern mainstream economics relies to a large degree on the notion of probability. To at all be amenable to applied economic analysis, economic observations have to be conceived as random events that are analyzable within a probabilistic framework. But is it really necessary to model the economic system as a system where randomness can only be analyzed and understood when based on an a priori notion of probability?
When attempting to convince us of the necessity of founding empirical economic analysis on probability models, neoclassical economics actually forces us to (implicitly) interpret events as random variables generated by an underlying probability density function.
This is at odds with reality. Randomness obviously is a fact of the real world. Probability, on the other hand, attaches (if at all) to the world via intellectually constructed models, and a fortiori is only a fact of a probability generating (nomological) machine or a well-constructed experimental arrangement or ‘chance set-up.’
Just as there is no such thing as a ‘free lunch,’ there is no such thing as a ‘free probability.’
To be able at all to talk about probabilities, you have to specify a model. If there is no chance set-up or model that generates the probabilistic outcomes or events – in statistics one refers to any process where you observe or measure as an experiment (rolling a die) and the results obtained as the outcomes or events (number of points rolled with the die, being e. g. 3 or 5) of the experiment – there strictly seen is no event at all.
Probability is a relational element. It always must come with a specification of the model from which it is calculated. And then to be of any empirical scientific value it has to be shown to coincide with (or at least converge to) real data generating processes or structures – something seldom or never done.
And this is the basic problem with economic data. If you have a fair roulette-wheel, you can arguably specify probabilities and probability density distributions. But how do you conceive of the analogous nomological machines for prices, gross domestic product, income distribution etc? Only by a leap of faith. And that does not suffice. You have to come up with some really good arguments if you want to persuade people into believing in the existence of socio-economic structures that generate data with characteristics conceivable as stochastic events portrayed by probabilistic density distributions.
We simply have to admit that the socio-economic states of nature that we talk of in most social sciences – and certainly in economics – are not amenable to analyze as probabilities, simply because in the real world open systems there are no probabilities to be had!
The processes that generate socio-economic data in the real world cannot just be assumed to always be adequately captured by a probability measure. And, so, it cannot be maintained that it even should be mandatory to treat observations and data – whether cross-section, time series or panel data – as events generated by some probability model. The important activities of most economic agents do not usually include throwing dice or spinning roulette-wheels. Data generating processes – at least outside of nomological machines like dice and roulette-wheels – are not self-evidently best modelled with probability measures.
When economists and econometricians – often uncritically and without arguments — simply assume that one can apply probability distributions from statistical theory on their own area of research, they are really skating on thin ice. If you cannot show that data satisfies all the conditions of the probabilistic nomological machine, then the statistical inferences made in mainstream economics lack sound foundations.
Statistical — and econometric — patterns should never be seen as anything other than possible clues to follow. Behind observable data, there are real structures and mechanisms operating, things that are — if we really want to understand, explain and (possibly) predict things in the real world — more important to get hold of than to simply correlate and regress observable variables.
Statistics cannot establish the truth value of a fact. Never has. Never will.
Sometimes we do not know because we cannot know
To understand real world ‘non-routine’ decisions and unforeseeable changes in behaviour, ergodic probability distributions are of no avail. In a world full of genuine uncertainty – where real historical time rules the roost – the probabilities that ruled the past are not those that will rule the future.
Time is what prevents everything from happening at once. To simply assume that economic processes are ergodic and concentrate on ensemble averages – and a fortiori in any relevant sense timeless – is not a sensible way for dealing with the kind of genuine uncertainty that permeates open systems such as economies.
When you assume the economic processes to be ergodic, ensemble and time averages are identical. Let me give an example: Assume we have a market with an asset priced at 100 €. Then imagine the price first goes up by 50% and then later falls by 50%. The ensemble average for this asset would be 100 € — because we here envision two parallel universes (markets) where the asset-price falls in one universe (market) with 50% to 50 €, and in another universe (market) it goes up with 50% to 150 €, giving an average of 100 € ((150+50)/2). The time average for this asset would be 75 € – because we here envision one universe (market) where the asset-price first rises by 50% to 150 €, and then falls by 50% to 75 € (0.5*150).
From the ensemble perspective nothing really, on average, happens. From the time perspective lots of things really, on average, happen.
Assuming ergodicity there would have been no difference at all. What is important with the fact that real social and economic processes are nonergodic is the fact that uncertainty – not risk – rules the roost. That was something both Keynes and Knight basically said in their 1921 books. Thinking about uncertainty in terms of ‘rational expectations’ and ‘ensemble averages’ has had seriously bad repercussions on the financial system.
Knight’s uncertainty concept has an epistemological founding and Keynes’ definitely an ontological founding. Of course, this also has repercussions on the issue of ergodicity in a strict methodological and mathematical-statistical sense.
The most interesting and far-reaching difference between the epistemological and the ontological view is that if one subscribes to the former — Knightian – view, you open up for the mistaken belief that with better information and greater computer-power we somehow should always be able to calculate probabilities and describe the world as an ergodic universe. As Keynes convincingly argued, that is ontologically just not possible.
To Keynes, the source of uncertainty was in the nature of the real — nonergodic — world. It had to do, not only — or primarily — with the epistemological fact of us not knowing the things that today are unknown, but rather with the much deeper and far-reaching ontological fact that there often is no firm basis on which we can form quantifiable probabilities and expectations at all.
Sometimes we do not know because we cannot know.
Keynes’ critique of econometrics — still valid after all these years
To apply statistical and mathematical methods to the real-world economy, the econometrician, as we have seen, has to make some quite strong assumptions. In a review of Tinbergen’s econometric work — published in The Economic Journal in 1939 — John Maynard Keynes gave a comprehensive critique of Tinbergen’s work, focusing on the limiting and unreal character of the assumptions that econometric analyzes build on:
(1) Completeness: Where Tinbergen attempts to specify and quantify which different factors influence the business cycle, Keynes maintains there has to be a complete list of all the relevant factors to avoid misspecification and spurious causal claims. Usually, this problem is ‘solved’ by econometricians assuming that they somehow have a ‘correct’ model specification. Keynes (1940:155) is, to put it mildly, unconvinced:
It will be remembered that the seventy translators of the Septuagint were shut up in seventy separate rooms with the Hebrew text and brought out with them, when they emerged, seventy identical translations. Would the same miracle be vouchsafed if seventy multiple correlators were shut up with the same statistical material? And anyhow, I suppose, if each had a different economist perched on his a priori, that would make a difference to the outcome.
(2) Homogeneity: To make inductive inferences possible — and being able to apply econometrics — the system we try to analyze has to have a large degree of ‘homogeneity.’ According to Keynes most social and economic systems — especially from the perspective of real historical time — lack that ‘homogeneity.’ It is not always possible to take repeated samples from a fixed population when we were analyzing real-world economies. In many cases, there simply are no reasons at all to assume the samples to be homogenous.
(3) Stability: Tinbergen assumes there is a stable spatio-temporal relationship between the variables his econometric models analyze. But Keynes argued that it was not really possible to make inductive generalisations based on correlations in one sample. As later studies of ‘regime shifts’ and ‘structural breaks’ have shown us, it is exceedingly difficult to find and establish the existence of stable econometric parameters for anything but rather short time series.
(4) Measurability: Tinbergen’s model assumes that all relevant factors are measurable. Keynes questions if it is possible to adequately quantify and measure things like expectations and political and psychological factors. And more than anything, he questioned — both on epistemological and ontological grounds — that it was always and everywhere possible to measure real-world uncertainty with the help of probabilistic risk measures. Thinking otherwise can, as Keynes wrote, ‘only lead to error and delusion.’
(5) Independence: Tinbergen assumes that the variables he treats are independent (still a standard assumption in econometrics). Keynes argues that in such a complex, organic and evolutionary system as an economy, independence is a deeply unrealistic assumption to make. Building econometric models from that kind of simplistic and unrealistic assumptions risk producing nothing but spurious correlations and causalities. Real-world economies are organic systems for which the statistical methods used in econometrics are ill-suited, or even, strictly seen, inapplicable. Mechanical probabilistic models have little leverage when applied to non-atomic evolving organic systems — such as economies.
Building econometric models can’t be a goal in itself. Good econometric models are means that make it possible for us to infer things about the real-world systems they ‘represent.’ If we cannot show that the mechanisms or causes that we isolate and handle in our econometric models are ‘exportable’ to the real-world, they are of limited value to our understanding, explanations or predictions of real-world economic systems.
(6) Linearity: To make his models tractable, Tinbergen assumes the relationships between the variables he study to be linear. This is still standard procedure today, but as Keynes (1939:564) writes:
It is a very drastic and usually improbable postulate to suppose that all economic forces are of this character, producing independent changes in the phenomenon under investigation which are directly proportional to the changes in themselves; indeed, it is ridiculous.
To Keynes, it was a ‘fallacy of reification’ to assume that all quantities are additive (an assumption closely linked to independence and linearity).
Econometric modelling should never be a substitute for thinking. From that perspective, it is really depressing to see how much of Keynes’ critique of the pioneering econometrics is still relevant today.
The limits of probabilistic reasoning
Probabilistic reasoning in science — especially Bayesianism — reduces questions of rationality to questions of internal consistency (coherence) of beliefs, but, even granted this questionable reductionism, it is not self-evident that rational agents really have to be probabilistically consistent. There is no strong warrant for believing so. Rather, there is strong evidence for us encountering huge problems if we let probabilistic reasoning become the dominant method for doing research in social sciences on problems that involve risk and uncertainty.
In many of the situations that are relevant to economics, one could argue that there is simply not enough of adequate and relevant information to ground beliefs of a probabilistic kind and that in those situations it is not possible, in any relevant way, to represent an individual’s beliefs in a single probability measure.
Say you have come to learn (based on own experience and tons of data) that the probability of you becoming unemployed in Sweden is 10%. Having moved to another country (where you have no own experience and no data) you have no information on unemployment and a fortiori nothing to help you construct any probability estimate on. A Bayesian would, however, argue that you would have to assign probabilities to the mutually exclusive alternative outcomes and that these have to add up to 1 if you are rational. That is, in this case – and based on symmetry – a rational individual would have to assign probability 10% to become unemployed and 90% to become employed.
That feels intuitively wrong though, and I guess most people would agree. Bayesianism cannot distinguish between symmetry-based probabilities from information and symmetry-based probabilities from an absence of information. In these kinds of situations, most of us would rather say that it is simply irrational to be a Bayesian and better instead to admit that we ‘simply do not know’ or that we feel ambiguous and undecided. Arbitrary and ungrounded probability claims are more irrational than being undecided in the face of genuine uncertainty, so if there is not sufficient information to ground a probability distribution it is better to acknowledge that simpliciter, rather than pretending to possess a certitude that we simply do not possess.
We live in a world permeated by unmeasurable uncertainty – not quantifiable stochastic risk – which often forces us to make decisions based on anything but rational expectations. Sometimes we ‘simply do not know.’ According to Bayesian economists, expectations tend to be distributed as predicted by theory.’ I rather think, as did Keynes, that we base our expectations on the confidence or ‘weight’ we put on different events and alternatives. Expectations are a question of weighing probabilities by ‘degrees of belief,’ beliefs that have preciously little to do with the kind of stochastic probabilistic calculations made by the rational agents modelled by probabilistically reasoning Bayesian economists.
We always have to remember that economics and statistics are two quite different things, and as long as economists cannot identify their statistical theories with real-world phenomena there is no real warrant for taking their statistical inferences seriously.
If you have a fair roulette-wheel, you can arguably specify probabilities and probability density distributions. But how do you conceive of the analogous ‘nomological machines’ for prices, gross domestic product, income distribution etc? Only by a leap of faith. And that does not suffice in science. You have to come up with some really good arguments if you want to persuade people to believe in the existence of socio-economic structures that generate data with characteristics conceivable as stochastic events portrayed by probabilistic density distributions! Not doing that, you simply conflate statistical and economic inferences.
The present ‘machine learning’ and ‘big data’ hype shows that many social scientists — falsely — think that they can get away with analysing real-world phenomena without any (commitment to) theory. But — data never speaks for itself. Without a prior statistical set-up, there actually are no data at all to process. And — using a machine learning algorithm will only produce what you are looking for. Theory matters.
Some economists using statistical methods think that algorithmic formalisms somehow give them access to causality. That is, however, simply not true. Assuming ‘convenient’ things like ‘faithfulness’ or ‘stability’ is to assume what has to be proven. Deductive-axiomatic methods used in statistics do no produce evidence for causal inferences. The real causality we are searching for is the one existing in the real world around us. If there is no warranted connection between axiomatically derived statistical theorems and the real-world, well, then we have not really obtained the causation we are looking for.
REFERENCES
Freedman, David (1997). From Association to Causation via Regression. Advances in Applied Mathematics.
Goldthorpe, John H (2000). On sociology: numbers, narratives, and the integration of research and theory. Oxford: Oxford University Press
Keynes, J M (1939). Professor Tinbergen’s Method. Economic Journal.
Keynes, J M (1940). Comment. Economic Journal.
From: pp.5-10 of WEA Commentaries 8(3), June 2018
https://www.worldeconomicsassociation.org/files/Issue8-3.pdf
This articles seems to me to have a very uncertain focus. Under the heading “econometrics” what exactly is Lars Syll criticising? Any exercise in econometrics is either testing a particular hypothesis or theory or is retaining such a hypothesis and is estimating parameter values that make it best correspond to data. Any hypothesis is likely to be at best an approximation to reality applying only in particular circumstances. Our hypotheses will often be simply wrong. That is obvious and universally acknowledged. The article wobbles between criticising specific theory (neo-classical approaches, rational expectations), specific simplifying assumptions that are often applied to a range of theories (e.g. linearity), operating without theory at all, i.e. exploring data to see if patterns emerge that suggest causal theories and he seems at times to be questioning Bayes theorem. If Mr Syll knows exactly what the problem is with econometrics he has not made it clear. He prefers a blunderbuss: shoot at everything; some target will get hit. Mr Syll is also very exercised by the realisation that social systems are evolving and unlikely to be stable over long periods – a fact of life of which all sentient economists are aware. No sensible econometrician imagines he or she is discovering immutable truths . They are taking existing theories and asking could the theory, allowing for a penumbra of uncertainty caused by omitted variables, really have generated the data we observe? Is it an adequate model? This question can only be posed and answered probabilistically. If you don’t like that approach, what is the alternative? Introspection, intuition, imagination, blind prejudice can all generate theories about society. Ultimately, if we are to make progress such theories have to confront data. We can never conclusively prove a theory. Despite Popper, we can not conclusively disprove a theory either, though we can dismiss it with a high degree of probability. One important means of dismissal is the intelligent application of econometrics. The fact that the tool can be misused and often has been misused does not change the fact that it is the most important tool we have for advancing knowledge. Econometrics does not have to assume linearity, exogeneity of explanatory variables, normal distribution of errors or any of the other assumptions of convenience that may be adopted. Sometimes those assumptions are pernicious, sometimes harmless in context. More to the point all can be tested in any particular case. It is hard to avoid the suspicion that Mr Syll simply does not like confronting numerical data and would rather live in a world of undisciplined speculation.
What Lars Syll points out, we all know. There is nothing new in it. The real issue is to find out the alternative methodology. Perhaps, the contemporary mathematics and statistics are not rich enough to analyze the real phenomena (with limited data) or if there are any, they are unknown to the economists. We are aware of the problems of covariance among the explanatory variables, but we do not have a dependable method to deal with the problem. We are aware of misspecification problems, but we do not have enough information to specify the model correctly. We acknowledge nonlinearity and perhaps multiple solutions (multiple optima and nonconvexity) but we do not have a foolproof method to handle it. We understand the issues of risk and uncertainty, but we do not have methods to deal with them appropriately. We understand the problem of non-quantifiability, but we do not have appropriate methods to deal with it.
Then, the answer is not to criticise and destroy. The answer is to develop new methods, learn from other disciplines, invent, and do a constructive work. It is much more difficult than simply criticising. I know well what it means, and others too know it.
Those of us in the economics community who are impolite enough to dare to question the preferred methods and models applied in mainstream economics and econometrics are as a rule met with disapproval. But although people seem to get very agitated and upset by the critique, defenders of “received theory” always say that the critique is “nothing new”, that they have always been “well aware” of the problems, and so on, and so on.
So, for the benefit of all mindless practitioners of mainstream econometric modelling — and who don’t want to be disturbed in their doings, eminent mathematical statistician David Freedman put together a very practical list of vacuous responses to criticism that can be freely used to save your peace of mind:
“We know all that. Nothing is perfect … The assumptions are reasonable. The assumptions don’t matter. The assumptions are conservative. You can’t prove the assumptions are wrong. The biases will cancel. We can model the biases. We’re only doing whateverybodyy else does. Now we use more sophisticated techniques. If we don’t do it, someone else will. What would you do? The decision-maker has to be better off with us than without us … The models aren’t totally useless. You have to do the best you can with the data. You have to make assumptions in order to make progress. You have to give the models the benefit of the doubt. Where’s the harm?”
Lars is absolutely right in pointing out that those who dare question the status quo are met with disapproval, even intimidation and contempt. Recently I presented a seminar on my new book “Econometrics as a Con Art”, in which I expressed the opinion that it is tragic to value ARCH models to be as good for humanity as penicillin. Later I was told that a hard core econometrician, who knows nothing about economics, described my presentation as “disastrous”. Well done, Lars–please keep it up.
I loved your comment ! Thanks. It nicely summarizes my assessment of contemporary economics:
‘Alche-nomic wishcraft’. Economists all agree that 2 + 2 = 4 but then they (unknowingly) embarrass themselves (and their discipline) by claiming that (because of increases in productivity, etc.), 2,000,000 + 2,000,000 = 5,000,000 !
It is most unfortunate that most economists seem have to forgotten the basic concept of the ‘circular flow’. It is equally unfortunate that Alfred Marshall used the word ‘scissors’ to describe equilibrium between ‘supply’ and ‘demand’. To this graduate engineer with an avocational addiction to economics, that ubiquitous ‘cross’ seems not unlike a Pareto-Edgeworth ‘contract curve’ (a pretty picture with absolutely no predictive usefulness).
Richard E. Planck, M.S.M.E.
President, The Association for the Advancement of non-Newtonian Economics (AAnNE)
aanne@frontier.com
Firstly, the use of formal and logical modeling of our macroeconomics system has taken us far, unlike the negative claim given above. It has taken us so far that it has strangely managed to show by the use of quite a simple numerical method, that an increment in the taxation of personal incomes has a positive effect on national prosperity, when taken at large, but that when the same sum is collected from the taxation of land values, the benefit is roughly 3 times as big.These facts were first derived in my recent book, which I suggest be taken a bit more seriously. I can send you an e-copy, so you can check the arithmetic chesterdh@hotmail.com,
Secondly any model cannot simply be wrong since it does represent something or some concept that by its nature deals with our subject, and it does have some (limited) resemblance to it. Unless the modeler is completely nuts, he/she surely knows about what is being represented and because we all need to model in our mind’s eye what we are considering, it surely is right for us. Consequently if “all models are wrong”, then the concept that all models are wrong is wrong too! –a zero-zero game.
Thirdly, probability has no place here. We do not model a situation with the belief that it has some probability of being accurate. Its accuracy may be far from the truth but as far as it goes it is 100% sure and it does manage to at least show part of the concept. That is the nature of taking this imaginative methodology.
I support Lars fully. The paper has actually an eye opener in it before the econometricians. Successful economic thoughts have least bothered about the metrics so far. Those who have influenced the transformations in civilizations have seen the past with full devotion, taken insights from there but not got driven by then. Historicity gives an easy pathway towards estimating forward but limits it to the lame contexts of the past and attempts to impose on the future. It’s now time that those who claim themselves to be economists should learn from the impacts of technology on the human progress to understand how far they are disjointed from trim the terrains of the respective plateau of historicity.
Thanks
Professor R P Banerjee
rpbanerjee10@hotmail.com
This article should be read by everyone who does econometrics.
I found Mr. Syll’s little reminder refreshing. It also reminds me of plutonomist’s query, asking what Buddhism has to do with economics. Schumacher replied, “Economics without compassion is like sex without love.”
I think he could have gone much further. Plutonometrics without compassionate wisdom and vast knowledge of the situation is rational justification of rape, pillage, plunder, normative psychopathology, systemic cultural corruption and the anti-ethical Piracy Paradigm that perpetuates it all.
The piece also reminds me of Warren Buffet’s admission: “Yes, it is a class war, and my class is winning.”
Clearly, as Syll and others see and understand, any model that fails to closely approximate the actuality of human culture and its activities, despite being officially accepted as a technically valid and useful tool, serves as diversionary, subversive camouflage, obfuscating the nature and purpose of The Plutonomy Game. If you doubt that, try playing Monopoly (the game), by the Fed & IRS rules, requiring tax payment every 4 turns.
I am confident that model predicts the outcome of every game, matching the current financial state of the world with exact accuracy. The only way to make the model more accurate would be to include religious factions, governments, armaments, armies, terrorists, police, multi-national corporations and every kind of psychopathic player available.
For deeper, more extensive consideration of the problem, a solution, holonomic ecometrics and monetary/credit systems, see my as yet unfinished paper at the Awareness & Value page
of my Greenbook blogsite. It includes new equations and formulas enabling integration of quantitative and qualitative data. It needs corrections and expansion, but you may find it helpful.
Thanks Lars, friends & allies ~ M
I agree with several of the assertions of Lars Syll, but it is true that his arguments are a bit of a mixture. Just as a way to improve the comprehension of the issues at stake, I propose to distinguish problems with econometrics in four different questions, followint Syll’s criticism:
a) The assumption of independence, linearity and additivity of explanatory variables.
b) Problems arising from the omission of variables, for the period(s) under analysis
c) The assumption that causal factors portrayed through the model will be stable along time.
d) The assumption that causes portrayed through the model will remain the same in different contexts (transferability of outcomes from the research context to other contexts).
Analysis based on econometrics (seems to) assume(s) the following, regarding each point:
a) Impacts are linear or tractable as linear (e.g. through logs), and the explanatory variables have not multiplying effects and can be assumed as independent.
b) Omission of variables may be treated through the hypothesis of random residuum, normally distributed (due to the central limit theorem)
c) The outcoming model may be used for other periods, assuming that the causation does not vary
d) The outcoming model may be applied to other context, following an analogous assumption
Assumptions (a) and (b) are strictly econometric, and are at its core. Assumptions (c) and (d) instead are more related to the use of econometric outcomes, both by theoreticians and practitioners. It is the belief that “universal scientific laws” are worth searching that leads to assumptions (c) and (d), as they allow “universal” conclusions. It is not Econometrics, as the Syll rightly points out.
Let’s turn now to (a) and (b).
• As to independence of variables, this is just a hypothesis. Econometrics per se is not able to solve this question, besides the usual calculus of correlations, as causality arguments are embedded in theory, and it is theory that have to decide whether there is or not relationship among two variables.
• Linearity and addictiveness are subtler issues. Theoretically we must allow that the influence of one variable over another may change in its intensity; e.g., the rate of increase of consumption of a given good when income increases is expected to vary with income (actually, a good may be normal for some bracket of income and next turn to be inferior). In this point, linearity may be too restrictive. But this is a rather technical issue, that perhaps more sophisticated econometrics will solve in the future: indeed, some non linear specifications are quite usual nowadays. Non linear forms are preferred when a constant elasticity is the desired outcome.
• Regarding the omission of variables (whatever they are), it is wrong to treat it through the notion of a random residuum. As Georgescu-Rögen pointed long ago, the omitted variables are by no means “random”, therefore the central limit theorem does not apply. On the other hand, the very notion of “random variable” is quite odd:
o The main idea is that we have a random process when for the same crucial inputs two or more outputs can occur. It is a deep (if solvable) epistemological problem to decide whether a process features randomness. Rolling a dice or throwing a coin – two noteworthy examples of randomness – are not random at all; are mechanical processes, just too complex to be predictable in analytical terms.
o The usual properties of a “random process” – especially, achieving similar probabilities for each possible outcome in a large number of trials – are inconsistent, because they somewhat assume that each trial is dependent from the others, when it is openly assumed that they are independent. It seems that there has been a confusion between the random tables o generators used for sampling and the very idea of random variable. The “random” function of the Excel, actually, is not a random process. And sampling actually is no more than a very useful heuristics, it is not “science” (see below).
There remain (c) and (d); as told before, these are economic questions, and therefore the answer to them must arise from Economics, not from Econometrics. The clue question is the following: Is Economics able or entitled to find empirical relationships that are transferable through time and context? This question is strongly related to the one asking if Economics can be thought of as a “Science”, in the same manner as Physics or Chemistry are deemed to be, as their outcomes are indeed transferable. The lab in the future or in other place is able to replicate the same processes and outcomes.
Syll clearly asserts that this kind of universal statements are precluded for Economics. Now, what allows Economics to be called or not “Science” is mainly a question of definition; and nowadays we do not have an agreed definition for it. As a matter of fact, we agree Syll’s idea that Economics’ universe cannot be treated as Physics’, for it is too complex and it undergoes evolutionary changes. Surely, if Economics is deemed to be a “science”, it will not be in the same sense as Physics, essentially because it deals with much more complex and evolving universe of phenomena.
But if this question – to be or not to be a Science – is rather definitional, it is not that important. We all agree that economic analysis is able to deliver useful assertions for the problems it has decided to face (at least for the problems that are really economic, i.e., leaving aside the pretension of being a Praxeology). For example, when Dani Rodrik argues that the Washington Consensus meant wrong policies for Latin America in the Nineties, this assertion is based both on empirical and theoretical grounds, and can be rationalized as such.
Of course, it is a contentious statement, and some people will assert therefore that it is not “scientific”, as it is not possible to “demonstrate” it is true. We may even agree that it is not “scientific”, if necessary; but this does not make it useless. On the same token, the assertion that car A performs better than car B is not scientific, but anyway the scientist when choosing a car surely pays attention to the opinion of experts and to test drives.
The trouble arises therefore when “Science” is called into the stage. Perhaps, it would be better to keep it in the backstage. We suspect that Economics cannot go much beyond this point.
Econometrics could then be accepted as a tool, a heuristics, and no more than that, especially if we bear in mind its limitations, as those pointed both here and in the Syll’s text. No more, but not less. As Syll states, the key of the issue of Econometrics’ use and usefulness is to be found in Economics.
Sorry for the length of this comment.
Lars’ critique of Econometrics, as conventionally practiced, is correct. This is because Economics, correctly understood, is a science of complexity, with chaotic features, evolving in both time and space in ways that are unpredictable according to conventional statistics and simplistic linear models. This is the “uncertainty” that Keynes was getting at.
This just says that economists have been applying the wrong mathematics and have had the wrong expectations. Nonlinear models can tell us a great deal if developed at an appropriate level of abstraction and scale in time and space, but they generally do so via numerical simulations of a variety of scenarios. Consider the famous limits-to-growth models from the Club of Rome work in the 1970s. These were dismissed by clueless economists, like Nordhaus, but did in fact forewarn us of the dangers of ecological overshoot and collapse that may be starting now (business-as-usual scenario) or within a few generations (more optimistic scenarios). But it is wrong to expect these models to predict a certain probability of collapse at a certain date.
Not that you couldn’t come up with such probabilities using powerful computers to run massive simulations of massive numbers of scenarios. But each scenario is based on a certain choice of parameters, so assumptions would have to be made on the probability distributions of those parameters. This is guess work at best, so one would have to try a variety of guesses as to the form and parameters of these probability distributions, and then how to weight the different guesses? The whole enterprise begins to get very doubtful.
Yet much can be learned about the risks even from a few well chosen scenarios, like the dozen or so computed by the limits-to-growth studies. In this case, these studies suggested that some form of overshoot and collapse would occur unless the most optimistic values of the original parameters were chosen, such rapid global prioritization of renewable energy over fossil fuels. And the business-as-usual scenario was like a worst-case example, with a strong risk of compounding failures as different macro variables interacted with each other, leading to a more rapid collapse than seems probably at the moment. Ugo Bardi calls this the “Seneca Cliff”. We could luck out and muddle through, but what if wars and disruption spread, resulting in debilitating pandemics and loss of production and trade – system breakdown? Only time will tell, but the risk, however difficult to quantify by conventional probabilities, is very real.
“As social researchers, we should never equate science with mathematics and statistical calculation. All science entail human judgement, and using mathematical and statistical models do not relieve us of that necessity. They are no substitutes for thinking and doing real science.”
Since when is economics a real science?
Social sciences are for idiots. For people who lack the intelligence to do real sciences, but still like to pretend they are.
Everything they produced till now is wrong and fails to explain or predict anything. Social sciences are doing what the real sciences have long since let go, namely to only do deductions. Staying in your mind and come up with wild theories and ideas. Wild concoctions, like the ones made by ancient greek philosophers and physicists.
,
Any model is by definition a simplification of reality, and so always is “wrong” in that sense. Models are necessary and useful in economics, where we have to strip away the less important factors to make an analysis manageable. When someone gives an argument in favor of a policy or provides an explanation for an economic occurrence, or a prediction, their reasoning is essentially a verbalization of a model. So there’s no way around models.
Econometric models provide a bridge between an economic model or concept of interest, e.g. an elasticity, and the data, and provide a story or motivation for numerical estimates that can add some data-based calibration to the economic model. None of it is true, but then again no one knows what is true, nor could they convince everyone else what is really true about the entirety of any economic question worth studying. The economic system is infinitely complex. It differs everywhere and at every point in time. Modelling helps researchers to identify sources of differing opinions and interesting issues for further study.
It’s not productive to argue about whether or not it is a science. Maybe the term “social science” creates more trouble than it’s worth and shouldn’t be used. This is just a semantic point and is in no way a criticism of the social sciences themselves. For example, most would agree that it is worthwhile to pay some people to devote their time to collecting and analyzing economic information to assist governments, businesses, and interested citizens. Econometricians are a small part of this.
Application of formal statistical inference in econometrics is not convincing to anyone who is not fully indoctrinated in the discipline, and even to some who are. This is because the model is not true, as mentioned above, and because the stochastic assumptions are laughably fanciful, except when the data set comes a from random sample survey or a lab experiment. I sense that most inference methods are not really taken seriously by anyone but econometric theorists. The issue is seldom raised within the profession because it is thought to be disrespectful to certain colleagues. I predict hypothesis testing and rote use of t-stats will fall by the wayside over time. Some statistical theory will remain in the discipline, but only to motivate the use of certain estimation methods in order to inform the estimator choice process for applied researchers.