Every measurement1 is a half-truth. Not a deliberate one, but an inevitable one. The kind of half-truth that arises from the gap between what we want to know and what our instruments can tell us. A thermometer does not measure temperature; it measures the expansion of mercury, from which we infer temperature. A bathroom scale does not measure mass; it measures the deformation of a spring. A telescope does not show us a star; it collects photons that left the star years ago and passed through the expanse of space and an atmosphere that scattered and bent them along the way. Between the thing we care about and the number we write down, there is always an instrument that distorts, degrades, smears, and obscures the signal.
This is not a problem unique to science. The inverse problem is a feature of all observation, all inference, all attempts to understand. When one listens to a friend speak in a crowded restaurant, one is solving an inverse problem; the problem of extracting a voice from a mixture of voices, clattering plates, background music. When doctors read an X-ray, they infer the conditions of their patient’s bones from shadows cast on a screen. When investors read a quarterly earnings report, they seek to separate genuine business performance from local choices and global macroeconomic effects that have nothing to do with the company itself.
The inverse problem is the deceptively simple question that unites all these situations: given what was observed, what truly happened? This inverse problem is among the most important and most underappreciated intellectual challenges in modern life.
* * *
The forward problem is easy. Given the true state of the world and the exact properties of one’s measuring instrument, however crude it may be, one can predict what the instrument will read. This is because the forward process is deterministic. That is not to say it is easy; as instruments and the distortions they induce into their measurements get increasingly complex, accurately modelling and applying the forward process can become correspondingly more challenging. Nonetheless, philosophically, the forward problem is a matter of simulation: start with truth, apply a known process, and arrive at observation.
The inverse problem, however, is almost always harder. Given the output of an amplifier, there is no recipe to deterministically uncover what the original signal was. Given a Polaroid photograph there is no algorithm that can conclusively reconstruct what the scene looked like. This is a fundamental, conceptual challenge intrinsic to the inverse problem, which is a consequence of the nature of information itself rather than the specifics of the complexity of the instrumentation. Once information is lost in the forward process, it is necessarily and irreversibly lost.
When a camera lens blurs an image, sharp edges become soft gradients, and many different sharp images could have produced the same soft one. When noise corrupts a signal, the original is hidden behind randomness that cannot be subtracted away through any amount of computational power any more than milk and sugar can be separated from coffee by stirring hard enough. The inverse problem is what mathematicians call ill-posed: there is no unique solution, and small errors in the observation can produce wild swings in the inferred underlying truth.
In this sense, ill-posedness, being a statement about the nature of knowledge itself, cannot be engineered around by throwing increasing amounts of computing power at the problem. Attempts to do so, I would argue, are rather wasteful. Because any observation, no matter how careful, always underdetermines reality, any attempt to solve the inverse problem must be grounded in the understanding that there is more than one story consistent with the evidence. Such methods therefore attempt to characterize the set of plausible stories and understand what distinguishes them, rather than boldly claiming to determine the single true story.
* * *
It is possible to read the history of science as a history of increasingly sophisticated approaches to this problem. Early astronomers corrected for atmospheric refraction with lookup tables. Nineteenth-century statisticians developed least-squares fitting to extract planetary orbits from noisy positional data. The twentieth century brought Fourier methods for deconvolving telescope images and Bayesian frameworks for incorporating prior knowledge into uncertain estimates.
Through this lens, each advance in science has followed a common pattern: someone recognized that a measurement was contaminated by a systematic distortion, developed a mathematical model of that distortion, and then figured out how to run the model backward, or in some cases, sideways, to recover something closer to the truth. The corrections have never been perfect, but they have always been better than ignoring the problem.
What changed in the twenty-first century was the scale and complexity of the distortions we could handle. Machine learning, for all the hype that surrounds it, is at its core a technology for learning complicated functions from data. And the functions that describe how instruments distort reality, the forward models, are often rather complicated indeed. Too complicated to write down analytically, but not too complicated to learn from examples. If one can simulate the forward process, one has, at a minimum, overcome the engineering challenge of working with this level of complexity. Even if it were to provide no further use in solving the inverse problem, machine learning would have provided one with a foot in the door through the forward problem.
I would, however, contend that machine learning methods have a much more far-reaching scope, so long as these tools are applied with a kind of disciplined humility that is easy to lose in the excitement of a new technique. A model can be taught to invert a distortion, if one remembers that its prediction of the truth is only as good as its understanding of what the distortion is. Trained on simulations that do not faithfully represent reality, it will produce answers that are confidently incorrect. The inverse problem punishes overconfidence more severely than almost any other setting in quantitative reasoning, because the ill-posedness means that small errors in the assumptions propagate into large errors in the conclusions.
* * *
This brings me to what I think is the deepest lesson of working with inverse problems: the critical importance of knowing what one does not know.
Across many human intellectual endeavors, the ultimate objective is getting the right answer. To that end, we celebrate accurate predictions, tight confidence intervals, models that match the data. But in inverse problems, the quality of one’s uncertainty estimates matters as much as the quality of one’s point estimate, perhaps more. A point prediction like \(x = 42\) says little on its own. A prediction of \(x = 42 ± 3\) with \(90\%\) confidence is informative only when that confidence is calibrated, that is, when, over many predictions of this kind, the true value falls within that interval about \(90\%\) of the time, neither systematically less nor systematically more. Otherwise, the estimate is not just imprecise, it is misleading. Decisions made on the basis of such a prediction, whether to build a bridge, approve a drug, or invest in a company, will be systematically miscalibrated.
This is why the best work in inference is not the work that produces the most impressive-looking results, but the work that most honestly characterizes what can and cannot be known from the available evidence. There is intellectual courage in reporting wide error bars when the data demand it and admitting that one cannot distinguish between two hypotheses. There is also a less-appreciated kind of intellectual courage in correctly reporting narrow error bars and publishing a result that might ruffle feathers rather than ducking behind inflated uncertainties to avoid breaking with consensus. The inverse problem teaches one to respect the limits of one’s data, because those limits are not negotiable.
* * *
Outside of science, inverse problems are everywhere, though they are rarely referred to by that name. For example, consider medicine. A patient presents with symptoms: fatigue, weight loss, intermittent fever. These are the observations. The underlying disease is the latent variable. Diagnosis is inversion: what condition, influenced and distorted by the “instrument” of the human body and then the diagnostic tools available to the physician, would produce these readings? The difficulty once again is that many diseases share symptoms, that the symptoms are reported imprecisely, and that some conditions are simply invisible to the available tests. Broadcast media often celebrates the “genius diagnostician” who always gets the right answer on the first try,2 not the physician who systematically considers the range of possibilities, orders tests that most efficiently distinguish between them, and updates their beliefs as new information arrives. The moral of the story, I suppose, is that solving an inverse problem with appropriate uncertainty quantification makes for rather boring television.
Finance offers perhaps the most consequential everyday example. Every price in a financial market is, in a sense, the output of a forward process. From the true state of the economy, filtered through the expectations, biases, liquidity constraints, and information asymmetries of millions of participants emerges a single price. The investor’s job is inversion: given the prices, what is the true underlying economic state?
This is hard for all the usual reasons. The market is not a passive instrument. It responds to the very inferences people draw from it, creating complex and layered feedback loops. For example, a large trade can move the price it is trying to estimate. The noise is not random; it has structure imposed by market microstructure, regulatory regimes, and the behavioral tendencies of other participants. And the latent state, the “true value” of an asset, may not even be well-defined in the same way that the temperature of a gas is well-defined whether or not one can correctly measure it.
Yet the framework of the inverse problem remains illuminating. It clarifies why some kinds of analyses work and others do not. Strategies that model the forward process carefully by understanding the market mechanisms that produce observable prices already have a leg up over strategies that simply look for statistical patterns in the prices themselves. The former is grounded in a causal model of how observations are generated; the latter is vulnerable to the same ill-posedness that plagues any attempt to invert without understanding the forward process.
It also clarifies why overconfidence is the most dangerous failure mode in quantitative finance, just as it is in scientific measurement. A model that claims to extract signal from noise with high precision is making an implicit claim about the structure of the noise. If that claim is wrong because the noise has structure that the model does not account for, then the “signal” the model finds may be an artifact. The history of quantitative finance is, in significant part, a history of mistaking artifacts for signals, of treating ill-posed problems as though they were well-posed, and of learning the hard way that uncertainty is not optional.
* * *
So, what does it take to see through noise? Not perfect instruments. Those do not exist. Nor unlimited data. That is never available. What it takes is a clear-eyed understanding of the forward process, an honest accounting of what information has been lost, and the discipline to report what the data support rather than what one wishes that they supported. It also requires a willingness to bring multiple perspectives to the same problem. The best solutions to inverse problems almost always combine quantitative methods with domain knowledge. An understanding of the physics, the biology, the economics, or whatever generative process produced the observations is invaluable and irreplaceable. Pure mathematics can determine that a problem is ill-posed. It can also determine which class of solutions is consistent with the observations. It cannot, however, determine which of the many consistent solutions is physically plausible. That judgment requires expertise in the domain, and it is the reason why the most productive work happens at the intersection of disciplines.
Finally, it requires a tolerance for ambiguity that can be psychologically uncomfortable. The inverse problem does not always have a clean answer. Sometimes the data support multiple interpretations, and the most honest conclusion is to admit that one does not yet know, and that it is possible that one may never know. In a culture that rewards confident assertions in science, in business, and in every aspect of public discourse, there is real pressure to overstate what the evidence shows. Rigor lies in resisting that pressure.
Every measurement is a half-truth, but not all half-truths are equal. Some measurements, made with care and interpreted with discipline, can bring one as close to the truth as the laws of information theory permit. The challenge in any endeavor that depends on inference is to understand exactly how close, and to act accordingly.