Rejoicing in Uncertainty

A review of "The Primacy of Doubt" by Timothy Palmer

The book review editor of Physics Today,the physics journal of the American Physical Society, commissioned me to write a review of Timothy Palmer’s book, The Primacy of Doubt, late last year. I submitted my review in the spring, and a month or so ago, I was informed it would appear in the September issue of the magazine, and I should expect proofs within a few weeks. Sometime last week I got a note that they had changed their minds, and the review would no longer appear, and gave what seemed to be a contrived excuse for this. This is a pity as it is an interesting and useful book, for physicists and non-physicists alike. Because it will not appear there, I am publishing it on Critical Mass.

The Primacy of Doubt, by Tim Palmer

There is perhaps no subject more misunderstood among the public than the nature of uncertainty in science.  In almost all other areas of human affairs, uncertainty is viewed with attitudes that range from suspicion to derision.   In science, however, uncertainty plays a central role.  More specifically, science is the only area of human intellectual inquiry where we can quantitatively describe the confidence with which we are making a prediction, or the accuracy of our measurement of some quantity.  Indeed, not only can we do this, we must.  Without error bars, numerical quantities quoted in the scientific literature would be largely meaningless.

Writing a popular book about the nature of uncertainty in science would itself be a daunting task but Oxford theoretical physicist Tim Palmer, whose work on meteorology and climate prediction is heralded, but who began his studies in the esoteric areas of general relativity, cosmology, and quantum gravity, has masterfully taken on a far greater challenge.  Instead of discussing the role of uncertainty in science, he tackles, head on, the science of uncertainty itself.  To do this, as only a polymath would be able to do, he describes not merely his own experience helping revolutionize meteorology and climate prediction by applying ideas from Monte Carlo analysis, fractals, and chaos theory, but also how these ideas might usefully be applied to studying pandemics, the financial world, conflict and risk analysis, and ultimately to the quantum world and consciousness itself.

Palmer’s own scientific experience began doing graduate with the distinguished cosmologist Dennis Sciama, who had assigned Palmer the task of considering the principle of maximum entropy production in the context of black holes.  He was then offered a postdoctoral position with Stephen Hawking, but a nagging worry about the conceptual basis of quantum gravity bothered him. A fortuitous meeting with a famous meteorologist Raymond Hide sealed the deal.  Hide described a paper by and Australian colleague on how the principle of maximum entropy production might apply to studying Earth’s climate, and Palmer was hooked. He accepted a position at the UK office of Meteorology, and over his career has dramatically improved weather forecasting and climate modeling through the tools he so clearly describes in his book.

Palmer makes it clear right off that doubt and uncertainty are his primary focus in the book not just because of practical concerns about reliability, but because of his conviction, validated by the examples he later provides in the book, that “we may be able to understand better the ways systems work by focusing on the ways in which they are or can become uncertain.” As he then puts it: “These reasons form the two themes of my book:  The science of uncertainty to predict our uncertain world and to understand our uncertain world.”

The book is divided into three parts:  The Science of Uncertainty; Predicting our Chaotic World; and, Understanding the Chaotic Universe and Our Place within it.   From my perspective, however, the book is really two distinct books.  The first deals with the well-tested real-world applications of the science of uncertainty and ensemble modelling.  The second is a more speculative, if albeit fascinating, musing on the possible fundamental nature of uncertainty in our theories of the world.  In particular, Palmer suggests that ultimately the inherent uncertainty in Quantum Mechanics may be a signal of an underlying non-quantum, yet chaotic, world, and in a similar vein—reminiscent of some ideas promoted by his Oxford colleague Roger Penrose and his collaborators—how a fundamental understanding of consciousness and free will may rely on an understanding of the geometry of chaotic systems.

The book begins with an anecdote about a poor BBC weather forecaster, Michael Fish, who on October 15, 1987 famously dismissed a viewer’s concerns about a possible imminent hurricane.  On Oct 16^th the worst storm to hit the south of England in over 300 years battered the coast with hurricane-strength winds.  This motivates his subsequent discussion.  This motivates his subsequent discussion about the tough problem of predicting uncertainty in systems where underlying causes that can seem small and insignificant can nevertheless produce big effects—a signature of chaos.

What follows is a wonderful and personal exposition of the key geometric ideas associated with chaotic systems.  These chapters show how exploring the phase space of possibilities, through ensemble modeling and Monte Carlo techniques—the tools that Palmer helped introduce into both meteorology and climate modeling—can be successfully used to address chaotic systems.  They provide a crystal clear and well-motivated explanation of the geometry of chaos and how it is central to understanding the long-term behavior of these systems.  Subsequent chapters on Pandemics, finances, and risk analysis then provide useful examples of how rational approaches to everyday problems are not necessarily intuitive—stemming from the fact that, as Steven Pinker emphasized in his recent book, Rationality, we are often anything but rational in part because probability and statistics produce non-intuitive results. 

In the last part of the book Palmer returns to his early concerns about the fundamental conceptual problems of physics, notably the question of whether quantum mechanics is truly fundamental or emergent.  Bell’s and subsequent authors’ demonstrations that hidden variable theories are not consistent with experiment is predicated on the difference between how classical physics and quantum physics deals with what is going on in unmeasured systems and how they predict what the results of a variety of possible measurements might have been.  Palmer bravely and creatively avoids inconsistency by arguing that at a basic level nature is chaotic, and certain alternative paths simply can’t exist.  If you measure a particle as having spin +1 in the z direction, it makes no sense to imagine what the result would have been had you measured another axis, because at a fundamental level that was never a part of nature’s phase space of allowed possibilities.

These are not crackpot ideas. Palmer has published them in reputable journals, and they are nothing if not creative.  I found them fascinating, even if I remain skeptical.   Nevertheless, I prefer it when authors do not include their own pet theories in popular books—a freedom several notable authors have abused over the past decades—because the public has a hard time distinguishing well tested science from more speculative ideas.  I don’t fault Palmer hard here, because these latter ideas do not form the central premise of his book, and he also makes it clear that they go against the current grain.

That quibble does not diminish the worth of this book. It just makes it more provocative.  In the end, Palmer has not just written a clear and fascinating book about uncertainty in science, he has written one that is sorely needed.