Friday Notes, March 3, 2023
Dear Friends –
Those of us who believe in the value of data and evidence to make people’s lives better through social policy peer over the fence at medicine with admiration and a bit of envy. Gosh, look at how much is spent by the government on rigorous health research! See how systematic reviews are used as the basis for practice guidelines issued by government authorities and professional membership groups! Watch national regulatory authorities like the Food and Drug Administration set the evidentiary bar so high that drug makers invest massive amounts in high quality clinical trials! Look at private insurers setting up independent expert panels to make recommendations for coverage decisions! Truly, we’re tempted to think, this is how the “evidence-to-policy pipeline” should work for all consequential public policy.
This is a compelling aspiration. We see it reflected in the attention given to findings from social experiments that mimic clinical trials, and in funding “what works.” It is this model that has inspired not only the increase in field experiments using randomization, but also the adoption of research transparency norms like pre-registration of study designs and attempts to replicate research findings.
Along with many others, I’ve contributed over the years to trying to figure out what aspects of the evidence-based medicine model can be successfully adapted to increase the use of empirical analysis within the policymaking process. Now, though, I’m worried that the medical research model is not the right analog, and we need an alternative. Here’s what’s concerning me:
Many of us have romanticized the objectivity of evidence-based medicine and, broadly, the use of research within clinical and public health decision making. Despite a lot of brilliant people working on establishing standards of evidence, there are massive internal debates within medicine about how to judge study quality – and which studies to include or exclude when aggregating evidence. There are questions about how studies of one population can be generalized to others. There are judgment calls about what counts as “cost-effective,” and what outcomes are more important than others. And all that is compounded by the disconnect between what practice guidelines say and what practitioners do.
The norms and structures that facilitate the use of evidence in medical decision making are not just absent for social policy; they are anathema to the status quo. For example, in clinical medicine, the use of evidence is part of doctors’ professional identity and provision of “proven” treatments is a selling point for hospitals, health systems, and payers. In social policy, evidence is at best secondary within decision makers’ identities to ideological affiliation and responsiveness to constituents. In medicine, there’s a basic shared understanding of what constitutes a good outcome (i.e., longer, healthier life) and so there can be rules and regulations that are meant to protect the vulnerable from phony claims and treatments that do more harm than good. In social policy, there is rarely consensus about what is a positive or negative outcome; I might think access to affordable childcare is good idea so both parents can go to work, but you might think it's better for children to have full-time parental care. If we cannot even decide on what outcome is best, it’s going to be hard to put in place processes that constrain policymaker choice.
So let’s say we give up on the medicine model. Is there another that might be better, or do we have to start from a blank sheet of paper? One that I’ve been thinking about is the market, or consumer, model.
Buyers, whether individual or corporate, use a lot of information about themselves to make the best choices. They know their own budget. They know their preferences – what features they care most about, and what compromises they can live with. And they know how much information they need about the universe of choices to make a particular decision with confidence. Budget, preferences, and information needs all have parallels in the policy arena.
To design products that respond to customer needs and preferences, sellers spend significant effort understanding users and context. In business-to-business sales, they develop deep understanding about who holds decision making power, and the incentives shaping their choices. Sellers reveal their prices, and know that they’ll be in the strongest competitive position if they anticipate and credibly answer potential customers’ questions.
Information isn’t available to buyers only from sellers, who may make extravagant claims. It’s also available from resources like Consumer Reports, which reports on product and service quality across multiple dimensions to which buyers can apply their own weights. And, increasingly, we all count on star ratings and reviews from hundreds or thousands of people who have direct experience.
Envisioning the application to social policy: What if those of us committed to the best-informed public policy thought of politicians and other government decision makers not as groups of doctors with a common aim of healthy patients, but rather as buyers of solutions to problems as they define them? Would we produce studies of program innovations that provide information about the types of performance that the policy makers care about – along with the cost of the getting the desired outcomes? Would we provide head-to-head comparisons, as Consumer Reports does? Would we figure out ways to capture and share the subjective experience of people affected by similar programs?
I’m pretty sure the evidence-based medicine model isn’t helping us move the ball forward as fast as demanded by the crucial social and environmental problems we face. We need a new way to think about this challenge. What could it be?
The National Park Service has done it again. First they brought us the dancing skunk. Now they’re warning us (again) against sacrificing a slower friend to charging bear.
The social media team at the NPS is super-savvy and offers a great illustration of how a government agency can communicate information with humor and personality. (Here’s a good interview with the head of the team.) Fun to follow, and an example for the rest of us.
Here are my new addictions, which I’m guessing you’ll like, too:
Worldle, a country-a-day geography game
Tradle, a country-a-day game about the composition of national exports
Have a good weekend,
-Ruth