On Monday, writer Steven Johnson responded to our criticism of his August 23 New York Times Magazine cover story with a follow-up story in the Times and a more data-centric reply in the comments of our blog.
We appreciate Johnson’s detailed response and his very kind words for our organization. Moreover, we appreciate his demeanor and good faith in responding to our critiques. It’s not fun to endure these kinds of criticisms (indeed, we’re not the only ones who had problems with that article), and he has done so with civility and good humor.
That, said, we stand by our criticism of Johnson’s reporting.
Johnson’s response to our critique is titled “Can Data Capture the True Health of the Creative Economy?” But that’s not what our disagreement is about. It’s about the simplified story Johnson tells, and whether the particular datasets he uses to tell it are an appropriate proxy for the musician population. It’s about whether he’s given his readers the necessary tools to meaningfully understand the data he presents, including the limitations of that data.
Let’s be clear: in criticizing Johnson’s selective presentation and framing of data, we don’t mean to suggest that he’s simply willfully pushing a rosy techno-utopian point of view. We think he’s sincere in wanting to see musicians and other creators thrive. Rather, Johnson’s thinking appears to reflect a number of extremely common blind spots and logical leaps that frequently emerge when generalist writers—even very good ones—venture into writing about the music business. Music is a particularly complex set of industries, but when you primarily experience music from the consumer side, much of the complexity is invisible. Without spending a lot of time navigating the systems and heterogeneous business models musicians encounter in their day to day lives, it can be hard to perceive just how much isn’t in your field of vision (or in your dataset). This can easily result in analytic errors that can calcify into faulty conventional wisdom when repeated uncontested, especially in public forums with the reach of the New York Times Magazine.
The best defense against these mistakes is familiarity with the broad diversity of the music industry; the kind of familiarity that is developed with deep and sustained work with a range of musicians who employ varying business models. That isn’t about accepting anecdotal accounts as evidence, but rather having enough interpretive perspective to contextualize data. Sometimes that means resisting the pressure to create cleaner simple narratives (we had to chuckle as just this morning the NYT reported on a new study that suggests the pressure to create such narratives leads researchers in other fields to overstate their conclusions).
Johnson doesn’t like the suggestion that he’s less than balanced in his presentation of data, and offers examples of passages he might have written had he simply cherry-picked encouraging data points or anecdotes. But a nod towards complexity by mentioning a handful of more neutral findings isn’t a substitute for an actual reckoning with the gaps in the data. It’s adding a veneer of balance without actually addressing the shortcomings of an oversimplified story.
Note that there are a number of topics where, 3000 words later, Johnson is essentially nonresponsive to our critiques. Among them:
- The failure to fully account for the gaps in data sets. (For example, is he actually comfortable making arguments about how professional songwriters fare based on a dataset where 55% of the population are people working in elementary/secondary schools?)
- The failure to assess how increases in gross live revenues translate to musicians revenue, or account for touring expenses.
- The failure to assess the methodological questions associated with Pollstar’s data (this criticism was echoed by Paul Krugman).
- The failure to meaningfully qualify who benefits from newer revenue streams, and how substantial they are for those who participate.
- The flattening of multiple distinct lines of economic, social, and technological criticism into one unified theory of a “creative apocalypse.”
Johnson seems to misunderstand our critique when he writes: “From my point of view, not looking at the data would be making the perfect the enemy of the good.” FMC would never advocate not looking at data. Instead, we call for closer examination of data, which includes understanding its limitations. In situations where the existing data is insufficient to prove or disprove a thesis, we call for more and better data, and to the extent we’re able to as a small non-profit, we’ve endeavored to build more complete datasets ourselves, with methodology that fills in some of the gaps and illuminates the complexities.
The post’s authors write: “If you want to know how musicians are faring, you have to ask musicians, preferably a whole lot of them. You’ll get different answers from different musicians, and they’ll all be correct in terms of their own experiences. But your overall understanding will better reflect the complexity of the landscape.” This is certainly true, and the F.M.C. is doing a tremendous job asking those kinds of questions. But as a society, we still find it useful and instructive to look at, say, national unemployment or G.D.P. statistics, even though there will be “different answers for different employers” inside those numbers. Sometimes it’s useful to zoom out and take the bird’s-eye perspective,
In fact, the shortcomings of the “birds-eye perspective” metaphor are quite instructive. There are approximately 10,000 species of birds in the world, and they are amazingly diverse. They thrive in different biomes. Some fly, and some swim. Some are easier to study than others. While their basic needs are the same—food, water, sustenance—changes to habitat impact different bird species in different ways. A “birds-eye view” will look very different depending on whether you’re talking about an emperor penguin, a California condor, or a house finch. So it is with musicians. Things look different from the perspective of New York session players than they do for Nashville songwriters, from jazz arrangers to hiphop producers. And yes, data points assessing the overall health of particular bird populations are possible, and desirable, whether it’s good news or bad, but it’s only possible with well-defined research designed by specialists to account for complexity. The Audubon Society wouldn’t be satisfied with a study that lumped easy-to-count factory-farmed chickens in with hard-to-count songbirds.
The F.M.C. authors are undoubtedly right that it is impossible to know exactly how many musicians are working in the United States right now, and how much money they are making. This is true of dentists and freelance writers and the manufacturing sector as well. It’s in the very nature of national economic storytelling that the data are rough and a little murky.
This too is instructive, because it suggests that he still fails to comprehend the unique challenges associated with studying the musician population, and how that compares to other industries.
For example, everyone agrees on who is a dentist. To be a dentist in the United States, you need to have met a certain set of educational requirements, and you must be licensed by your state board of dentistry. That makes it much easier to collect good numbers. No such requirements exist for musicians. There are no agreed upon standards for who counts as a musician.
Dentists have much less variation in their business models. Dentists earn their living by taking care of people’s teeth. Their compensation comes primarily from insurance companies and from consumers. There are fewer roles and revenue streams and fewer kinds of expense to take into account. Further more, there is much less variation in how dentists participate in the marketplace. No one works service jobs so they can spend a few months out of the year practicing dentistry. Government statistics that aren’t designed with any particular population in mind can thus do a much better job of capturing information about the dentist population and their gross revenues.
These research challenges are real, and should not be minimized. Neither should they be considered insurmountable. Once again, the key, in all cases, is listening to musicians themselves; preferably lots of ‘em. If Johnson had done so, we suspect he’d have written a better, truer story.
(image via Shutterstock.com)