Colleen Dilenschneider, The Cultural Sector (Unknowingly?) Reports Misleading Numbers. Why It Needs to Stop in 2021, Know Your Own Bone,  01 January 2021

Good riddance 2020, and welcome 2021! Let’s say farewell to historically misleading and skewed sector trendlines, too.

Understanding the perceptions and behaviors surrounding cultural entities has arguably never been as important as it is right now. While there’s hope on the horizon for a better year, it’s not likely to improve right away.  Early research suggests that a sizable percentage of the population may be waiting to be vaccinated before returning to visit cultural entities at usual levels. Still, there’s reason to hope better engagement numbers may be in sight!

With fewer visitors, tighter wallets, and less revenue, organizations may not have funds to fuel initiatives that aren’t likely to pay off from a revenue or mission standpoint. When cultural entities invest in swinging their metaphorical bats this year, there’s an elevated need to hit the ball. And hit it out of the park, preferably. (At the very least, this is not the time for striking out.) The best way to make sure that happens is to have good data informing strategies and plans.

But it’s not just about having good data about the impact of the pandemic. It’s also time to right some wrongs that have long been fuzzing up success for cultural entities even before 2020. For instance, the population of the United States has grown younger and more diverse in the last few decades. Are cultural organizations changing along with the nation’s demographics? It’s a critical question, and to get to the bottom of it, organizations need to have their numbers right.

As a sector, however, cultural entities generally don’t have their numbers right.

This Fast Facts Video kicks off the year by shining a light on some frustratingly prevalent ways in which the cultural industry – knowingly or not – historically skews its own attendance findings and thus has made it harder to spot urgent trends. It’s time to right the ship. Now more than ever, the long-term survival of the sector will depend upon accurate data and analysis. This video contemplates misleading practices that were already in place from 2010 to 2019. Given the events of 2020, they especially need to stop.

The good news? Once aware of these data “manipulations” – intentional or not – cultural executives may be primed to use findings more effectively and appropriately.

Bad Practice 1: Overlooking population growth in attendance trendlines

The cultural industry rarely adjusts for population growth, and this has long been an obstacle for understanding the relevance of our institutions. According to the US Census Bureau, the US population increased by nearly 7% for the ten-year period spanning 2010 through 2019. So, simply to keep pace with population growth, organizations should have experienced a 7% increase during the same duration. But of the 224 entities we track at IMPACTS Experience, 67% had flat or declining attendance in 2019 compared to their year 2010 baseline. That’s right – even before the pandemic, most were simply doing the same or even worse in terms of onsite attendance than they were ten years ago!

It is terrific if your entity increased attendance between 2010 and 2019… but it may not be due to your fantastic programs. It could also simply be a function of population growth. When organizations don’t adjust for population growth, they risk misleadingly proclaiming to boards, “Our attendance increased compared to last year!” Well, it did, technically. But if the population growth in your region was greater than that percent increase, then attendance actually decreased or flatlined, meaning the organization may be underperforming.

Why fixing this matters even more in 2021:

Needless to say, the pandemic changed things. If we ignore the anomaly that was 2020, then attendance growth for “flatlining” organizations would align with the 1.5% US population increase from 2019 through the beginning of 2021. However, in 2020, nearly every organization IMPACTS monitors – upwards of 99% – experienced flat or declining attendance compared to 2019. On average, when long-term closing and social distancing capacity constraints are taken into account, organizations achieved approximately 35% of their 2019 attendance levels in 2020.

There’s hope that come mid-2021, cultural entities may start moving from survival mode into a kind of recovery mode. Reaching 2019 attendance levels is an important focus to start, but to truly be back on track entities will need to attract 2019 attendance with consideration to population growth over the past year.

Bad Practice 2: Counting door swings as if they are all different people

The cultural industry generally counts every door swing as a different person. Hey, it’s a cheap and easy number to count! And it sure makes for impressive statistics.

The thing is… it’s woefully misleading. This assumption overlooks that the kind of people who visit cultural organizations tend to be, well, the kind of people who visit cultural organizations. Some people visit a single cultural entity multiple times. (Members and subscribers are more likely to do this.) Industry-wide studies that add up door swings are even more detrimental, as the people who have visited one zoo are likely to visit another – and they are often the same people who visit history museums or botanical gardens, too. Adding door swings double-counts people in a big way, leading entities to believe that the pool of people in the US who are interested in attending cultural organizations is larger than it actually is.

This practice also overlooks the importance of reaching new audiences. Data from the 224 entities we monitor shows that repeat attendance increased 44% from 2011 through 2020. At the same time, the percentage of people in the US who reported having attended any cultural organization declined by nearly 3%. We’re double-counting the same people at the expense of understanding the urgency of reaching new ones. When they count door swings as different people, cultural organizations neglect accurate, alternative measures of how many people we are actually reaching.

To figure out how many people visit, entities need to quantify unique visitors. This is a task for high-confidence, representative market research, and not for the addition button on a calculator. Organizations must stop saying, “We welcomed X guests this year” if they are not measuring unique visitation. Instead, entities counting door swings should say, “We received X visits this year.”

The first is a lie. Don’t lie.

Why fixing this matters even more in 2021:

Check this out! There are exciting reasons to pay special attention to fixing this bad practice this year, in particular.

A) Museums were even more trusted to provide accurate information in 2020. Museums, in particular, increased in their perceptions of being credible sources of information in 2020! To further strengthen this growing superpower, cultural entities may need to pay even more attention to what they report – not only about their content, performances, or exhibits but about their operations as well.

B) Unique visitation increased in 2020. How’s this for a win? Cultural entities are activating more inactive visitors during the pandemic than they did before. IMPACTS Experience measures the percentage of first-time and non-recent visitors (three years or longer) coming through the doors of 164 cultural entities in the United States. In 2020, there were far fewer door swings, but the people coming in those doors were different – and that matters. When an organization only counts door swings or establishes this as the most important metric, then it becomes unable to recognize what is happening with the actual people behind the metric. Thus, it becomes unable to recognize opportunities to actually reach and engage individuals.

“What?! This is a dramatic increase! What’s going on here?!” It sure is. We’ll write a dedicated article in the coming weeks with more information. For starters, people currently prefer day trips in their own vehicles, and so local audiences are more likely to visit than people coming from further away. Safety perceptions, redistribution of demand for leisure time, competition, and planning timeframes have also changed due to the pandemic. These factors are also contributing to the engagement of new audiences. (We’ll publish more on this later. Subscribe here so you don’t miss that article.)

For now, the point is this: When you only count door swings instead of people, you miss opportunities.

C) Who said your quantity of guests matters most? There’s a strong data-based case for the impact of repeat visitors in cultivating support, changing up audiences, and rallying people around your mission. If a goal is to connect communities to your cause, then why all the emphasis and misleading metrics touting the glory of one-and-done guests?

 Your members and subscribers matter – a lot – and there’s reason to believe that they will be critical for recovery over the next couple of years. We saw an increase in how much they mattered in 2020 and we expect that to continue in 2021. They are more revenue efficient, they inspire others to visit and join, and they tend to feel a connection to your mission.

No doubt about it: Attendance numbers matter. But what matters most are people, not door swings. We hope that organizations will be more accurate about this in 2021 so that they don’t miss opportunities for recovery.

Bad Practice 3: “Omitting” the addition of new organizations or programs to totals

Unlike the previous two points, this practice relates mostly to industry-wide reports. This is data trickery of which cultural executives should be aware: Several industry groups tend not to adjust for new enterprise in their reporting.  Some associations and industry groups have become fond of quoting impressive attendance-related statistics along the lines of “our member organizations experienced X million more visits in the last ten years.” Well, of course they did…

Consider this: Major League Baseball welcomed nearly 15 million more visits to its ballparks in 1992 than it did in 1998. That’s a massive increase! Did Major League Baseball figure out some magical engagement formula? No. It was the product of new enterprise. In those intervening years, Major League Baseball added four new teams.

Now, think about how many new aquariums, or art museums, or theaters opened between 2010 and 2019.  Did these new institutions mean the sector was better engaging audiences?  Or did they merely provide more opportunities for many of the same people to visit?

Why fixing this matters even more in 2021:

There’s no problem with reporting this number! It’s often the truth, and it’s important that associations look good. They can help secure sector funding – and that benefits cultural entities! The problem comes when independent cultural executives grab onto the number without considering the broader context. “X more people visited my-organization-type last year than the year before!” It sounds good, but it can give executives a misleading view of the state of the industry when it isn’t tied to the context of additional enterprise in the number.

With the 2020s containing some closures for the cultural industry, it’s time to regret past touting of misleading numbers and do better moving forward. (To be fair, there’s no reason yet to believe that this bad practice won’t be canned by associations and industry groups this year. It probably will.)

Bad Practice 4: Adding together different survey methodologies

Sometimes organizations with which we partner will ask us if we can “combine” our data with their own (collected using different methods) in a single study. They are hoping for a single number or two that provides an overview.

Nope.

It is generally impossible to do this and still be able to maintain data integrity. This is true for studies that combine metrics within a single organization using different methodologies, and especially those that are self-reported and combined from multiple organizations. If you’re looking at a survey or report that combines data from multiple sources or methodologies, put it down and run away fast. Non-standardized inputs yield useless outputs. Period.

This point calls out a few low-budget, bad-math, regrettably cited, and detrimental studies used within the cultural sector. It’s alluringly easy to throw a bunch of similar metrics collected by different people and methodologies into a pot to claim overarching findings. Of course professionals want to empower one another and share information. (That’s great!) Indeed, the clinical rigor required for accurate data can be pricey. But having no data is better than having bad data – no matter the intention.

There are solutions here: Make sure methodologies are rigorously standardized, for one. Partner with other institutions and share the investment to engage a professional market research firm – or request that associations take on this task. At the very least, make sure your data and evaluation talent on staff has a heavy hand in these projects. Data collection and evaluation is a skillset. Engage the people who have it on the front end so that you don’t make investments based on bad data on the backend.

Why are these bad practices so prevalent in the cultural industry?

How did this happen? How have cultural entities managed to become so accustomed to misleading information that leaders don’t even recognize it when they see it?  Well, that’s the primary answer: Too often, bad data is “How we’ve always done it.” Today, technology enables more precision in understanding audiences and their perceptions and behaviors than ever before. But two other reasons may explain why leaders have grown accustomed to charging forward without thinking critically about data.

A) Organizations are hungry for data, and it takes talent and treasure to secure it.

Cultural executives are increasingly understanding the importance of making data-informed decisions. Unfortunately, money is often tight. When one combines a low budget, a lack of research expertise, a want for data, and magical thinking, then bad data may be inevitable.

It’s a common saying that something can be fast and cheap, cheap and good, or good and fast. You can’t have all three. Generally, this is true for data, too. If “good” is not one of those factors when it comes to data, then an organization is wasting money and harming itself. Bad numbers make for bad decisions. The only way to get good data is to invest in it. Whether you are conducting evaluations or audience research (which can in some cases be done thoughtfully with tools like Survey Monkey, etc.), or investing in market research (which often requires more advanced methodologies and practices), good data requires time, talent, and treasure. There’s no way around it.

B) Executives forget that associations sometimes need data for different reasons than individual entities.

This is an important distinction. Our associations are the go-to for information for lobbyists, funders, and reporters. It is their responsibility to make the case for culture. The number of door swings to museums and performing arts entities is an impressive number, whether it tells the whole story or not! Presuming it derives from a study conducted by data scientists and abides by best data practices, the number is not a lie.  Associations and industry groups can be our biggest sector advocates. To an extent, they need good-news numbers…and we all need them to have the best-news numbers possible!

But you shouldn’t overlook this when you make business plans for your own organization. To better spot opportunities to engage audiences, leaders may benefit by being more aware of the context of the numbers they’re using. Cultural executives need numbers for the purpose of successfully leading and growing the organizations that associations are supporting. It’s not necessarily the role of associations to call this out to executives. It may be a leader’s job to realize that numbers tell stories, and the stories that some associations need to tell to do their jobs successfully can be different than the more detailed stories that you need to do yours.

Associations need the “good news” to garner support for organizations because that may be the most important way that they support cultural entities – especially now. Perhaps the second most important way that associations support entities is by connecting leaders…so that those leaders can work together to uncover the opportunities. Outlining opportunities means investing in seeking out the “bad new” to find critical areas for improvement.

As independent companies, IMPACTS Experience and other market research entities are better able to report these resource-intensive, detailed, not-often-sunny numbers for the sector. We are an insider resource for executives, not a public go-to for broader sector funding and morale-boosting when needed. If the “tough love” hat looked as good on associations, it may be harder for them to connect leaders and garner external support for the organizations they serve.

Success in 2021 and beyond may require even greater connection between organizations. It may depend upon leadership sharing insights, lessons learned, successes, and war stories to get through this still-very-difficult time. Let’s be hopefully optimistic, but let’s also realize that 2021 is likely to be rather difficult. (To that end, we’ll provide market potential regarding expected attendance for exhibit and performance-based organizations on Wednesday, January 20^th.)

In the meantime…

Three cheers for cultural organizations.

Three cheers for the associations that support and champion them.

Three cheers to other partners (we see you!) – providing behind-the-scenes assists to safely point out blind spots, aiding in proverbial air-traffic control, and doling out expertise to support beloved institutions.

And three cheers for you, cultural leader.

We’re all on a great team. Let’s make this year a good one.