Sparking Change: How to Design, Administer, and Use Survey Data to Drive Organizational Growth

How can survey data spark change?

Cantate, one of the Washington DC area’s premier choral organizations composed of the 32-voice Chamber Singers ensemble and the larger Concert Choir, is using data to measure its progress on reaching a representative swath of its community. According to executive director Cara Schaefer, Cantate started doing demographic surveys in 2023 because a glance at its audience showed that it was not representative of the organization’s larger community. Cantate initially set out to collect baseline data on those attending its concerts, and is now comparing current data to its baseline and to county census data. “It’s stark,” Schaefer says. “We have a lot of work to do. But it shows us […] what direction we’re going, where we need to be stronger, and it will inform how we build our ADEI plan.” Cantate knows that there is a disparity in the diversity of its audience versus the diversity of its community, and communicating its desire to change that via the data it is collecting from surveys has emphasized its importance to stakeholders. “Data-ize big ideas,” Schaefer says. “It makes it easier for people to process them.” Once Cantate measures and understands this disparity, it can find ways to build bridges and if it is successful, its audience surveys will measure the closure of the disparity over time. So far, the data has energized Cantate’s board to reinstate its advisory board, with members intentionally recruited to reflect the area’s demographics. The advisory board will be compensated for meeting several times a year and providing insights into its community, according to Schaefer. Furthermore, each member will be given six tickets to Cantate’s concerts to share in their circles, and audience members in their parties will be awarded gift cards for responding to surveys about the impact of each performance. “We’re playing the long game here, as there is no evaluating trends before at least two years of data is collected, there is no responding to these trends until after that, and there is no judging the efficacy of these responses until well after that,” says Schaefer. “Though it’s a truly long-term investment, our people appear to be on board, and we’re confident in the buy-in of our attendees and the worthiness of the effort.” VOX Femina, a 45-member women’s ensemble located in Los Angeles, has three goals for its audience surveys, according to executive director Rebecca Wink. The first goal is to find out what the audience thinks about its concerts and make sure the organization is meeting their needs. The second is to collect demographic data in order to respond to funder questions. The third goal is to help segment its marketing and social media efforts and develop an audience “avatar” to help describe its audience to potential advertisers. Wink shares that while response rates can be frustrating, surveys have helped VOX Femina to understand and respond to the needs of its audience. For example, after using supertitles at one concert, Vox Femina received overwhelming feedback that its audience wants supertitles whenever possible. The chorus now always has supertitles in venues that can accommodate it, and it is exploring ways of including supertitles at venues that are not already equipped for it. Another example is renting risers to improve sightlines in a new venue due to feedback received from an audience survey. But be cautious: it’s not possible to respond to every request. Wink reminds us that we can’t change our programming just because one person wants something different, but overall, surveys help VOX Femina to improve the audience experience. The Pacific Youth Choir, which is based in Portland, Oregon, and annually serves hundreds of students ages 5–18, has recently focused on surveying participant families in order to collect data on participant demographics and tuition assistance, says executive director Andrew Hansen. That focus will change to audience surveys at the organization’s spring concert.

How can your organization increase survey response rates?

Prior to taking the role of executive director with the Pacific Youth Choir, Hansen consulted with dozens of nonprofit clients around the country on projects and initiatives to increase their revenue, supporters, and impact. In his experience, being thoughtful about communicating up front what to expect and how the information will be used will always increase your response rates. “From a hospitality perspective, if you’re going to do it, you want to do it well,” he says. “If you can’t afford to do it well because that’s just not the season your organization is in, that’s ok. I think that folks have greater grace for that than we sometimes think. Doing it in a way that feels relatable, that is a two-way conversation, is going to carry a lot of goodwill.” Accessibility also helps to increase response rates. The Pacific Youth Choir prioritizes digital communication, and while emailing a link is a convenient way to ask people to fill out a survey when connecting with audiences online, it finds QR codes to be ideal for in-person events. In a post-pandemic world, people are more familiar with how to use QR codes, and it’s easy to add them to programs or display them on signs.“ The delivery of a survey has to be filtered through that care that we want to show folks,” Hansen says. “That begins to start shaping every other decision you make with data collection, because ultimately you’re asking a person to give you their time, so how are we doing that in a way that makes them feel seen and known?” VOX Femina’s surveys are also online only and are administered after every concert via an emailed link to a Google Form. After a recent concert, 16% of the total audience responded to the survey, a response rate that Wink says is typical for them. Cantate’s surveys are administered in person; they are included in hard copy in the programs for each concert (each includes a QR code for an online response option). Schaefer finds that a majority of Cantate’s audience is comfortable with pen and paper, though younger audience members are more likely to access the digital survey. “If there are takeaways, it’s simplicity, and ease and speed of doing the survey,” she says. “It’s really how you deliver it, and the best way to explain it is in person." Schaefer gives a “spiel” about surveys at each Cantate concert that she says is extremely effective. “It’s really about a few certain points,” she says. “Our organization is here to serve the community. So, the first reason we do this is to understand how much of the community we’re serving and who we’re serving. It also helps us understand who in our community we are not serving, so we can develop a plan to serve them better. “I say this and suddenly you start to hear pens clicking,” she says. She also tells the audience that funders ask for demographic data, and that filling out a survey is an easy way to help the organization without even opening your wallet. Encouraging audiences to fill out surveys has become a major part of her curtain speech. If for some reason she doesn’t give this speech, response rates drop. But incorporating these points into a heartfelt request has been so effective that Cantate has seen its response rate go from 20% to a consistent 80%. At one small event for a consulting client, Schaefer even saw a 100% response rate. “People say they found the appeal to do the survey really moving,” she says with a laugh. “They want to hear those reasons.” Cantate has seen response rates continue to increase this year, as audience members become more aware of the importance of the surveys and feel like they are participating in the wellbeing of the organization. “I have found so much connection with our audiences when I circulate with baskets for collecting surveys,” Schaefer says. “Spectators ask questions about upcoming programming, have genuine interest in the survey results, talk to me about making donations […] It turns out that once the habit is formed in a sincere and heartfelt way, people really get on board.”

How many questions should you ask, and how do you formulate them?

“The way you phrase a question should always aid in the data you’re trying to collect and how you report on it,” Wink advises. For the most part, that means phrasing questions so that answers will be quantifiable (i.e., 90% of audience members reported that __________). But don’t lose sight of the value of open-ended questions. Having a combination of quantifiable and open-ended questions will get you both quotes and data that you can report on. VOX Femina’s survey has 10–14 questions and sub-questions. Wink suggests that it can be interesting to fill out surveys of other organizations or businesses, as the experience can help you think about your own survey questions. Cantate’s survey has 11 questions and fits on one side of one sheet of paper. “Brevity is really important,” Schaefer says. Cantate’s surveys are also fully anonymous, utilizing Google Forms for online surveys and not collecting email addresses. Schaefer stresses to audiences that their responses are not traceable, so they should feel free to be open and honest. For organizations creating their own surveys, she emphasizes the importance of always having a “prefer not to answer” option. This helps respondents to feel that their privacy is respected and can encourage them to fill out the survey even if there are one or two questions they would rather not answer. When formulating questions, it’s important to work backward to determine what to ask on your survey. “What is it we need to know?” Hansen asks. “Then we work backward from it to think about which questions will get us there in the most efficient way possible.” He also encourages organizations to be thoughtful about the mixture of qualitative and quantitative questions. “Quantitative questions that have defined answers will always be fruitful and [are] easier for folks to fill out,” he says. “A lot of times we’ll put open-ended questions on our surveys, and they produce valuable insights, but it’s so difficult to compare that information.” Despite the challenges, qualitative questions are worth including as they can provide beneficial feedback that wouldn’t be captured by quantitative questions. Hansen also recommends including fewer questions on a survey that is intended for people with less commitment or less of an established relationship with your organization, and more questions for people with a deeper relationship and commitment. “Someone you’ve been in relationship with for a long time is going to be more willing to give you their time,” he says. “They’re also going to feel that you value their perspective that much more when you’re asking a handful of questions. The exact number is relative.”

Incentivizing survey responses—the nuts and bolts

Incentives can be a useful tool to help encourage survey responses, but Hansen cautions not to overspend on an incentive. The ideal incentive, he says, is one that is perceived by potential respondents to be valuable but costs the organization less than the perceived value. An example might be entering respondents into a drawing for free or reduced tuition for a youth choir if you’re asking parents to fill out a survey. VOX Femina enters respondents into a drawing for free tickets and a $50 Amazon gift card. According to Wink, the gift card gives extra incentive for people to fill out the survey, and since it’s just one gift card, it doesn’t break the budget. Cantate has not incentivized its surveys yet, but Schaefer says that she is considering entering respondents into a drawing for a gift card or a dinner. The incentive will be something outside the organization, because Cantate particularly hopes to encourage people outside its normal audience—for example, someone who comes to a program of Brazilian music but doesn’t necessarily have a strong interest in choral music in general— and it wants the incentive to be valuable to that type of audience member. However, Schaefer points out that incentivizing is challenging if you also want to keep your survey anonymous. This is part of the reason she hasn’t incentivized the surveys yet; she feels that anonymity is important to solicit honest feedback, and collecting respondents’ data to enter them into a drawing may make patrons feel that their responses are traceable. Even if they are not being traced, it might be enough to discourage honesty.

Caitlin Patton is the Executive Director of the National Music Festival and a Standards for Excellent Licensed Consultant. She lives on a small farm in Maryland and is a choral singer, violinist, and former board chair of the Chester River Chorale