Where is the future of survey research?

The technical context in which survey methodology lives is evolving quickly. Where will surveys fit into this context in the future?

In the past, surveys were a valuable and unique source of data. As society became more focused on customization and understanding certain populations, surveys became in invaluable tool for data collection. But at this point, we are inundated with data. The amount of content generatd every minute on the net is staggering. In an environment where content is so omnipresent, what role can surveys play? How can we justify our particular brand of data?

Survey methodology has become structured around a set of ethics and practices, including representativeness and respect for the respondents. Without that structure, the most vocal win out, and the resulting picture is not representative.

I recently had the pleasure of reading a bit of Don Dillman’s rewrite of ‘The Tailored Design Method,’ which is the defining classic reference in survey research. The book includes research based strategies for designing and targeting a survey population with the highest possible degree of success. It is often referred to as a bible of sorts to survey practitioners. This time around, I began to think about why the suggestions in the book are so successful. I believe that the success of Dillman’s suggestions has to do with his working title- it is a tailored method, designed around the respondents. And, indeed, the book borrows some principles of respondent or user centered design.

So where does text analysis fit into that? In a context where content is increasingly targeted, and people expect content to be increasingly targeted, surveys as well need to be targeted and tailored for respondents. In an era where the cost benefit equation of survey response is increasingly weighted against us, where potential responses are inundated with web content and web surveys, any misstep can be enough to drive respondents away and even to cause a potential viral backlash. It has never been more important to get it right.

And yet we are pressured not to get it right but to get it fast. So the traditional methods of focus groups and cognitive interviews are increasingly too costly and too timely to use. But their role is an important one. They act to add a layer of quality control to the surveys we produce. They keep us from thinking that because we are the survey experts we are also the respondent experts and the response experts.

A good example of this is Shaeffer’s key idea of native terms. I have a brief story to illustrate this. Our building daycare is about to close, and I have been involved in many discussions about the impact of its closure as well as the planning and musing about the upcoming final farewell reunion celebration. The other day I ran into one of the kids’ grandparents, someone who I have frequentky discussed the daycare with. She asked me if I was planning to go to Audrey’s party. I told her I didn’t know anything about it and wasn’t planning to go. I said this, because I associate the terms she used with retirement celebrations. I assumed that she was talking about a party specifically in honor of the director, not the reunion for all of the kids.

It’s easy as a survey developer to assume that if you ask something that is near enough to what you want to know, the respondent can extrapolate the rest. But that belies the actual way in which we communicate. When it comes to communication, we are inundated with verbal information, and we only really consciously take the gloss of it. That’s what linguistics is all about; unpacking the aspects of communication that communicators don’t naturally focus on, don’t notice, and even can’t notice in the process of communication.

So where am I going with all of this?

One of the most frequent aspects of text analysis is a word frequency count. This is often used as a psuedo content analysis, but that is a very problematic extrapolation, for reasons that I’ve mentioned before in this blog and in my paper on ths topic. However, word frequency counts are a good way of extrapolating native terms from which to do targeting.

Text analytics aren’t representative, but they have the ability of being more representative than many of the other predevelopment methods that we employ. Their best use may not be so much as a supplement to our data analysis as a precursor to our data collection.

However, that data has more uses than this.

It CAN be used as a supplement to data analysis as well, but not by going broad. By going DEEP. Taking segments and applying discourse analytic methodology can be a way of supplementing the numbers and figures collected with surveys with a deeper understanding of the dynamics of the respondent population.

Using this perspective, linguistics has a role both in the development of tailored questionnaires and in the in depth analysis of the respondenses and respondents.