Category Archives: commentary

Can they just get along? Situated Cognition and Survey Response

Finally, I’m going to take a moment to talk about Norbert Schwarz’s JPSM Distinguished Lecture on March 30! I’ve attended a few events and had a few experiences lately that I’m eager to blog about, but sometimes life has plans for us that don’t involve blogging. Today, I would say, is no different, except that I woke up thinking about this lecture!

Ok, enough about me, more about Schwartz.

I should start by saying that I am a longtime fan of Schwartz. In Fall 2009, I had just discovered the MLC program and finished what was a whirlwind application process, and I was first trying to wrap my head around the field of sociolinguistics and its intersection with my career in survey methodology. I had attended a presentation of an ethnography of communication pilot study to the McDonough School of Business, and, to my great shock, I came across a survey methodology paper that spoke of the Logic of Conversation and the role of Gricean maxims in survey responses. This fantastic piece is the work of Norbert Schwarz, and I’ve kept it nearby ever since. In it, Schwartz addresses the conversational expectations of survey respondents and shows how they respond not only to the question at hand, but also to these expectations.

It’s common in every survey to look at some of the responses and wonder how in the world they could have come about. I addressed this in an earlier blog post, where one researcher had gone as far as to call respondents stupid. Oftentimes we think of respondents “getting it right” or “getting it wrong.” But there is a larger phenomena underlying what appear to be strange responses, and it’s something that we experience when we attempt to respond to surveys.

We write survey questions with a mechanistic expectation, that if we ask a question, we will hear back the answer to that question, but we neglect to consider the fact that communication is not mechanistic. Of course, we are not necessarily aware of this. We’re aware of misunderstandings, but we’re not often aware of the tiny sphere of focus and interpretive frames that we apply to every utterance we here and utter. This is no fault of our own. This is a survival tool. We simply cannot process all of the information that we’re constantly inundated with.

In survey research, we’re aware that small differences in question format can influence responses. We’re aware that changing a scale will change the numeric range of the responses. We see that changing labels on a scalar question changes the results. We’re aware that sometimes answers appear to be absolute contradictions and seem to us to be impossible. These are especially large challenges for us, and they are the purview of linguistics.

Schwartz, however, is not a linguist. He is a cognitice scientist. And his lecture was not about the linguistic basis behind apparently wonky response phenomenon. Instead, he spoke about situated cognition.

Situated cognition makes a lot of intuitive sense. It is a proven psychological phenomena that shows that we don’t hold attitudes, beliefs and responses at a certain location in our mind, rather we recreate them each time. Instead we create or recreate them each time. This process allows for much more of an influence from “what’s on our mind,” making situational or contextual factors much more important, and decreasing the reliability, or repeatability, of survey responses. This is not a hard egg for someone (me) with a background in cognitive science and sociolinguistics to swallow, but the effect on the audience was remarkable. How does someone from a field that thrives on the mechanistic nature of responses take the suggestion that what they’re measuring is not a distinctly measurable entity so much as a complicated, potentially unreliable act of nature?

One of the discussants used a couple that he was not very fond of as an example of a stable opinion. I believe that this example lends itself well to further exploration. If he had just met the couple, and he had had a negative experience with them, his evaluation of his opinion toward the couple would depend on the degree of negativity of the experience, his predisposition to give or not give them the benefit of the doubt, and his degree of concern about expressing a negative opinion to the interviewer or survey researchers. After this point, these factors will be increasingly influenced by his further experiences with the people and the degree of negativity, positivity or neutrality of the experiences, and the recency and saliency of the experiences. Essentially, his response would reflect a complicated underlying equation and be the output of situated cognition.

But what is a survey researcher supposed to do with this information?

It would be easy at this point to throw the baby out with the bathwater and cast doubt on the whole survey and response process. But that’s not necessary, and that’s not the point.

The point is that each method of analysis has its own unique set of strengths and weaknesses. It is important to know the strengths and weaknesses of your methods in order to better understand what exactly you are finding and what your findings mean. And it also behooves us to supplement across methodologies. A reliable survey response is a strong finding, but it can mask underlying factors that can be accessed through other methodologies. As Pew demonstrated in their Kony 2012 report, mixing methodologies can lead to a more clear, nuanced narrative than any single method could yield.

It would be easy to dismiss Schwartz’s reporting, or to dismiss survey methodology. But dismissing either would be foolish, rash and unnecessary. Instead, let’s build on both. A wider foundation can build a better house, but the best house will need to take down some old walls and rethink its floorplan.

Don’t fear Big Data

I really enjoyed this RTI blog post about embracing big data:

https://blogs.rti.org/surveypost/2012/03/22/why-you-should-not-fear-but-embrace-the-age-of-big-data/

I suspect that oftentimes fear of big data is motivated by a concern that new, less tested, still evolving methods will replace the time tested methods that we have grown to have so much faith in. I sincerely believe that the foundation that we have is a strong one, and the knowledge we have developed through those processes should be embraced, especially the quality controls. But SUPPLEMENTING an analysis through a measured combination of data sources can lead to a more complete picture.

This week I spent some time analyzing Pew’s report on the Kony 2012 video. I believe that this report is an excellent example of what researchers are capable of when they look outside the artificial divisions of research group (this was a collaborative effort) and research methodology. Seven days after the release of the video, Pew was able to reconstruct a comprehensive narrative of the video’s dissemination, using traditional survey methods, sentiment analytic snapshots over time, and a careful breakdown of the media coverage of influential parties.

 

Dana Boyd also has an interesting analysis of the Kony phenomena on her Apophenia blog:

http://www.zephoria.org/thoughts/

Zen as a Research Ethic

I have a Zen calendar on my desk for 2012. It has such gems as: “Although the world is full of suffering, it is also full of the overcoming of it” (Helen Keller)

The more I look at the calendar, the more it relates to everything I think about.

I read “To see is to forget the name of the thing one sees,” (Paul Valery) and I think of the Charles Goodwin paper I cited in a recent post about Professional Vision. He talks about ways of seeing as kind of coding structures, inculturation, or ways of foregrounding certain parts of what we see. Truly, being able to see deeper than that requires shedding that inculturation and observing more closely. As researchers, we often become so deeply incultured into our way of thinking, that we lose sight of our research goals. As survey researchers, we can easily fall into the pattern of first asking “who should we survey?” and “what should we ask?” before taking more time to consider whether a survey is even an appropriate methodology for the specific topic of focus. Of course, not this action based on praxis is not limited to survey researchers. Far from it! Every person, every field, every community of practice, every language has a way of thinking. And often instead of seeing or observing, we quickly begin to navigate our networks of inculturation.

These two are similarly meaningful in my interpretation:

“Zen is not to confuse spirituality with thinking about God while one is peeling potatoes. Zen is just to peel the potatoes.” (Alan Watts)

“If all beings are Buddha, why all this striving?” (Dogen)

These are a reminder to boil things down to what they simply are and not try to describe them as what you want them to be. In survey research, this comes up often in the process of reporting research results. If I know that I intended to measure something about Project Based Learning or STEM education, it is easily for me to begin to frame my findings by my intentions. But that is not true to my findings or my methodology, and it doesn’t make for good research. I can’t say that 10% of my respondents were using project based learning methods in the classroom if I asked about the number of group activities they conducted. I must simply say that 10% were using group activities (daily/monthly/occasionally- whatever the answer choices were)

In this way, my Zen calendar not only provides something to think about in a larger sense, but it keeps my research anchored.

Why Social Media couldn’t predict Super Tuesday

This piece is a nice reminder not only, as the authors conclude, that sentiment analysis has not fully matured, but also that sentiment analysis and social media analysis probably don’t accomplish what they think they are accomplishing:

 

http://www.retargeter.com/political-advertising/why-social-media-couldnt-predict-super-tuesday

Language as a way of seeing

A conversation in my Intercultural Communication class on Tuesday inspired me to think about language in a different way.

There is a fantastic paper by Charles Goodwin called Professional Vision that shows how people train each other into ways of seeing. The example he uses is an anthropological investigation of soil, and he shows how the new anthropologist is trained by the more experienced anthropologist to notice and describe gradations in the color or quality of the soil. These gradations have a specific meaning within their investigation, and being able to describe them is an important tool of the study.

Language provides two important functions that I want to focus on today. The one we think of most often is the tool function. Language gives us a way to communicate what we see. But it is much less often that we think about language as a way of seeing.

I’ve talked about our constant stimulation before on this blog. We are constantly exposed to more sensory data than we could ever process. Language provides a lens through which to see the world in front of us. Languages have inherent sets of coding structures to apply to the world.

There is another function of language that corpus and computational linguistics really highlights, and that’s habituation. We may have the tools within a language to describe something in any number of ways, but very few of these ways are actually used with any frequency. We learn in corpus linguistics that language is never random. In many ways, in fact, language follows a zipfian distribution, with a high concentration of high frequency words or expressions, and each after the most frequent exponentially less frequent.The language itself may be large, but the language we use is much smaller.

This is an important element to consider in the conceptualization of translation. One person who reviewed the original Russian translation of our Global Survey of Physicists complained that if they did not know English, they would not have understood the Russian. The more I learn about language, the more I understand how that could be possible. Language is not just grammatical, it is habitual, and it holds a way of seeing the world.

I don’t believe this is an idea whose time has come

I ran across this presentation of a Living Social style performance review system:

http://i.bnet.com/video/itunes/12z0315_GeorgeHu_Salesforce_sd_DL.m4v

I’m pretty sure that it is not yet April 1st, so I’m going to assume this program is for real. If it catches on, it represents a giant step toward the gamification of performance reviews.

We are currently in the process of adapting to a new performance review system at my workplace, and the process is immensely complicated. There is definitely some crowd sourcing potential in these systems and room for improvement. But I think that what’s really missing from these systems is a basis for internal motivation. I’m a firm believer that people will naturally try to excel. If people are not trying to excel, it should be taken as a sign that something is disrupting the system. Given that basic assumption, the best kind of performance review system is one that naturally allows, rewards and encourages people to strive to do their best, one that respects and recognizes peoples’ natural talents and interests. Gamification of performance just makes a disrespectful joke out of this. I have to wonder if this is a consequence of an increasingly pervasive artificial system of punishments and rewards. Instead of people finding and experiencing the natural consequences of their actions, are people instead looking for external rewards, such as badges on social media interfaces?

While I truly believe in embracing technology, I don’t believe that this is a step forward. In my ideal work situation, we would all be encouraged to do our best, credited for the work that we do, and given work that we’re most enthusiastic about.

If you think that is naive, please prove me wrong!

Calling Respondents Stupid

From a Politico article titled “How much do voters know?” by Alexander
Burns:

“The first lesson you learn as a pollster is that people are stupid,”
said Tom Jensen of Public Policy Polling, a Democratic polling firm. “I
tell a client trying to make sense of numbers on a poll that are
inherently contradictory that at least once a week.”

Full article at:

http://www.politico.com/news/stories/0312/73947.html

Message to Tom Jenson:
It’s not so much that people are stupid as it is that Tom Jenson is ignorant of the cognitive underpinnings of response methodology and filling in his gaps with unnecessary condescension. Truly, all survey researchers deal with these contradictions. We might be able to garner more consistency in our response sets if we could instead survey computers or robots, but, alas, that’s not why we conduct surveys.

Qualitative and Quantitative methods revisited

At the GURT 2012 conference last week, another graduate student whose work was primarily quantitative asked me about the issue of data quality in qualitative research. She asked how, if your results are not repeated over a large group of people, do you know that your results are reliable or representative? In the process of answering her, I realized that in some ways quantitative and qualitative research are ideologically opposed. Whereas quantitative research is validated in part by the reliability of a result across a variety of people and contexts, the people and contexts for which the result is being quantified are not the focus but the rationale. However, in qualitative research, the context and the people are the focus. Qualitative research is more about putting a microscope up to an element in the data and closely observing how it works in the context of its surroundings. The research questions are starkly different, but they do clearly complement each other.

In the world of text analytics, the analytic focus is largely quantitative, which in some ways counteracts the very nature of the data. The unbalanced nature of the data collection then calls the quality of the quantitative analysis into question, and the analysis is not used in a traditional way to recreate the microfocus, so the advantages of the qualitative microfocus are also diminished.

We spoke a lot at GURT about the paucity of theory in the field of text analytics. The technical ability applied to the problems is quite strong. There are quite a few very talented programmers who are hard at work at conquering some of the technical issues inherent in text analysis, some on the computer science end of things, some at the computational linguistic end of things and some fortunate enough to work from both ends with knowledge from both fields. But it is not enough to be able to solve problems and answer questions, we have to know which questions to ask.

It has been relatively easy to blame the fast growing set of consumers for their immediate hunger for the data analysis. In order to satisfy this hunger, companies like Open Amplify double their output monthly and are working as hard and fast as they can to keep up with the demand. But the demand generally comes with the same level of linguistic knowledge that most laypeople have. We, as language users, are constantly inundated with language, and we only consciously process a very small proportion of it. So we don’t instinctively ask the questions that our data is really best suited to answer. The text analytic world is responding to questions of “what are people talking about?” with word frequencies, comparative word frequencies and sentiment analyses that are tied to those word frequencies. But we don’t use language that way! If I ask you about your phone, you’re likely to respond about its features of its usefulness of its price, or how well you’ve adapted to it. If I ask 100 people about their phones, how much good will it do me to aggregate across responses? There is a good deal of work that needs to be done in terms of finding intertextual references to phones (e.g. “springy keypad” or “data plan”) and assigning a negative value to “limited calling plan” and a positive to “limited call interference.” When I asked a coworker how our advisory committee meeting was going while I was at the conference, she answered “delicious.” We communicate by keying on shared knowledge, and as we communicate we build senses of particular topics that are related specifically to our conversation. If my coworker had answered with a comment about the potato salad, and I had played off of that, would we be talking about potato salad in any equivalent way to the way we might at a summer picnic? I would argue that as we joke on, we be talking in fact about something quite different than the potato salad itself. In fact, we would likely use the potato salad as a stand-in for the meeting that we were really discussing. Should that conversation be used by market researchers in a potato salad corporation?

In fact, the topics that we discuss are quite variable. The specific meanings that elements take on within a conversation is best understood in the context of that conversation as a part of a qualitative analysis than an aggregated quantitative analysis.

The big essential questions that we need to grapple with, as a field, at this point are questions like:

‘what kind of questions is this kind of data best suited to answer?’

‘how can our knowledge of linguistics and discourse be transferred into quantifiable questions that could feed the field of text analysis’

‘what kinds of questions can we ask of textual data that will reframe the way that people think about the usefulness of textual data?’

‘how can we best harness this fast growing mass of textual data in the most useful, reliable ways?’

I would argue that these are questions that discourse analysts are best suited to answer, but in order to ask them, they/we must be able to leave our qualitative bunkers and open our minds to the complementary potential for quantitative analysis. I would also argue that a popularized appreciation for the value of discourse analysis would also lend some legitimacy to a field that is largely unknown.

On the way to work this morning, I listened to an interview with Naomi Wolf. She spoke in part of the chutzpah of presenting academic knowledge in a widely accessible format. Academic perspective, she argued, is too often maintained in academic circles, far away from the general population who could really use and appreciate it. Georgetown professor Deborah Tannen made some important steps in the popularization of sociolinguistics. I believe that what I am suggesting is a quantitative extension of the popularization. People could not have imagined that a book about something as obscure as conversation analysis could be interesting or so widely applicable to their own lives. There was no rushing the doors from the general population of people desperate for these books. Hers was not a case of giving people what they wanted. It was a case of giving people something that would be widely useful. And people embraced Dr Tannen’s books as such.

Let us now use the luxury of time that the academic sector has but the commercial sector certainly does not to do what we do best: theorize! A new, great plane awaits. Let us head west!

Keeping Research Responsible

Reliability and validity are the two most important principals in social science research. They are the measures that maintain the integrity, quality and ultimate usefulness of our work.

Reliability refers to the replicability of findings, which is a crucial element in any scientific process. Repeatability under varying conditions helps to establish the consistency and boundaries of a phenomena. Oftentimes, we like to compare ourselves to scientists and emphasize our take on the scientific method. I would argue that some of the most important lessons we can learn from science include the basic doubt (inherent in statistics as well, thanks to the null hypothesis) that assures that no one study can make knowledge so much as suggest knowledge that can lead to further testing and potential verification. Research is inherently tied to its underlying questions, and a well conducted study based on one question could easily lead to very different findings than another, even slightly different, research question. Reliability cannot be valued highly enough in social science research.

Validity refers to the value of our findings. What do these findings mean? What do they refer to? In survey research, understanding the validity of a finding often entails retracing what question was asked to whom under what circumstances and anchoring any conclusions to those basic truths instead of extrapolating to a wider principle that we would like to have observed.

In text analytics, their are two more anchors that guide research; precision and recall. Recall is the measure of how many of the correct instances of a phenomena you were able to isolate in your programming, and precision refers to the percent of the instances that you collected that were indeed instances of your target phenomena. Text analysis is a dance of queries, toggling between collecting correct matches and dropping incorrect matches. It is within the context of this dance that language seems most staggering. Here we see how little people say what they mean or mean what they say, how much context matters, how often people refer to subjects by proxy, how dependent on ongoing conversation new elements are, …

As users of language, we are constantly inundated with words. We cope with this by only focusing on selective elements. We focus on metamessages, not mechanics. It is easy to assume, from our perspective, that language is straightforward. Indeed, if it was straightforward, text analytics would be easy and misunderstandings would not be so rampant! All of the linguistic data that we are flooded with would be harnessed regularly, and society would reflect that data better through ubiquitously increased customization.

But the reality of language stands in stark contrast to what we assume it to be, and that reality is the lifeblood of linguistics.

Rethinking the Future of Survey Methodology; Finding a Place for Linguistics

Where is the future of survey research?

The technical context in which survey methodology lives is evolving quickly. Where will surveys fit into this context in the future?

In the past, surveys were a valuable and unique source of data. As society became more focused on customization and understanding certain populations, surveys became in invaluable tool for data collection. But at this point, we are inundated with data. The amount of content generatd every minute on the net is staggering. In an environment where content is so omnipresent, what role can surveys play? How can we justify our particular brand of data?

Survey methodology has become structured around a set of ethics and practices, including representativeness and respect for the respondents. Without that structure, the most vocal win out, and the resulting picture is not representative.

I recently had the pleasure of reading a bit of Don Dillman’s rewrite of ‘The Tailored Design Method,’ which is the defining classic reference in survey research. The book includes research based strategies for designing and targeting a survey population with the highest possible degree of success. It is often referred to as a bible of sorts to survey practitioners. This time around, I began to think about why the suggestions in the book are so successful. I believe that the success of Dillman’s suggestions has to do with his working title- it is a tailored method, designed around the respondents. And, indeed, the book borrows some principles of respondent or user centered design.

So where does text analysis fit into that? In a context where content is increasingly targeted, and people expect content to be increasingly targeted, surveys as well need to be targeted and tailored for respondents. In an era where the cost benefit equation of survey response is increasingly weighted against us, where potential responses are inundated with web content and web surveys, any misstep can be enough to drive respondents away and even to cause a potential viral backlash. It has never been more important to get it right.

And yet we are pressured not to get it right but to get it fast. So the traditional methods of focus groups and cognitive interviews are increasingly too costly and too timely to use. But their role is an important one. They act to add a layer of quality control to the surveys we produce. They keep us from thinking that because we are the survey experts we are also the respondent experts and the response experts.

A good example of this is Shaeffer’s key idea of native terms. I have a brief story to illustrate this. Our building daycare is about to close, and I have been involved in many discussions about the impact of its closure as well as the planning and musing about the upcoming final farewell reunion celebration. The other day I ran into one of the kids’ grandparents, someone who I have frequentky discussed the daycare with. She asked me if I was planning to go to Audrey’s party. I told her I didn’t know anything about it and wasn’t planning to go. I said this, because I associate the terms she used with retirement celebrations. I assumed that she was talking about a party specifically in honor of the director, not the reunion for all of the kids.

It’s easy as a survey developer to assume that if you ask something that is near enough to what you want to know, the respondent can extrapolate the rest. But that belies the actual way in which we communicate. When it comes to communication, we are inundated with verbal information, and we only really consciously take the gloss of it. That’s what linguistics is all about; unpacking the aspects of communication that communicators don’t naturally focus on, don’t notice, and even can’t notice in the process of communication.

So where am I going with all of this?

One of the most frequent aspects of text analysis is a word frequency count. This is often used as a psuedo content analysis, but that is a very problematic extrapolation, for reasons that I’ve mentioned before in this blog and in my paper on ths topic. However, word frequency counts are a good way of extrapolating native terms from which to do targeting.

Text analytics aren’t representative, but they have the ability of being more representative than many of the other predevelopment methods that we employ. Their best use may not be so much as a supplement to our data analysis as a precursor to our data collection.

However, that data has more uses than this.

It CAN be used as a supplement to data analysis as well, but not by going broad. By going DEEP. Taking segments and applying discourse analytic methodology can be a way of supplementing the numbers and figures collected with surveys with a deeper understanding of the dynamics of the respondent population.

Using this perspective, linguistics has a role both in the development of tailored questionnaires and in the in depth analysis of the respondenses and respondents.