Telling your story in qualitative and quantitative terms

This is a really interesting piece by Robery Krulwich:

http://www.npr.org/blogs/krulwich/2012/03/21/149095154/mirror-mirror-on-the-wall-do-the-data-tell-it-all?sc=fb&cc=fp

In it, he asks a writer and a physicist to talk about their lives. They describe their lives in vastly different terms. The writers tells his story in narrative form, and the physicist describes his in terms of data. In fact, the physicist has amassed a sizable collection of what he calls ‘personal data.’ He provides an interesting an unusual perspective and begs the question ‘what does this data mean?’ or, as Dr Castillo Ayometzi would ask, “what is being constituted here?”

Enjoy!

Language as a way of seeing

A conversation in my Intercultural Communication class on Tuesday inspired me to think about language in a different way.

There is a fantastic paper by Charles Goodwin called Professional Vision that shows how people train each other into ways of seeing. The example he uses is an anthropological investigation of soil, and he shows how the new anthropologist is trained by the more experienced anthropologist to notice and describe gradations in the color or quality of the soil. These gradations have a specific meaning within their investigation, and being able to describe them is an important tool of the study.

Language provides two important functions that I want to focus on today. The one we think of most often is the tool function. Language gives us a way to communicate what we see. But it is much less often that we think about language as a way of seeing.

I’ve talked about our constant stimulation before on this blog. We are constantly exposed to more sensory data than we could ever process. Language provides a lens through which to see the world in front of us. Languages have inherent sets of coding structures to apply to the world.

There is another function of language that corpus and computational linguistics really highlights, and that’s habituation. We may have the tools within a language to describe something in any number of ways, but very few of these ways are actually used with any frequency. We learn in corpus linguistics that language is never random. In many ways, in fact, language follows a zipfian distribution, with a high concentration of high frequency words or expressions, and each after the most frequent exponentially less frequent.The language itself may be large, but the language we use is much smaller.

This is an important element to consider in the conceptualization of translation. One person who reviewed the original Russian translation of our Global Survey of Physicists complained that if they did not know English, they would not have understood the Russian. The more I learn about language, the more I understand how that could be possible. Language is not just grammatical, it is habitual, and it holds a way of seeing the world.

I don’t believe this is an idea whose time has come

I ran across this presentation of a Living Social style performance review system:

http://i.bnet.com/video/itunes/12z0315_GeorgeHu_Salesforce_sd_DL.m4v

I’m pretty sure that it is not yet April 1st, so I’m going to assume this program is for real. If it catches on, it represents a giant step toward the gamification of performance reviews.

We are currently in the process of adapting to a new performance review system at my workplace, and the process is immensely complicated. There is definitely some crowd sourcing potential in these systems and room for improvement. But I think that what’s really missing from these systems is a basis for internal motivation. I’m a firm believer that people will naturally try to excel. If people are not trying to excel, it should be taken as a sign that something is disrupting the system. Given that basic assumption, the best kind of performance review system is one that naturally allows, rewards and encourages people to strive to do their best, one that respects and recognizes peoples’ natural talents and interests. Gamification of performance just makes a disrespectful joke out of this. I have to wonder if this is a consequence of an increasingly pervasive artificial system of punishments and rewards. Instead of people finding and experiencing the natural consequences of their actions, are people instead looking for external rewards, such as badges on social media interfaces?

While I truly believe in embracing technology, I don’t believe that this is a step forward. In my ideal work situation, we would all be encouraged to do our best, credited for the work that we do, and given work that we’re most enthusiastic about.

If you think that is naive, please prove me wrong!

Amazing Presentation on Infographics

I had the privilege this week of attending a webinar by Matthew Erickson of the New York Times about innovative graphic presentations of data. There were some truly amazing interactive displays included in this presentation, and the presenter had a lot of very insightful suggestions for rethinking data presentation:

 

http://www.ericson.net/files/aapor-shared.pdf

 

He spoke about the role of good interactive presentation in situating data, providing context, developing layers, and telling a story. A lot of times, the distribution of the data, and it’s relationship with data from other sources, is its most interesting layer. In an innovative presentation of data, we must balance the expectations of the audience, who become interactants with the data and must be able to manipulate it easily, with a complementary layer of expertise or context.

For example, data about Manny Rivera’s pitching style could best be understood by the placement of the ball at the hitter’s decision making point. In a graphic about Rivera’s success, the reporters were able to show how radically different pitches were virtually indistinguishable at the crucial decision making point for the hitter.

He referred to infographics as the “gamification of news.”

 

To connect his presentation to this ongoing discussion of text analytics, check out the way he displayed word frequencies:

http://www.ericson.net/presentations/aiga-for-web.pdf

Interestingly, it is still problematic, but it is super cool looking…

 

And, speaking of infographics, check out this awesome one that Pew debuted today:

http://features.pewforum.org/religious-migration/map.php#/Destination/UnitedStates/

Calling Respondents Stupid

From a Politico article titled “How much do voters know?” by Alexander
Burns:

“The first lesson you learn as a pollster is that people are stupid,”
said Tom Jensen of Public Policy Polling, a Democratic polling firm. “I
tell a client trying to make sense of numbers on a poll that are
inherently contradictory that at least once a week.”

Full article at:

http://www.politico.com/news/stories/0312/73947.html

Message to Tom Jenson:
It’s not so much that people are stupid as it is that Tom Jenson is ignorant of the cognitive underpinnings of response methodology and filling in his gaps with unnecessary condescension. Truly, all survey researchers deal with these contradictions. We might be able to garner more consistency in our response sets if we could instead survey computers or robots, but, alas, that’s not why we conduct surveys.

Qualitative and Quantitative methods revisited

At the GURT 2012 conference last week, another graduate student whose work was primarily quantitative asked me about the issue of data quality in qualitative research. She asked how, if your results are not repeated over a large group of people, do you know that your results are reliable or representative? In the process of answering her, I realized that in some ways quantitative and qualitative research are ideologically opposed. Whereas quantitative research is validated in part by the reliability of a result across a variety of people and contexts, the people and contexts for which the result is being quantified are not the focus but the rationale. However, in qualitative research, the context and the people are the focus. Qualitative research is more about putting a microscope up to an element in the data and closely observing how it works in the context of its surroundings. The research questions are starkly different, but they do clearly complement each other.

In the world of text analytics, the analytic focus is largely quantitative, which in some ways counteracts the very nature of the data. The unbalanced nature of the data collection then calls the quality of the quantitative analysis into question, and the analysis is not used in a traditional way to recreate the microfocus, so the advantages of the qualitative microfocus are also diminished.

We spoke a lot at GURT about the paucity of theory in the field of text analytics. The technical ability applied to the problems is quite strong. There are quite a few very talented programmers who are hard at work at conquering some of the technical issues inherent in text analysis, some on the computer science end of things, some at the computational linguistic end of things and some fortunate enough to work from both ends with knowledge from both fields. But it is not enough to be able to solve problems and answer questions, we have to know which questions to ask.

It has been relatively easy to blame the fast growing set of consumers for their immediate hunger for the data analysis. In order to satisfy this hunger, companies like Open Amplify double their output monthly and are working as hard and fast as they can to keep up with the demand. But the demand generally comes with the same level of linguistic knowledge that most laypeople have. We, as language users, are constantly inundated with language, and we only consciously process a very small proportion of it. So we don’t instinctively ask the questions that our data is really best suited to answer. The text analytic world is responding to questions of “what are people talking about?” with word frequencies, comparative word frequencies and sentiment analyses that are tied to those word frequencies. But we don’t use language that way! If I ask you about your phone, you’re likely to respond about its features of its usefulness of its price, or how well you’ve adapted to it. If I ask 100 people about their phones, how much good will it do me to aggregate across responses? There is a good deal of work that needs to be done in terms of finding intertextual references to phones (e.g. “springy keypad” or “data plan”) and assigning a negative value to “limited calling plan” and a positive to “limited call interference.” When I asked a coworker how our advisory committee meeting was going while I was at the conference, she answered “delicious.” We communicate by keying on shared knowledge, and as we communicate we build senses of particular topics that are related specifically to our conversation. If my coworker had answered with a comment about the potato salad, and I had played off of that, would we be talking about potato salad in any equivalent way to the way we might at a summer picnic? I would argue that as we joke on, we be talking in fact about something quite different than the potato salad itself. In fact, we would likely use the potato salad as a stand-in for the meeting that we were really discussing. Should that conversation be used by market researchers in a potato salad corporation?

In fact, the topics that we discuss are quite variable. The specific meanings that elements take on within a conversation is best understood in the context of that conversation as a part of a qualitative analysis than an aggregated quantitative analysis.

The big essential questions that we need to grapple with, as a field, at this point are questions like:

‘what kind of questions is this kind of data best suited to answer?’

‘how can our knowledge of linguistics and discourse be transferred into quantifiable questions that could feed the field of text analysis’

‘what kinds of questions can we ask of textual data that will reframe the way that people think about the usefulness of textual data?’

‘how can we best harness this fast growing mass of textual data in the most useful, reliable ways?’

I would argue that these are questions that discourse analysts are best suited to answer, but in order to ask them, they/we must be able to leave our qualitative bunkers and open our minds to the complementary potential for quantitative analysis. I would also argue that a popularized appreciation for the value of discourse analysis would also lend some legitimacy to a field that is largely unknown.

On the way to work this morning, I listened to an interview with Naomi Wolf. She spoke in part of the chutzpah of presenting academic knowledge in a widely accessible format. Academic perspective, she argued, is too often maintained in academic circles, far away from the general population who could really use and appreciate it. Georgetown professor Deborah Tannen made some important steps in the popularization of sociolinguistics. I believe that what I am suggesting is a quantitative extension of the popularization. People could not have imagined that a book about something as obscure as conversation analysis could be interesting or so widely applicable to their own lives. There was no rushing the doors from the general population of people desperate for these books. Hers was not a case of giving people what they wanted. It was a case of giving people something that would be widely useful. And people embraced Dr Tannen’s books as such.

Let us now use the luxury of time that the academic sector has but the commercial sector certainly does not to do what we do best: theorize! A new, great plane awaits. Let us head west!

Keeping Research Responsible

Reliability and validity are the two most important principals in social science research. They are the measures that maintain the integrity, quality and ultimate usefulness of our work.

Reliability refers to the replicability of findings, which is a crucial element in any scientific process. Repeatability under varying conditions helps to establish the consistency and boundaries of a phenomena. Oftentimes, we like to compare ourselves to scientists and emphasize our take on the scientific method. I would argue that some of the most important lessons we can learn from science include the basic doubt (inherent in statistics as well, thanks to the null hypothesis) that assures that no one study can make knowledge so much as suggest knowledge that can lead to further testing and potential verification. Research is inherently tied to its underlying questions, and a well conducted study based on one question could easily lead to very different findings than another, even slightly different, research question. Reliability cannot be valued highly enough in social science research.

Validity refers to the value of our findings. What do these findings mean? What do they refer to? In survey research, understanding the validity of a finding often entails retracing what question was asked to whom under what circumstances and anchoring any conclusions to those basic truths instead of extrapolating to a wider principle that we would like to have observed.

In text analytics, their are two more anchors that guide research; precision and recall. Recall is the measure of how many of the correct instances of a phenomena you were able to isolate in your programming, and precision refers to the percent of the instances that you collected that were indeed instances of your target phenomena. Text analysis is a dance of queries, toggling between collecting correct matches and dropping incorrect matches. It is within the context of this dance that language seems most staggering. Here we see how little people say what they mean or mean what they say, how much context matters, how often people refer to subjects by proxy, how dependent on ongoing conversation new elements are, …

As users of language, we are constantly inundated with words. We cope with this by only focusing on selective elements. We focus on metamessages, not mechanics. It is easy to assume, from our perspective, that language is straightforward. Indeed, if it was straightforward, text analytics would be easy and misunderstandings would not be so rampant! All of the linguistic data that we are flooded with would be harnessed regularly, and society would reflect that data better through ubiquitously increased customization.

But the reality of language stands in stark contrast to what we assume it to be, and that reality is the lifeblood of linguistics.