A brave new vision of the future of social science

I’ve been typing and organizing my notes from yesterday’s dc-aapor event on the past, present and future of survey research (which I still plan to share soon, after a little grooming). The process has been a meditative one.

I’ve been thinking about how I would characterize these same phases- the past, present and future… and then I had a vision of sorts on the way home today that I’d like to share. I’m going to take a minute to be a little post apocalyptic and let the future build itself. You can think of it as a daydream or thought experiment…

The past, I would characterize as the grand discovery of surveys as a tool for data collection; the honing and evolution of that tool in conjunction with its meticulous scientific development and the changing landscape around it; and the growth to dominance and proliferation of the method. The past was an era of measurement, of the total survey error model, of social Science.

The present I would characterize as a rapid coming together, or a perfect storm that is swirling data and ideas and disciplines of study and professions together in a grand sweeping wind. I see the survey folks trudging through the wind, waiting for the storm to pass, feet firmly anchored to solid ground.

The future is essentially the past, turned on its head. The pieces of the past are present, but mixed together and redistributed. Instead of examining the ways in which questions elicit usable data, we look at the data first and develop the questions from patterns in the data. In this era, data is everywhere, of various quality, character and genesis, and the skill is in the sense making.

This future is one of data driven analytic strategies, where research teams intrinsically need to be composed of a spectrum of different, specialized skills.

The kings of this future will be the experts in natural language processing, those with the skill of finding and using patterns in language. All language is patterned. Our job will be to find those patterns and then to discover their social meaning.

The computer scientists and coders will write the code to extract relevant subsets of data, and describe and learn patterns in the data. The natural language processing folks will hone the patterns by grammar and usage. The netnographers will describe and interpret the patterns, the data visualizers will make visual or interactive sense of the patterns, the sociologists will discover constructions of relative social groupings as they emerge and use those patterns. The discourse analysts will look across wider patterns of language and context dependency. The statisticians will make formulas to replicate, describe and evaluate the patterns, and models to predict future behaviors. Data science will be a crucial science built on the foundations of traditional and nontraditional academic disciplines.

How many people does it take to screw in this lightbulb? It depends on the skills of the people or person on the ladder.

Where do surveys fit in to this scheme? To be honest, I’m not sure. The success of surveys seems to rest in part on the failure of faster, cheaper methods with a great deal more inherent error.

This is not the only vision possible, but it’s a vision I saw while commuting home at the end of a damned long week… it’s a vision where naturalistic data is valued and experimentation is an extension of research, where diversity is a natural assumption of the model and not a superimposed dynamic, where the data itself and the patterns within it determine what is possible from it. It’s a vision where traditional academics fit only precariously; a future that could just as easily be ruled out by the constraints of the past as it could be adopted unintentionally, where meaning makers rush to be the rigs in the newest gold rush and theory is as desperately pursued as water sources in a drought.

Question Writing is an Art

As a survey researcher, I like to participate in surveys with enough regularity to keep current on any trends in methodology. As a web designer, an aspect of successful design is a seamlessness with the visitor’s expectations. So if the survey design realm has moved toward submit buttons on the upper right hand corner of individual pages, your idea (no matter how clever) to put a submit button on the upper left can result in a disconnect on the part of the user that will effect their behavior on the page. In fact, the survey design world has evolved quite a bit in the last few years, and it is easy to design something that reflects poorly on the quality of your research endeavor. But these design concerns are less of an issue than they have been, because most researchers are using templates.

Yet there is still value in keeping current.

And sometimes we encounter questions that lend themselves to an explanation of the importance of question writing. These questions are a gift for a field that is so difficult to describe in terms of knowledge and skills!

Here is a question I encountered today (I won’t reveal the source):

How often do you purchase potato chips when you eat out at any quick service and fast food restaurants?

2x a week or more
1x a week
1x every 2-3 weeks
1x a month
1x every 2-3 months
Less than 1x every 3 months
Never

This is a prime example of a double barreled question, and it is also an especially difficult question to answer. In my care, I rarely eat at quick service restaurants, especially sandwich places, like this one, that offer potato chips. When I do eat at them, I am tempted to order chips. About half the time I will give in to the temptation with a bag of sunchips, which I’m pretty sure are not made of potato.

In bigger firms that have more time to work through, this information would come out in the process of a cognitive interview or think aloud during the pretesting phase. Many firms, however, have staunchly resisted these important steps in the surveying process, because of their time and expense. It is important to note that the time and expense involved with trying to make usable answers out of poorly written questions can be immense.

I have spent some time thinking about alternatives to cognitive testing, because I have some close experience with places that do not use this method. I suspect that this is a good place for text analytics, because of the power of reaching people quickly and potentially cheaply (depending on your embedded TA processes). Although oftentimes we are nervous about web analytics because of their representativeness, the bar for representativeness is significantly lower in the pretesting stage than in the analysis phase.

But, no matter what pretesting model you choose, it is important to look closely at the questions that you are asking. Are you asking a single question, or would these questions be better separated out into a series?

How often do you eat at quick service sandwich restaurants?

When you eat at quick service restaurants, do you order [potato] chips?

What kind of [potato] chips do you order?

The lesson of all of this is that question writing is important, and the questions we write in surveys will determine the kind of survey responses we receive and the usability of our answers.

To go big, first think small

We use language all of the time. Because of this, we are all experts in language use. As native speakers of a language, we are experts in the intricacies of that language.

Why, then, do people study linguistics? Aren’t we all linguists?

Absolutely not.

We are experts in *using* language, but we are not experts in the methods we employ. Believe it or not, much of the process of speaking and hearing is not conscious. If it was, we would be sensorally overwhelmed with the sheer volume of words around us. Instead, listening comprehension involves a process of merging what we expect to hear with what we gauge to be the most important elements of what we do hear. The process of speaking involves merging our estimates of what the people we communicate with know and expect to hear with our understanding of the social expectations surrounding our words and our relationships and distilling these sources into a workable expression. The hearer will reconstruct elements of this process using cues that are sometimes conscious and sometimes not.

We often think of language as simple and mechanistic, but it is not simple at all. As conversational analysts, our job is to study conversation that we have access to in an attempt to reconstruct the elements that constituted the interaction. Even small chunks of conversation encode quite a bit of information.

The process of conversation analysis is very much contrary to our sense of language as regular language users. This makes the process of explaining our research to people outside our field difficult. It is difficult to justify the research, and it is difficult to explain why such small pieces of data can be so useful, when most other fields of research rely on greater volumes of data.

In fact, a greater volume of data can be more harmful than helpful in conversation analysis. Conversation is heavily dependent on its context; on the people conversing, their relationship, their expectations, their experiences that day, the things on their mind, what they expect from each other and the situation, their understanding of language and expectations, and more. The same sentence can have greatly different meanings once those factors are taken into account.

At a time when there is so much talk of the glory of big data, it is especially important to keep in mind the contributions of small data. These contributions are the ones that jeopardize the utility and promise of big data, and if these contributions can be captured in creative ways, they will be the true promise of the field.

Not what language users expect to see, but rather what we use every day, more or less consciously.

When Code Is Hot

Excellent article on TechCrunch by Jon Evans, “When Code is Hot”

http://techcrunch.com/2012/04/07/when-code-is-hot/

Excerpt:

“That first cited piece above begins with “Parlez-vous Python?”, a cutesy bit that’s also a pet peeve. Non-coders tend to think of different programming languages as, well, different languages. I’ve long maintained that while programming itself — “computational thinking”, as the professor put it — is indeed very like a language, “programming languages” are mere dialects; some crude and terse, some expressive and eloquent, but all broadly used to convey the same concepts in much the same way.

 
Like other languages, though, or like music, it’s best learned by the young. I am skeptical of the notion that many people who start learning to code in their 30s or even 20s will ever really grok the fundamental abstract notions of software architecture and design.

 
Stross quotes Michael Littman of Rutgers: “Computational thinking should have been covered in middle school, and it isn’t, so we in the C.S. department must offer the equivalent of a remedial course.” Similarly, the Guardian recently ran an excellent series of articles on why all children should be taught how to code. (One interesting if depressing side note there: the older the students, the more likely it is that girls will be peer-pressured out of the technical arena.)”

Research and Little League

I recently had a revelation about research methodology.

In my Intercultural Communication class, a presenter showed a picture of a moment in a baseball game. The conversation that followed was about baseball and about Little League. It missed the point.

Look around you. You are flooded with visual data. Open your ears. You are flooded with auditory data. Open your senses. What are you touching? Do you smell anything? The world is full of sensory data, so much data, in fact, that we could never take it all in.

This is where attention come in. Focus. Foreground. We quickly filter out sounds to focus on, points in the visual field that are the most meaningful at any given moment. In this way, we are efficient and capable. But we are not researchers.

To conduct research is to focus on a moment in time, an interaction, a photograph, etc. and look more deeply at it. Research begins with careful observation. Research includes deconstructing an element into its constituent pieces and thinking carefully about those pieces.

What does a linguist do? A linguist takes the time to look at language and unpack it to reconstitute its context, creation and motivation. Linguistics is asking ‘what is happening?’ ‘what tools are being used’ and ‘what is being accomplished?’ Linguistics is taking the time to look more closely at the elements of the picture and not restrict oneself to the natural foreground.

Laypeople talk about the content of language. People talk about the boy in the picture who is jumping for joy. Researchers look at the trajectory of the eyes in the crowd to see where people are focusing their attention. They notice the fence between the audience and the players and the way people interact with it. They notice the baseball on the ground. They notice the sunshine and the clothing that the people are wearing. They can uncover the deeper story of what was happening in that moment, instead of surmising about the apparent focus.

These are the skills we are learning.

Renewing my Vows?

Anne Steen came to our Proseminar class last week to discuss personality types, the strong skills inventory, and the search for an ideal job. She used an excellent analogy to describe the difference between doing work that we are and are not suited for. She said that doing work that we are well suited for is like using our dominant hand to write. We do it without really thinking about it. Writing with our other hand is possible, but it is more difficult. I have always felt that way about my work. I remember when I first began to work in research. I had done other kinds of work before, some of which I had enjoyed more than others. But work in research was almost meditative, because it came so naturally to me.

Because of that feeling, it has always been hard for me to see outside my field. I don’t feel as though I have a great understanding of what other people do in their work on a day to day basis, and I can’t easily envision myself doing any different kind of work. The results of the strong skills inventory and MBTI reinforced my contentedness with my current position. Many of my daily work activities were listed as work activities that I would particularly enjoy. I feel like, professionally speaking, I married my first love and am too content in my marriage to imagine being with anyone else. In fact, I work at an organization where people tend to stay for their entire careers, so the metaphor is particularly apt. In a culture where people don’t consider looking elsewhere or changing jobs, it feels particularly backhanded to explore other options. In this way, I feel like I have seen my matches on a dating website, and my partner was right on top of the list of potential matches.

Because of this, I feel like I need to go deeper in order to ‘think outside the box.’

One aspect of my current job as a survey methodologist that I really love is designing surveys. For paper surveys, I really love obsessing about the color of the survey and the mood it will immediately evoke. I love playing with space. I love the way that each aspect of the space is meaningful and every design element must be consistent, because it will be taken as meaningful. I love the way rearranging the questions changes the context and meaning of the questions themselves. I love reading the research about what people like to encounter first, and what you need to squirrel away last, and what questions increase respondent’s confidence in the survey. I love the crisp, professional look of a well-designed survey, and I love the way each aspect of the design is based on research.

I also love designing web surveys. Many of my other job activities are based on programming, so working with CSS really resonates with me. I started out working with html and then discovered CSS, so the flexibility of CSS was a real revelation. It turned out that I could set text styles, just like on a gui editor, design all aspects of their appearance, and tweak them with ease. Tweaking margin spaces until I feel content looking at them appeals to my artistic sensibilities. I really love survey design for its technical challenges and artistic rewards. When I am in a design phase, I even tend to read design books, like Norman Cook’s The Design of Everyday Things or Steve Krug’s web design classic Don’t Make Me Think, in my spare time.

Although I really love most aspects of my job, survey design is the one aspect that I have volunteered to specialize in. My small department’s workflow style has been best described as “jack of all trades, master of none,” but I have approached my directors and told them that we could use an expert in survey design, because it’s such an important part of collecting quality answers. Since then our tech guru has created a survey template, so I no longer get to do much survey design. I am interested in finding new ways to apply those skills, or ways to bring those skills back into my work life.

However, I worry about this kind of a step, because I’m so interested in research methodology.

I suppose that having too many interests is not a bad problem to have!