Monthly Archives: November 2012

What do all of these polling strategies add up to?

Yesterday was a big first for research methodologists across many disciplines. For some of the newer methods, it was the first election that they could be applied to in real time. For some of the older methods, this election was the first to bring competing methodologies, and not just methodological critiques.

Real time sentiment analysis from sites like this summarized Twitter’s take on the election. This paper sought to predict electoral turnout using google searches. InsideFacebook attempted to use Facebook data to track voting. And those are just a few of a rapid proliferation of data sources, analytic strategies and visualizations.

One could ask, who are the winners? Some (including me) were quick to declare a victory for the well honed craft of traditional pollsters, who showed that they were able to repeat their studies with little noise, and that their results were predictive of a wider real world phenomena. Some could call a victory for the emerging field of Data Science. Obama’s Chief Data Scientist is already beginning to be recognized. Comparisons of analytic strategies will spring up all over the place in the coming weeks. The election provided a rare opportunity where so many strategies and so many people were working in one topical area. The comparisons will tell us a lot about where we are in the data horse race.

In fact, most of these methods were successful predictors in spite of their complicated underpinnings. The google searches took into account searches for variations of “vote,” which worked as a kind of reliable predictor but belied the complicated web of naturalistic search terms (which I alluded to in an earlier post about the natural development of hashtags, as explained by Rami Khater of Al Jezeera’s The Stream, a social network generated newscast). I was a real-world example of this methodological complication. Before I went to vote, I googled “sample ballot.” Similar intent, but I wouldn’t have been caught in the analyst’s net.

If you look deeper at the Sentiment Analysis tools that allow you to view the specific tweets that comprise their categorizations, you will quickly see that, although the overall trends were in fact predictive of the election results, the data coding was messy, because language is messy.

And the victorious predictive ability of traditional polling methods belies the complicated nature of interviewing as a data collection technique. Survey methodologists work hard to standardize research interviews in order to maximize the reliability of the interviews. Sometimes these interviews are standardized to the point of recording. Sometimes the interviews are so scripted that interviewers are not allowed to clarify questions, only to repeat them. Critiques of this kind of standardization are common in survey methodology, most notably from Nora Cate Schaeffer, who has raised many important considerations within the survey methodology community while still strongly supporting the importance of interviewing as a methodological tool. My reading assignment for my ethnography class this week is a chapter by Charles Briggs from 1986 (Briggs – Learning how to ask) that proves that many of the new methodological critiques are in fact old methodological critiques. But the critiques are rarely heeded, because they are difficult to apply.

I am currently working on a project that demonstrates some of the problems with standardizing interviews. I am revising a script we used to call a representative sample of U.S. high schools. The script was last used four years ago in a highly successful effort that led to an admirable 98% response rate. But to my surprise, when I went to pull up the old script I found instead a system of scripts. What was an online and phone survey had spawned fax and e-mail versions. What was intended to be a survey of principals now had a set of potential respondents from the schools, each with their own strengths and weaknesses. Answers to common questions from school staff were loosely scripted on an addendum to the original script. A set of tips for phonecallers included points such as “make sure to catch the name of the person who transfers you, so that you can specifically say that Ms X from the office suggested I talk to you” and “If you get transferred to the teacher, make sure you are not talking to the whole class over the loudspeaker.”

Heidi Hamilton, chair of the Georgetown Linguistics department, often refers to conversation as “climbing a tree that climbs back.” In fact, we often talk about meaning as mutually constituted between all of the participants in a conversation. The conversation itself cannot be taken outside of the context in which it lives. The many documents I found from the phonecallers show just how relevant these observations can be in an applied research environment.

The big question that arises from all of this is one of a practical strategy. In particular, I had to figure out how to best address the interview campaign that we had actually run when preparing to rerun the campaign we had intended to run. My solution was to integrate the feedback from the phonecallers and loosen up the script. But I suspect that this tactic will work differently with different phonecallers. I’ve certainly worked with a variety of phonecallers, from those that preferred a script to those that preferred to talk off the cuff. Which makes the best phonecaller? Neither. Both. The ideal phonecaller works with the situation that is presented to them nimbly and professionally while collecting complete and relevant data from the most reliable source. As much of the time as possible.

At this point, I’ve come pretty far afield of my original point, which is that all of these competing predictive strategies have complicated underpinnings.

And what of that?

I believe that the best research is conscious of its strengths and weaknesses and not afraid to work with other strategies in order to generate the most comprehensive picture. As we see comparisons and horse races develop between analytic strategies, I think the best analyses we’ll see will be the ones that fit the results of each of the strategies together, simultaneously developing a fuller breakdown of the election and a fuller picture of our new research environment.

Education from the Bottom Up?

Last night I attended a talk by Shirley Bryce Heath about her new book, Words at Work and Play, moderated by Anne Harper Charity Hudley and Frederick Erickson. Dr Bryce Heath has been following a group of 300 families for 30 years, and in her talk she addressed many of the changes she’d seen in the kids in the time she’d been observing them. She made one particularly interesting point. She mentioned that the world of assessment, and, in fact much of the adult world hasn’t kept up with the kids’ evolution. The assessments that we subject kids to are traditional, reflecting traditional values and sources. She went as far as to say that we don’t know how to see, appreciate or notice these changes, and she pointed out that much of new styles of learning came outside of the school environment.

This part of her talk reminded me of an excellent blog post I read yesterday about unschooling. Unschooling is the process of learning outside of a structured environment. It goes further than homeschooling, which can involve structured curricula. It is curricularly agnostic and focused on the learning styles, interests, and natural motivation of the students. I mentioned the blog post to Terrence Wiley, president of the Center for Applied Linguistics, and he emphasized the underlying idealism of unschooling. It rests on the basic belief that everyone is naturally academically motivated and interested and will naturally embrace learning, in their own way, given the freedom to do it. Unschooling is, as some would say my “spirit animal.” I don’t have the time or the resources to do it with my own kids, and I’m not sure I would even if I were fully able to do it. I have no idea how it could be instituted in any kind of egalitarian or larger scale way. But I still love the idea, in all it’s unpracticality. (Dr Wiley gave me a few reading assignments, explaining that ‘everything old in education is new again’)

Then today I read a blog about the potential of using Wikipedia as a textbook. This idea is very striking, not just because Wikipedia was mostly accurate, freely available, covered the vast majority of the material in this professor’s traditional textbooks, and has an app that will help anyone interested create a custom textbook, but because it actually addresses what kids do anyway! Just this past weekend, my daughter was writing a book report, and I kept complaining that she chose to use Wikipedia to look up the spelling of a character’s name rather than walk upstairs and grab the book. Kids use Wikipedia often and for all kinds of things, and it is often more common for parents and educators to forbid or dismiss this practice than to jump right in with them. I suggest that the blogger not only use Wikipedia, but use the text as a way to show what is or is not accurate, how to tell, and where to find other credible, collaborative sources when it doubt. What an amazing opportunity!

So here’s the question that all of this has been leading to: Given that the world around is is rapidly changing and that our kids are more adept at staying abreast of these changes than they are, could it be time to turn the old expert-novice/ teacher-student paradigm on its head, at least in part? Maybe we need to find ways to let some knowledge come from the bottom up. Maybe we need to let them be the experts. Maybe we need to, at least in part, rethink our role in the educating process?

Frederick Erickson made an excellent point about teaching “You have to learn your students in order to teach them.” He talked about spending the first few days in a class gathering the expertise of the students, and using that knowledge when creating assignments or assigning groups. (I believe Dr Hudley mentioned that she did this, too. Or maybe he supplied the quote, and she supplied the example?)

All of this makes me wonder what the potential is for respecting the knowledge and expertise of the students, and working from there. What does bottom-up or student-led education look like? How can it be integrated into the learning process in order to make it more responsive, adaptive and modern?

Of course, this is as much a dream for a wider society as unschooling is for my own family. To a large extent, practicality shoots it all in the foot with the starting gun. But a girl can dream, no?