Category Archives: education

Reporting on the AAPOR 69th national conference in Anaheim #aapor

Last week AAPOR held it’s 69th annual conference in sunny (and hot) Anaheim California.

Palm Trees in the conference center area

My biggest takeaway from this year’s conference is that AAPOR is a very healthy organization. AAPOR attendees were genuinely happy to be at the conference, enthusiastic about AAPOR and excited about the conference material. Many participants consider AAPOR their intellectual and professional home base and really relished the opportunity to be around kindred spirits (often socially awkward professionals who are genuinely excited about our niche). All of the presentations I saw firsthand or heard about were solid and dense, and the presenters were excited about their work and their findings. Membership, conference attendance, journal and conference submissions and volunteer participation are all quite strong.

 

At this point in time, the field of survey research is encountering a set of challenges. Nonresponse is a growing challenge, and other forms of data and analysis are increasingly en vogue. I was really excited to see that AAPOR members are greeting these challenges and others head on. For this particular write-up, I will focus on these two challenges. I hope that others will address some of the other main conference themes and add their notes and resources to those I’ve gathered below.

 

As survey nonresponse becomes more of a challenge, survey researchers are moving from traditional measures of response quality (e.g. response rates) to newer measures (e.g. nonresponse bias). Researchers are increasingly anchoring their discussions about survey quality within the Total Survey Error framework, which offers a contextual basis for understanding the problem more deeply. Instead of focusing on an across the board rise in response rates, researchers are strategizing their resources with the goal of reducing response bias. This includes understanding response propensity (who is likely not to respond to the survey? Who is most likely to drop out of a panel study? What are some of the barriers to survey participation?), looking for substantive measures that correlate with response propensity (e.g. Are small, rural private schools less likely to respond to a school survey? Are substance users less likely to respond to a survey about substance abuse?), and continuous monitoring of paradata during the collection period (e.g. developing differential strategies by disposition code, focusing the most successful interviewers on the most reluctant cases, or concentrating collection strategies where they are expected to be most effective). This area of strategizing emerged in AAPOR circles a few years ago with discussions of nonresponse propensity modeling, a process which is surely much more accessible than it sounds, but it has really evolved into a practical and useful tool that can help any size research shop increase survey quality and lower costs.

 

Another big takeaway for me was the volume of discussions and presentations that spoke to the fast-emerging world of data science and big data. Many people spoke of the importance of our voice in the realm of data science, particularly with our professional focus on understanding and mitigating errors in the research process. A few practitioners applied error frameworks to analyses of organic data, and some talks were based on analyses of organic data. This year AAPOR also sponsored a research hack to investigate the potential for Instagram as a research tool for Feed the Hungry. These discussions, presentations and activities made it clear that AAPOR will continue to have a strong voice in the changing research environment, and the task force reports and initiatives from both the membership and education committees reinforced AAPOR’s ability to be right on top of the many changes afoot. I’m eager to see AAPOR’s changing role take shape.

“If you had asked social scientists even 20 years ago what powers they dreamed of acquiring, they might have cited the capacity to track the behaviors, purchases, movements, interactions, and thoughts of whole cities of people, in real time.” – N.A.  Christakis. 24 June 2011. New York Times, via Craig Hill (RTI)

 

AAPOR a very strong, well-loved organization and it is building a very strong future from a very solid foundation.

 

 

2014-05-16 15.38.17

 

MORE DETAILED NOTES:

This conference is huge, so I could not possibly cover all of it on my own, so I will try to share my notes as well as the notes and resources I can collect from other attendees. If you have any materials to share, please send them to me! The more information I am able to collect here, the better a resource it will be for people interested in the AAPOR or the conference-

 

Patrick Ruffini assembled the tweets from the conference into this storify

 

Annie, the blogger behind LoveStats, had quite a few posts from the conference. I sat on a panel with Annie on the role of blogs in public opinion research (organized by Joe Murphy for the 68th annual AAPOR conference), and Annie blew me away by live-blogging the event from the stage! Clearly, she is the fastest blogger in the West and the East! Her posts from Anaheim included:

Your Significance Test Proves Nothing

Do panel companies manage their panels?

Gender bias among AAPOR presenters

What I hate about you AAPOR

How to correct scale distribution errors

What I like about you AAPOR

I poo poo on your significance tests

When is survey burden the fault of the responders?

How many survey contacts is enough?

 

My full notes are available here (please excuse any formatting irregularities). Unfortunately, they are not as extensive as I would have liked, because wifi and power were in short supply. I also wish I had settled into a better seat and covered some of the talks in greater detail, including Don Dillman’s talk, which was a real highlights of the conference!

I believe Rob Santos’ professional address will be available for viewing or listening soon, if it is not already available. He is a very eloquent speaker, and he made some really great points, so this will be well worth your time.

 

Professional Identity: Who am I? And who are you?

Last night I acted as a mentor at the annual Career Exploration Expo sponsored by my graduate program. Many of the students had questions about developing a professional identity. This makes sense, of course, because graduate school is an important time for discovering and developing a professional identity.

People enter our program (and many others) With a wide variety of backgrounds and interests. They choose from a variety of classes that fit their interests and goals. And then they try to map their experience onto job categories. But boxes are difficult to climb into and out of, and students soon discover that none of the boxes is a perfect fit.

I experienced this myself. I entered the program with an extensive and unquestioned background in survey research. Early in my college years (while I was studying and working in neuropsychology) I began to manage a clinical dataset in SPSS. Working with patients and patient files was very interesting, but to my surprise working with data using statistical software felt right to me much in the way that Ethiopian meals include injera and Japanese meals include rice (IC 2006 (1997) Ohnuki Tierney Emiko). I was actually teased by my friends about my love of data! This affinity served me well, and I enjoyed working with a variety of data sets while moving across fields and statistical programming languages.

But my graduate program blew my mind. I felt like I had spent my life underwater and then discovered the sky and continents. I discovered many new kinds of data and analytic strategies, all of which were challenging and rewarding. These discoveries inspired me to start this blog and have inspired me to attend a wide variety of events and read some very interesting work that I never would have discovered on my own. Hopefully followers of this blog have enjoyed this journey as much as I have!

As a recent graduate, I sometimes feel torn between worlds. I still work as a survey researcher, but I’m inspired by research methods that are beyond the scope of my regular work. Another recent graduate of our program who is involved in market research framed her strategy in a way that really resonated with me: “I give my customers what they want and something else, and they grow to appreciate the ‘something else.'” That sums up my current strategy. I do the survey management and analysis that is expected of me in a timely, high quality way. But I am also using my newly acquired knowledge to incorporate text analysis into our data cleaning process in order to streamline it, increasing both the speed and the quality of the process and making it better equipped to handle the data from future surveys. I do the traditional quantitative analyses, but I supplement them  with analyses of the open ended responses that use more flexible text analytic strategies. These analyses spark more quantitative analyses and make for much better (richer, more readable and more inspired) reports.

Our goal as professionals should be to find a professional identity that best capitalizes on  our unique knowledge, skills and abilities. There is only one professional identity that does all of that, and it is the one you have already chosen and continue to choose every day. We are faced with countless choices about what classes to take, what to read, what to attend, what to become involved in, and what to prioritize, and we make countless assessments about each. Was it worthwhile? Did I enjoy it? Would I do it again? Each of these choices constitutes your own unique professional self, a self which you are continually manufacturing. You are composed of your past, your present, and your future, and your future will undoubtedly be a continuation of your past and present. The best career coach you have is inside of you.

Now your professional identity is much more uniquely or narrowly focused that the generic titles and fields that you see in the professional marketplace. Keep in mind that each job listing that you see represents a set of needs that a particular organization has. Is this a set of needs that you are ready to fill? Is this a set of needs that you would like to fill? You are the only one who knows the answers to these questions.

Because it turns out that you are your best career coach, and you have been all along.

In praise of getting things wrong and working toward better

“An expert is a man who has made all the mistakes which can be made in a very narrow field” -Niels Bohr

I’ve been reading “In the Plex,” a book about the history of Google by Steven Levy. I highly recommend this book, because as I read it I am increasingly aware of the ways in which Google’s constant presence invisibly shapes our daily lives. Levy makes a point in the book of attributing some of Google’s constant evolution to its obsession with failure. In search terms, isolating failures is relatively easy- if people soon return to the search page, reframe their query, or continue down through lower ranked results their search was a relative failure. Failures are identified and isolated by Google and then obsessed over until the PageRank algorithm can be appropriately tweaked in a way that passes rigorous testing protocols.

In this way, Google is similar to an increasing number of failure- focused initiatives, including some of the engineering based models that have been applied to healthcare and more. These voices are increasingly the source of innovations that are continually shaping and reshaping our future. But the rhetoric of failure and success of its evangelizers can be hard for us to wrap our heads around, as people who naturally fear, avoid and focus on failure in a negative way.

Over the weekend, while I was practicing Yoga I told one of my kids my favorite part of the practice (note: not a good time for chatting). I love that Yoga is a process. One day you will be able to do something that you may or may not be able to do the next day, and vice versa. My practice involves quite a bit of balancing on one foot, and there are days when that balance feels effortless and days when that balance feels impossible. But the effortless days only come because I continue to practice despite the disappointments of my wobblier days. Yoga instructors sometimes talk about the power of intentions and working in ways that align with our intentions. One of my kids pointed out that the wobbly days, as I call them, are exactly the reason why she hates Yoga. She’s believes that she’s no good at it, and because of her assessment she will avoid it. You can probably guess that this conversation is far from over between us.

We see attitudes like these affecting people (including ourselves) every day. Some people theorize that the lower representation of women in STEM (Science, Technology, Engineering and Math) fields is due to a larger proportion of women than men who doubt their abilities or judge their abilities more harshly. We hear about graduate students who experience what is sometimes called the ‘imposter syndrome.’ I remember some students in my graduate classes who chose not to participate in class for fear they would sound stupid. I’ve heard of medical practitioners who were so worried that they would make another mistake that they were afraid to practice. As a writer, I know that the power of self doubt can cause writers block, but I also know how much easier it is to edit or rewrite.

I would encourage all of you to embrace your failures, your mistakes, your shortcomings, your missteps and your errors and see them as part of a process and not an endpoint. These stumbling points are the key points of growth- the key moments for us to learn and to redirect our actions to better suit our intentions. To err is human, but to learn from our missteps is surely something greater.

The holiday season and the post-degree process

I haven’t blogged much this month.

Yesterday I didn’t blog because I was wandering around my neighborhood with my kids and my winter boots, looking for the ultimate sledding hill that wasn’t just mud.

 

2013-12-10 13.03.42

I did get this cool shot of the snow melting:

2013-12-10 14.11.15

This past weekend I didn’t blog because I was trying to get some holiday shopping done. Holiday shopping is a mess of contradictions. The music and festive spirit are relaxing and wonderful, but the task at hand is to reckon with our wants. My goal lately has been exactly the opposite of this- to appreciate what abundance already constitutes my life and not to focus on needing or wanting more. This is an important part of my post degree process.

At the time of my graduation I joked with a close friend about expecting life to be like a musical, with the people around me singing and dancing my accomplishments and those of my classmates. For those of you still in school I hate to break your bubble, but there will likely not be a musical in your honor, as deserving as you may be.

Graduation is not the end of your work as a student. Your work will extend beyond graduation and in to what I’ve come to think of as an extra semester of undetermined length. This is the time when we try to make all of our hard work pay off. We learn that the world will not recognize our accomplishments unless we learn how to be our own best advocates, and we learn how difficult it is to advocate for ourselves across lines of field and areas of practice.

This process involves a reckoning between the idealized notions of our future that motivated us through late-nighters and all-nighters and the realities of our post degree lives. It also involves a surprisingly long transition from the frenetic pace of student life to the appreciative pace of real life. We learn how to channel the energy that is no longer focused on school work but free to roam across a wide range of interests and responsibilities. We forge a new set of priorities. We realize that we will not find jobs that are as well rounded as we are. We see that we are not frozen in place after our degrees but will continue the lifelong process of learning. We begin to find peace in the knowledge that what we have is enough. We may not have the yacht and the private plane, but we have food on our plates and in our bellies. And what we have is enough.

Graduates (especially in today’s employment market) have to wrestle with the responsibilities of post-degree life, the lack of recognition of their academic accomplishments, and the transition [back?] into the swing of daily work life. We have to transition from the big dreams of school life to the small rewards of real life. For me this process involves a compacting. It involves tightening the family budget and saving for bigger goals. It involves family challenges to see how long we can go between trips to the grocery store and the fun set of culinary challenges that rise from the emptier cupboards (Have you seen those cooking shows where the contestants are challenged to invent a meal based on a small number of random ingredients?). It involves decluttering my house to get rid of extra stuff, appreciate what we have and lessen our responsibilities (less stuff to clean!), and it involves spoiling my family with the time and attention I couldn’t give them before.

This all seems to directly contradict the goals of holiday shopping. I wandered through aisle after aisle of stuff that I couldn’t imagine needing or wanting, thinking of needs and wants as a kind of black hole where needing and wanting can simply lead to more needing and wanting. I’m not sure how my holiday shopping process will shake out this year, but I do know that my happiness and the happiness of those I love can’t be found on any store shelves.

For you students, recent graduates and professional researchers and other readers, I wish you all the peace and gratitude of the season. May the new year bless you with continued curiosity. May we never stop learning and growing. May the process and daily rituals of our lives be reward enough. We can’t anticipate the challenges 2014 will bring, but let us be grateful that we have the tools that we will need to greet them with.

And most of all, I want to thank those of you who read my blog posts. Thank you for your time and attention and for encouraging me to continue to explore. I hope to reward you soon with a rundown of some particularly great events I’ve attended lately!

Great readings that might shake you to your academic core? I’m compiling a list

In the spirit of research readings that might shake you to your academic core, I’m compiling a list. Please reply to this thread with any suggestions you have to add. They can be anything from short blog posts (microblog?) to research articles to books. What’s on your ‘must read’ list?

Here are a couple of mine to kick us off:

 

Charles Goodwin’s Professional Vision paper

I don’t think I’ve referred to any paper as much as this one. It’s about the way our professional training shapes the way we see the things around us. Shortly after reading this paper I was in the gym thinking about commonalities between the weight stacks and survey scales. I expect myself to be a certain relative strength, and when that doesn’t correlate with the place where I need to place my pin I’m a little thrown off.

It also has a deep analysis of the Rodney King verdict.

 

Revitalizing Chinatown Into a Heterotopia by Jia Lou

This article is based on a geosemiotic analysis of DC’s Chinatown. It is one of the articles that helped me to see that data really can come in all forms

 

After method: Mess in Social Science Research by John Law

This is the book that inspired this list. It also inspired this blog post.

 

Data science can be pretty badass, but…

Every so often I’m reminded of the power of data science. Today I attended a talk entitled ‘Spatiotemporal Crime Prediction Using GPS & Time-tagged Tweets” by Matt Gerber of the UVA PTL. The talk was a UMD CLIP event (great events! Go if you can!).

Gerber began by introducing a few of the PTL projects, which include:

  • Developing automatic detection methods for extremist recruitment in the Dark Net
  • Turning medical knowledge from large bodies of unstructured texts into medical decision support models
  • Many other cool initiatives

He then introduced the research at hand: developing predictive models for criminal activity. The control model in this case use police report data from a given period of time to map incidents onto a map of Chicago using latitude and longitude. He then superimposed a grid on the map and collapsed incidents down into a binary presence vs absence model. Each square in the grid would either have one or more crimes (1) or not have any crimes (-1). This was his training data. He built a binary classifier and then used logistic regression to compute probabilities and layered a kernel density estimator on top. He used this control model to compare with a model built from unstructured text. The unstructured text consisted of GPS tagged Twitter data (roughly 3% of tweets) from the Chicago area. He drew the same grid using longitude and latitude coordinates and tossed all of the tweets from each “neighborhood” (during the same one month training window) into the boxes. Then, using essentially a one box=one document for a document based classifier, he subjected each document to topic modeling (using LDA & MALLET). He focused on crime related words and topics to build models to compare against the control models. He found that the predictive value of both models was similar when compared against actual crime reports from days within the subsequent month.

This is a basic model. The layering can be further refined and better understood (there was some discussion about the word “turnup,” for example). Many more interesting layers can be built into it in order to improve its predictive power, including more geographic features, population densities, some temporal modeling to accommodate the periodic nature of some crimes (e.g. most robberies happen during the work week, while people are away from their homes), a better accommodation for different types of crime, and a host of potential demographic and other variables.

I would love to dig deeper into this data to gain a deeper understanding of the conversation underlying the topic models. I imagine there is quite a wealth of deeper information to be gained as well as a deeper understanding of what kind of work the models are doing. It strikes me that each assumption and calculation has a heavy social load attached to it. Each variable and each layer that is built into the model and roots out correlations may be working to reinforce certain stereotypes and anoint them with the power of massive data. Some questions need to be asked. Who has access to the internet? What type of access? How are they using the internet? Are there substantive differences between tweets with and without geotagging? What varieties of language are the tweeters using? Do classifiers take into account language variation? Are the researchers simply building a big data model around the old “bad neighborhood” notions?

Data is powerful, and the predictive power of data is fascinating. Calculations like these raise questions in new ways, remixing old assumptions into new correlations. Let’s not forget to question new methods, put them into their wider sociocultural contexts and delve qualitatively into the data behind the analyses. Data science can be incredibly powerful and interesting, but it needs a qualitative and theoretical perspective to keep it rooted. I hope to see more, deeper interdisciplinary partnerships soon, working together to build powerful, grounded, and really interesting research!

 

Reflections on Social Network Analysis & Social Media Research from #SMSociety13

A dispatch from a quantitative side of social media research!

Here are a few of my reflections from the Social Media & Society conference in Halifax and my Social Network Analysis class.

I should first mention that I was lucky in two ways.

  1. I finished the James Bond movie ‘Skyfall’ as my last Air Canada flight was landing. (Ok, I didn’t have to mention that)
  2. I finished my online course on Social Network Analysis  hours before leaving for a conference that kicked off with an excellent  talk about Networks and diffusion. And then on the second day of the conference I was able to manipulate a network visualization with my hands using a 96 inch touchscreen at the Dalhousie University Social Media Lab  (Great lab, by the way, with some very interesting and freely available tools)

 

This picture doesn't do this screen justice. This is *data heaven*

This picture doesn’t do this screen justice. This is *data heaven*

Social networks are networks built to describe human action in social media environments. They contain nodes (dots), which could represent people, usernames, objects, etc. and edges, lines joining nodes that represent some kind of relationship (friend, follower, contact, or a host of other quantitative measures). The course was a particularly great introduction to Social Network Analysis, because it included a book that was clear and interesting, a set of youtube videos and a website, all of which were built to work together. The instructor (Dr Jen Golbeck, also the author of the book and materials) has a very unique interest in SNA which gives the class an important added dimension. Her focus is on operational definitions and quantitative measures of trust, and because of this we were taught to carefully consider the role of the edges and edge weights in our networks.

Sharad Goel’s plenary at #SMSociety13 was a very different look at networks. He questioned the common notion of viral diffusion online by looking at millions of cases of diffusion. He discovered that very few diffusions actual resemble any kind of viral model. Instead, most diffusion happens on a very small scale. He used Justin Bieber as an example of diffusion. Bieber has the largest number of followers on Twitter, so when it he posts something it has a very wide reach (“the Bieber effect”). However, people don’t share content as often as we imagine. In fact, only a very small proportion of his followers share it, and only a small proportion of their followers share it. Overall, the path is wide and shallow, with less vertical layers than we had previously envisioned.

Goel’s research is an example of Big Data in action. He said that Big Data methods are important when the phenomenon you want to study happens very infrequently (e.g. one in a million), as is the case for actual instances of viral diffusion.

His conclusions were big, and this line of research is very informative and useful for anyone trying to communicate on a large scale.

Sidenote: the term ‘ego network’ came up quite a few times during the conference, but not everyone knew what an ego network is. An ego network begins with a single node and is measured by degrees. A one degree social network looks a bit like an asterisk- it simply shows all of the nodes that are directly connected to the original node. A 1.5 degree network would include the first degree connections as well as the connections between them. A two degree network contains all of the first degree connections to these nodes that were in the one degree network. And so on.

One common research strategy is to compare across ego networks.

My next post will move on from SNA to more qualitative aspects of the conference

Source: https://twitter.com/JeffreyKeefer/status/378921564281921537/photo/1 This was the backdrop for a qualitive panel

Source: https://twitter.com/JeffreyKeefer/status/378921564281921537/photo/1
This was the backdrop for a qualitative panel. It says “Every time you say ‘data driven decision’ a fairy dies.

MOOC’s, Libraries, Online learning and Thirsting for knowledge

Let me begin by telling you a story.

The story began when I was in high school searching for the right college. My mom and I took a road trip the summer after my junior year of college. We took our time and covered quite a bit of ground. I discovered Hot97 in New York and Pepto Bismol in North Carolina. I fell in love with upstate NY. After our return, I began the application and interview process. The most memorable moment came during my interview with a representative from Cornell. She asked if I had any burning questions, and I decided to go ahead and ask her a question that had really been nagging at me: What is the difference between a class at Cornell and a class at a community college? She was shocked and deeply offended. She told me that anyone could get a great education anywhere they could find a library, and obviously I wasn’t right for Cornell.

This exchange has haunted me ever since. I do love to read, sure, but a library alone could never create the magic that a classroom can create. And the most magical classes happen when the students are engaged, interested, attentive, involved, participating, excited and following through with the homework. Part of this magic comes from the teacher. A great teacher can cultivate this kind of environment with ease, but most really struggle when it doesn’t happen organically.

I’m not sure everyone would agree that classrooms can be magical. I may have been spoiled with great classes. I’ve just finished a masters’ program where I loved the classes, loved the reading and loved the assignments, but I’m not sure that every student would approach school with as much relish. I love learning.

Tomorrow I begin an educational experiment. I will start a course in Social Network Analysis from Statistics.com. This is a paid course, and I’ve chosen to be held responsible for my work (you can choose whether or not to submit homework for grading). Next month, the experiment will deepen when I begin my first MOOC. The MOOC is a data analysis class that teaches R. I’m very eager to learn R and to revisit some statistical methods that I haven’t been able to use much. The experiment will not be pure, because three of my coworkers have decided to attend the class as well. We’ll be fortunate enough to experience part of the course in-person.

I’m not sure how I feel about distance education before beginning this experiment. Learning is something that I really love to do in-person. But so many things that happen online can be evaluated the same way. I recently read articles and commentary about a controversial paper on Twitter research SSRN-id2235423. The research is fodder for some great discussion, but many commenters on the news articles simply chose to trash Twitter. They bemoaned the 140 character limit so strongly that one would think that Twitter is a land of Paris Hilton’s and cats. I’d like the critics to know that yes, you can find Paris Hilton and cats on Twitter or just about anywhere else online. But you can also find something deeper, something that interests you. I recently introduced my nephew to Twitter. He’s a news junkie of sorts, and he was fascinated to see how much emerging news and quality commentary was available. The weekly #wjchat’s alone are reason to follow Twitter (#wjchat is a weekly methods chat between social media journalists) The reach of people on Twitter is unparalleled, and the ability to follow specific areas of interest in deeply engaged ways is also unparalleled. When used correctly, Twitter is a powerful tool.

Online learning as well as the potential to be a powerful tool. But it will require engagement from the people involved. We will need to suspend our natural hesitancy and develop the necessary competencies. I really hope that my classmates will be willing to embrace the experience!