I had wanted to write these notes up into paragraphs, but I think the notes will be more timely, relevant and readable if I share them as they are. This was a really great conference- very relevant and timely- based on a really great issue of Public Opinion Quarterly. As I was reminded at the DC African Festival (a great festival, lots of fun, highly recommended) on Saturday, “In order to understand the future you must embrace the past.”

DC AAPOR Annual Public Opinion Quarterly Special Issue Conference

75^th Anniversary Edition

The Past, Present and Future of Survey Methodology and Public Opinion Research

Look out for slides from the event here: http://www.dc-aapor.org/pastevents.php

 

Note: Of course, I took more notes in some sessions than others…

Peter Miller:

–       Adaptive design- tracking changes in estimates across mailing waves and tracking response bias, is becoming standard practice at Census

–       Check out Howard Schuman’s article tracking attitudes toward Christopher Columbus

• Ended up doing some field research in the public library, reading children’s books

Stanley Presser:

–       Findings have no meaning independent of the method with which they were collected

–       Balance of substance and method make POQ unique (this was a repeated theme)

Robert Groves:

–       The survey was the most important invention in Social Science in the 20^th century – quote credit?

–       3 era’s of Survey research (boundaries somewhat arbritrary)

• 1930-1960
  • Foundation laid, practical development
• 1960-1990
  • Founders pass on their survey endeavors to their protégés
  • From face to face to phone and computer methods
  • Emergence & Dominance of Dillman method
  • Growth of methodological research
  • Total Survey Error perspective dominates
  • Big increase in federal surveys
  • Expansion of survey centers & private sector organizations
  • Some articles say survey method dying because of nonresponse and inflating costs. This is a perennial debate. Groves speculated that around every big election time, someone finds it in their interest to doubt the polls and assigns a jr reporter to write a piece calling the polls into question.
• 1990à
  • Influence of other fields, such as social cognitive psychology
  • Nonresponse up, costs up à volunteer panels
  • Mobile phones decrease cost effectiveness of phone surveys
  • Rise of internet only survey groups
  • Increase in surveys
  • Organizational/ business/ management skills more influential than science/ scientists
  • Now: software platforms, culture clash with all sides saying “Who are these people? Why do they talk so funny? Why don’t they know what we know?”
  • Future
    • Rise of organic data
    • Use of administrative data
    • Combining data sets
    • Proprietary data sets
    • Multi-mode
    • More statistical gymnastics

Mike Brick:

• Society’s demand for information is Insatiable
• Re: Heckathorn/ Respondent Driven samples
  • Adaptive/ indirect sampling is better
  • Model based methods
    • Missing data problem
    • Cost the main driver now
    • Estimation methods
    • Future
      • Rise of multi-frame surveys
      • Administrative records
      • Sampling theory w/nonsampling errors at design & data collection stages
        • Sample allocation
        • Responsive & adaptive design
        • Undercoverage bias can’t be fixed at the back end
          • *Biggest problem we face*
          • Worse than nonresponse
          • Doug Rivers (2007)
            • Math sampling
            • Web & volunteer samples
            • 1^st shot at a theory of nonprobability sampling
          • Quota sampling failed in 2 high profile examples
            • Problem: sample from interviews/ biased
            • But that’s FIXABLE
          • Observational
            • Case control & eval studies
            • Focus on single treatment effect
            • “tougher to measure everything than to measure one thing”

Mick Couper:

–       Mode an outdated concept

• Too much variety and complexity
• Modes are multidimensional
  • Degree of interviewer involvement
  • Degree of contact
  • Channels of communication
  • Level of privacy
  • Technology (used by whom?)
  • Synchronous vs. asynchronous
• More important to look at dimensions other than mode
• Mode is an attribute of a respondent or item
• Basic assumption of mixed mode is that there is no difference in responses by mode, but this is NOT true
  • We know of many documented, nonignorable, nonexplainable mode differences
  • Not “the emperor has no clothes” but “the emperor is wearing suggestive clothes”
  • Dilemma: differences not Well understood
    • Sometimes theory comes after facts
    • That’s where we are now- waiting for the theory to catch up (like where we are on nonprobability sampling)

–       So, the case for mixed mode collection so far is mixed

• Mail w/web option has been shown to have a lower response rate than mail only across 24-26 studies, at least!!
  • (including Dillman, JPSM, …)
  • Why? What can we do to fix this?
  • Sequential modes?
    • Evidence is really mixed
    • The impetus for this is more cost than response rate
    • No evidence that it brings in a better mix of people

–       What about Organic data?

• Cheap, easily available
• But good?
• Disadvantages:
  • One var at a time
  • No covariates
  • Stability of estimates over time?
  • Potential for mischief
    • E.g. open or call-in polls
    • My e.g. #muslimrage
• Organic data wide, thin
• Survey data narrow, deep

–       Face to face

• Benchmark, gold standard, increasingly rare

–       Interviewers

• Especially helpful in some cases
  • Nonobservation
  • Explaining, clarifying

–       Future

• Technical changes will drive dev’t
• Modes and combinations of modes will proliferate
• Selection bias The Biggest Threat
• Further proliferation of surveys
  • Difficult for us to distinguish our work from “any idiot out there doing them”

–       Surveys are tools for democracy

• Shouldn’t be restricted to tools for the elite
• BUT
• There have to be some minimum standards

–       “Surveys are tools and methodologists are the toolmakers”

Nora Cate Schaeffer:

–       Jen Dykema read & summarized 78 design papers- her summary is available in the appendix of the paper

–       Dynamic interactive displays for respondent in order to help collect complex data

–       Making decisions when writing questions

• See flow chart in paper
  • Some decisions are nested
• Question characteristics
  • E.g. presence or absence of a feature
    • E.g. response choices

Sunshine Hillygus:

–       Political polling is “a bit of a bar trick”

• The best value in polls is in understanding why the election went the way it did

–       Final note: “The things we know as a field are going to be important going forward, even if it’s not in the way they’ve been used in the past”

Lori Young and Diana Mutz:

–       Biggest issues:

• Diversity
• Selective exposure
• Interpersonal communication

–       2 kinds of search, influence of each

• Collaborative filter matching, like Amazon
  • Political targeting
  • Contentious issue: 80% of people said that if they knew a politician was targeting them they wouldn’t vote for that candidate
    • My note: interesting to think about peoples relationships with their superficial categories of identity- it’s taken for granted so much in social science research, yet not by the people within the categories

–       Search engines: the new gatekeepers

• Page rank & other algorithms
• No one knows what influence personalization of search results will have
• Study on search learning: gave systematically different input to train engines are (given same start point), results changes Fast and Substantively

Rob Santos:

–       Necessity mother of invention

• Economic pressure
• Reduce costs
• Entrepreneurial spirit
• Profit
• Societal changes
  • Demographic diversification
    • Globalization
    • Multi-lingual
    • Multi-cultural
    • Privacy concerns
    • Declining participation

–       Bottom line: we adapt. Our industry Always Evolves

–       We’re “in the midst of a renaissance, reinventing ourselves”

• Me: That’s framing for you! Wow!

–       On the rise:

• Big Data
• Synthetic Data
  • Transportation industry
  • Census
  • Simulation studies
    • E.g. How many people would pay x amount of income tax under y policy?
• Bayesian Methods
  • Apply to probability and nonprobability samples
• New generation
  • Accustomed to and EXPECT rapid technological turnover
  • Fully enmeshed in social media

–       3 big changes:

• Non-probability sampling
  • “Train already left the station”
  • Level of sophistication varies
  • Model based inference
  • Wide public acceptance
  • Already a proliferation
• Communication technology
  • Passive data collection
    • Behaviors
      • E.g. pos (point of service) apps
      • Attitudes or opinions
    • Real time collection
      • Prompted recall (apps)
      • Burden reduction
        • Gamification
• Big Data
  • What is it?
  • Data too big to store
    • (me: “think “firehoses”)
    • Volume, velocity, variety
    • Fuzzy inferences
    • Not necessarily statistical
    • Coursenes insights

–       We need to ask tough questions

• (theme of next AAPOR conference is just that)
• We need to question probability samples, too
  • Flawed designs abound
  • High nonresponse & noncoverage
  • Can’t just scrutinize nonprobability samples
• Nonprobability designs
  • Some good, well accepted methods
  • Diagnostics for measurement
    • How to measure validity?
    • What are the clues?
    • How to create a research agenda to establish validity?
• Expanding the players
  • Multidisciplinary
    • Substantive scientists
    • Math stats
    • Modelers
    • Econometricians
• We need
  • Conversations with practitioners
  • Better listening skills

–       AAPOR’s role

• Create forum for conversation
• Encourage transparency
• Engage in outreach
• Understanding limitations but learning approaches

–       We need to explore the utility of nonprobability samples

–       Insight doesn’t have to be purely from statistical inferences

–       The biggest players in big data to date include:

• Computational scientists
• Modelers/ synthetic data’ers

–       We are not a “one size fits all” society, and our research tools should reflect that

My big questions:

–       “What are the borders of our field?”

–       “What makes us who we are, if we don’t do surveys even primarily?”

Linguistic notes:

–       Use of we/who/us

–       Metaphors: “harvest” “firehose”

–       Use of specialized vocabulary

–       Use of the word “comfortable”

–       Interview as a service encounter?

Other notes:

–       This reminds me of Colm O’Muircheartaigh- from that old JPSM distinguished lecture

• Embracing diversity
• Allowing noise
• Encouraging mixed methods

I wish his voice was a part of this discussion…