Category Archives: commentary

Reflections and Notes from the Sentiment Analysis Symposium #SAS14

The Sentiment Analysis Symposium took place in NY this week in the beautiful offices of the New York Academy of Sciences. The Symposium was framed as a transition into a new era of sentiment analysis, an era of human analytics or humetrics.

The view from the New York Academy of Sciences is really stunning!

The view from the New York Academy of Sciences is really stunning!

Two main points that struck me during the event. One is that context is extremely important for developing high quality analytics, but the actual shape that “context” takes varies greatly. The second is a seeming disconnect between the product developers, who are eagerly developing new and better measures, and the customers, who want better usability, more customer support, more customized metrics that fit their preexisting analytic frameworks and a better understanding of why social media analysis is worth their time, effort and money.

Below is a summary of some of the key points. My detailed notes from each of the speakers, can be viewed here. I attended both the more technical Technology and Innovation Session and the Symposium itself.

Context is in. But what is context?

The big takeaway from the Technology and Innovation session, which was then carried into the second day of the Sentiment Analysis Symposium was that context is important. But context was defined in a number of different ways.

 

New measures are coming, and old measures are improving.

The innovative new strategies presented at the Symposium made for really amazing presentations. New measures include voice intonation, facial expressions via remote video connections, measures of galvanic skin response, self tagged sentiment data from social media sharing sites, a variety of measures from people who have embraced the “quantified self” movement, metadata from cellphone connections (including location, etc.), behavioral patterning on the individual and group level, and quite a bit of network analysis. Some speakers showcased systems that involved a variety of linked data or highly visual analytic components. Each of these measures increase the accuracy of preexisting measures and complicate their implementation, bringing new sets of challenges to the industry.

Here is a networked representation of the emotion transition dynamics of 'Hopeful'

Here is a networked representation of the emotion transition dynamics of ‘Hopeful’

This software package is calculating emotional reactions to a Youtube video that is both funny and mean

This software package is calculating emotional reactions to a Youtube video that is both funny and mean

Meanwhile, traditional text-based sentiment analyses are also improving. Both the quality of machine learning algorithms and the quality of rule based systems are improving quickly. New strategies include looking at text data pragmatically (e.g. What are common linguistics patterns in specific goal directed behavior strategies?), gaining domain level specificity, adding steps for genre detection to increase accuracy and looking across languages. New analytic strategies are integrated into algorithms and complementary suites of algorithms are implemented as ensembles. Multilingual analysis is a particular challenge to ML techniques, but can be achieved with a high degree of accuracy using rule based techniques. The attendees appeared to agree that rule based systems are much more accurate that machine learning algorithms, but the time and expertise involved has caused them to come out of vogue.

 

“The industry as a whole needs to grow up”

I suspect that Chris Boudreaux of Accenture shocked the room when he said “the industry as a whole really needs to grow up.” Speaking off the cuff, without his slides after a mishap and adventure, Boudreaux gave the customer point of view toward social media analytics. He said said that social media analysis needs to be more reliable, accessible, actionable and dependable. Companies need to move past the startup phase to a new phase of accountability. Tools need to integrate into preexisting analytic structures and metrics, to be accessible to customers who are not experts, and to come better supported.

Boudreaux spoke of the need for social media companies to better understand their customers. Instead of marketing tools to their wider base of potential customers, the tools seem to be developed and marketed solely to market researchers. This has led to a more rapid adoption among the market research community and a general skepticism or ambivalence across other industries, who don’t see how using these tools would benefit them.

The companies who truly value and want to expand their customer base will focus on the usability of their dashboards. This is an area ripe for a growing legion of usability experts and usability testing. These dashboards cannot restrict API access and understanding to data scientist experts. They will develop, market and support these dashboards through productive partnerships with their customers, generating measures that are specifically relevant to them and personalized dashboards that fit into preexisting metrics and are easy for the customers to understand and react to in a very practical and personalized sense.

Some companies have already started to work with their customers in more productive ways. Crimson Hexagon, for example, employs people who specialize in using their dashboard. These employees work with customers to better understand and support their use of the platform and run studies of their own using the platform, becoming an internal element in the quality feedback loop.

 

Less Traditional fields for Social Media Analysis:

There was a wide spread of fields represented at the Symposium. I spoke with someone involved in text analysis for legal reasons, including jury analyses. I saw an NYPD name tag. Financial services were well represented. Publishing houses were present. Some health related organizations were present, including neuroscience specialists, medical practitioners interested in predicting early symptoms of diseases like Alzheimer’s, medical specialists interested in helping improve the lives of people with diseases like Autism (e.g. with facial emotion recognition devices), pharmaceutical companies interested in understanding medical literature on a massive scale as well as patient conversation about prescriptions and participation in medical trials. There were traditional market research firms, and many new startups with a wide variety of focuses and functions. There were also established technology companies (e.g. IBM and Dell) with innovation wings and many academic departments. I’m sure I’ve missed many of the entities present or following remotely.

The better research providers can understand the potential breadth of applications  of their research, the more they can improve the specific areas of interest to these communities.

 

Rethinking the Public Image of Sentiment Analysis:

There was some concern that “social” is beginning to have too much baggage to be an attractive label, causing people to think immediately of top platforms such as Facebook and Twitter and belying the true breadth of the industry. This prompted a movement toward other terms at the symposium, including human analytics, humetrics, and measures of human engagement.

 

Accuracy

Accuracy tops out at about 80%, because that’s the limit of inter-rater reliability in sentiment analysis. Understanding the more difficult data is an important challenge for social media analysts. It is important for there to be honesty with customers and with each other about the areas where automated tagging fails. This particular area was a kind of elephant in the room- always present, but rarely mentioned.

Although an 80% accuracy rate is really fantastic compared to no measure at all, and it is an amazing accomplishment given the financial constraints that analysts encounter, it is not an accuracy rate that works across industries and sectors. It is important to consider the “fitness for use” of an analysis. For some industries, an error is not a big deal. If a company is able to respond to 80% of the tweets directed at them in real-time, they are doing quite well, But when real people or weightier consequences are involved, this kind of error rate is blatantly unacceptable. These are the areas where human involvement in the analysis is absolutely critical. Where, honestly speaking, are algorithms performing fantastically, and where are they falling short? In the areas where they fall short, human experts should be deployed, adding behavioral and linguistic insight to the analysis.

One excellent example of Fitness for Use was the presentation by Capital Market Exchange. This company operationalizes sentiment as expert opinion. They mine a variety of sources for expert opinions about investing, and then format the commonalities in an actionable way, leading to a substantial improvement above market performance for their investors. They are able to gain a great deal of market traction that pure sentiment analysts have not by valuing the preexisting knowledge structures in their industry.

 

Targeting the weaknesses

It is important that the field look carefully at areas where algorithms do and do not work. The areas where they don’t represent whole fields of study, many of which have legions of social media analysts at the ready. This includes less traditional areas of linguistics, such as Sociolinguistics, Conversation Analysis (e.g. looking at expected pair parts) and Discourse Analysis (e.g. understanding identity construction), as well as Ethnography (with fast growing subfields, such as Netnography), Psychology and Behavioral Economics. Time to think strategically to better understand the data from new perspectives. Time to more seriously evaluate and invest in neutral responses.

 

Summing Up

Social media data analysis, large scale text analysis and sentiment analysis have enjoyed a kind of honeymoon period. With so many new and fast growing data sources, a plethora of growing needs and applications, and a competitive and fast growing set of analytic strategies, the field has been growing at an astronomical rate. But this excitement has to be balanced out with the practical needs of the marketplace. It is time for growing technologies to better listen to and accommodate the needs of the customer base. This shift will help ensure the viability of the field and free developers up to embrace the spirit of intellectual creativity.

This is an exciting time for a fast growing field!

Thank you to Seth Grimes for organizing such a great event.

 

Advertisement

Free Range Research will cover the Sentiment Symposium in NYC next week #SAS14

Next week Free Range Research will be in NYC to cover the Sentiment Symposium and Innovation session, and I can’t tell you how excited I am about it!

The development of useful analytics hinges on constant innovation and experimentation, and binary positive/negative measures don’t come close to describing the full potential of social media data. This year’s symposium is an effort to confront the limitations of calcified measures of sentiment head on by introducing new measures and new perspectives.

As a programmer, a quantitative and qualitative analyst, a recent academic, and a fervent believer in the power of the power of mixed methods and interdisciplinary research, I am eager to cover the Symposium as both an enthusiastic and a critical voice. The new directions that will be represented are exciting and interesting, and I expect to gain a better feel for many cutting edges analytic practices. But the proprietary and competitive nature of the social media marketplace has led to countless overblown claims. I do not plan to simply be a conduit for these. My goal will be to share as much as possible of what I learn at the Symposium in a grounded and accessible way, as timely as possible, offering counterpoints and data driven examples when possible, on both my blog and through my Twitter handle @FreeRangeRsrch

I hope you’ll join me!

 

today in research & zen: “What is known as ‘realizing the mystery’ is nothing more than breaking through to grab an ordinary person’s life” Te-Shan

Medical Errors: detecting them, preventing them, and dealing with their aftermath

This is primarily a report on an event, but I’ve added links, stories and examples to my notes.

The event:

Bioethics lecture on Error: https://www.eventbrite.com/e/conversations-in-bioethics-tickets-10276951639

Brief description: “Join distinguished national experts John James, PhD, former chief toxicologist at NASA and founder of Patient Safety America, Brian Goldman, MD, emergency physician-author and host of the CBC’s White Coat, Black Art, Beth Daley Ullem, MBA, nationally-recognized advocate for patient safety and quality and SFS alum, for a lively discussion and Q&A moderated by Maggie Little, Director of the Kennedy Institute of Ethics.”

At the Beautiful Kennedy Institute, Georgetown University

The follow-up:

For those who are particularly interested in this topic, the Kennedy Institute has an upcoming Bioethics MOOC starting 4/15: http://kennedyinstitute.georgetown.edu/about/news/bioethics-mooc-spring-launch.cfm

Why Create this resource?

What follows is a long resource- an in depth summary of the lecture I attended last night, complete with many links to other resources and a few stories and examples. Like the members of this panel, I have experienced a dramatic medical error. In 2012 my mother was on life support after experiencing a period of time with no oxygen to her brain. Her heart had stopped twice, and she was unresponsive. I am her only child, and I had essentially moved into the hospital with her in order to be her advocate. It was my decision whether or not to continue life support, and the main deciding factor was whether or not she was brain dead. She was given an EEG test, and it did not look good. There was a delay before I heard the results of the test, and I spent that delay researching her EEG patterns to try to understand what was going on. The next day the medical staff involved in her case sat me down and told me she was indeed brain dead. It wasn’t until my cousin had announced her passing on Facebook, I was saying my final goodbyes, and my aunt was on the phone with the funeral home that the doctors on the case realized they had miscommunicated. Another patient in another hospital was brain dead, but my mother was not officially brain dead. Her brain activity appeared to be seizure activity, and it wasn’t clear if there was anything else going on. The group apologized, and we were forced to reverse the story and try to explain to friends and family (and ourselves) that she was not actually dead, but she was still very close to it. There was a tide of “Go get em!” cries, which were difficult to deal with when we did indeed have to remove life support a few days later.

After this event, one of the physicians involved in the miscommunication focused my attention on a collaborative project. We began work on a grand rounds presentation for the hospital. We planned to talk about errors in general and, more specifically, what could be learned from this error. I did quite a bit of reading and research. We had some great discussions, and I started to attend a medical discourse group in my graduate linguistics department. At some point I will probably return to the notes from that collaboration and assemble a blog post about them.

It is because of that experience and that project that I assembled the resource below. I sincerely hope that you find it interesting and useful.

Please note that this is based on many pages of notes. Unfortunately my notes did not attribute points to individual panelists. I apologize for that omission.

Prevalence and Detection

An estimated 100,000 lives are lost each year from preventable medical error (according to 1999 landmark Institute Of Medicine report), but this data is old (1984, New York State) and focused on errors of commission. There are many other kinds of errors, including omission, context, diagnostic and communication. Measuring preventable deaths is easier than measuring mistakes overall, but mistakes that do not directly lead to death cause plenty of heartache every day as well.

One more recent attempt to detect medical errors involved isolating common trigger words that accompany medical mistakes in medical files and then having the cases reviewed by medical professionals to see if the deaths were indeed preventable. By this method, the estimate was closer to 210,000 preventable deaths. This method was more comprehensive, but records don’t have the right parameters or standardization to make this process ideal. Some estimates are as high as 440,000 deaths per year.

Regardless of the exact numbers, for physicians, there is a near 100% possibility of making a mistake at some point. This fact alone should change the paradigm from avoiding errors altogether to openly anticipating and working with errors as they happen.

Aftermath

After a medical error occurs, heartache abounds. But contrary to social conventions outside of the medical establishment, contact is often strictly controlled and regulated after the incident, and the physician is rarely able to say “I’m sorry.” This can cause a lack of closure for both the patients and the doctors. The aftermath of one of these errors forms a second layer of trauma for all of those involved.

The first target for any kind of error is often the individual who made the mistake, not the system that enabled the mistakes. The system quickly closes around this individual. The hospital risk administration sets in. Privacy walls are erected, and it becomes very difficult to take responsibility for one’s errors. A perfect storm of system and culture clash together, resulting in ill-advised words and actions on the part of those involved. At such a sensitive time, the words of care providers are often burned into the minds of the deceased patient’s advocates and family members. Blame is often tossed around indiscriminately. The survivors are often left feeling confused. One of the panelists remembers her physician counseling her with “I really don’t know why God needed your baby more than you did.”

The medical providers at this point are isolated from their patients and often prohibited from discussing these incidents with each other. At such a vulnerable moment, they are left to deal with it alone, taking each incident as a private failure when mistakes are a universal human condition. If other providers hear about the incident, they will often exacerbate the problem by not making eye contact, demonstrating their vicarious shame, reinforcing the problem as a repudiation of all a doctor is supposed to be.

System level Problems

The medical system is large and complicated enough to really enable errors. There are so many medical professionals, patients, laypeople and touchpoints, and the body itself is quite a complicated system- some of which is better understood and some of which is still largely undocumented territory. The medical system is evolving fast from the mom and pop doctors of the past to the large complexes of today. The modern medical system has its hand in businesses that no one would have imagined before. Some hospitals boast dental facilities, nursing homes, outpatient clinics, and even foster care facilities. The changing rules for insurance payments and the increasing role of legal actors also have a significant influence on the system.

In order for hospitals to make money, many end up adjusting the patient care ratios. Some stretch these ratios to the breaking point, putting medical staff in a position where they can barely keep up. The pressure for productivity is much higher now than it was in the recent past. Many facilities are over capacity, and space is at a premium. This can put medical staff in an awkward position where there are constant workarounds and makeshift solutions. These kinds of problems can lead to  errors of context. The same patient may be treated differently in the ambulatory care area of the same wing than in the rapid assessment area. In the words of one panelist “geography is destiny in the E.R.” Movement in space within a medical facility is both physical and cognitive.

Scheduling is also a huge issue in medical facilities. Long stretches of work without sleep are a better known precursor to many medical errors.

Technology

Technology is integral to the modern medical system and has saved many lives. But technology training and interface design are extremely important. One panelist reported that a medical professional confessed to him years after his son’s preventable death that the MRI machine was new, and no one onhand knew how to use it properly. Others have reported on the influence of signal fatigue- it is very hard amidst a constant stream of signals to ferret out the most important among them.

Technology was a real point of frustration for me when I had my first child. I was induced in the evening and felt increasingly strong contractions all night. When the nurses came to check on me, I reported that I was in labor, but the pattern on the monitor was not consistent with what they would call labor. Once I started to push I called them back and requested an exam, and fortunately, although my doctor and the doctor on call were not available, they were qualified to catch the baby.

Medical culture

One of the panelists told the story of a physician who began his shift by calling together his team, warning them that he did not get much of a night’s sleep the night before, and asking them to watch his back a bit more closely than usual. This runs starkly contrary to typical medical enculturation. Medical culture makes it harder to admit mistakes or to be human. One panelist commented “We’re very defensive about our mistakes.” This is emblematic  of a culture that can’t handle its own humanity. This repulsion by error is compounded by a system that doesn’t comment but rather expects good performance. The “no news is good news” ethic means that a physician can go his or her entire career without ever hearing any feedback, and that can be a good thing.

In medicine, the smartest person in the room is quickly the person in charge. One of the panelists, Brian “didn’t want to be a high-maintenance student” as a resident by asking too many questions or requesting help too often. This attitude wound up fatal for one of his patients. Errors are a reminder of human fallibility, and medical professionals are supposed to be infallible. Brian talked more about this in a TED talk. In it, he spoke of batting averages. We assume that error is a natural part of other jobs, but what is an acceptable batting average for a surgeon? A mistake can mean that one was lazy or incompetent or had a lapse. Which one does the physician want to admit to? None! Instead, they often live in terror when one mistake happens that another will soon follow. One panelist said the words he most fears as a medical professional are “Do you remember?”

Instead of the culture of shame and blame, we could benefit from being scientific about error: exhibiting genuine curiosity about errors, measuring them, and developing and testing treatments for them. One panel member mentioned a surgeon who developed a kind of flight data recorder for surgery: http://www.icee-con.org/papers/2008/pdf/O-100.pdf . Apparently this surgeon has been dubbed “the most dangerous man in surgery.”

Isolation and selective training

People are trained in the context of the settings where they have worked. Different settings see different kinds of challenges. Shouldn’t there be a better system for sharing challenges and solutions across institutions?

Handwriting

It is pretty incredible that such a high stakes field rests on human handwriting. This is made worse by the lack of value placed on making handwriting legible and on the decreasing abilities of a technologically savvy population to decipher human handwriting. How many of you can read cursive?

Science or Art?

One interesting aspect of medicine is the way it is a field composed of scientists who view themselves as artists. This is evident in the total lack of standardization in medical care. You will have a different experience, even with the same condition, across locations and providers. Even within a single hospital individual doctors act as subcontractors, providing individualized service as only they can, despite the common environment. Sometimes there are standards or guidelines set for specific areas of medicine with a goal of instituting consistency. But the adaptation of these standards and attitudes toward these standards are far from universal. The standards take shape differently across locations and providers.

The panel members mentioned the success of VA hospitals in this area. They are better at standardization. Vertically integrated healthcare can be much more progressive.

Areas for improvement

So what kind of changes would improve the system? Some prominent authors liken error models to those in the airplane industry. This is tricky, because medicine is far more complicated that aviation- although both are high stakes fields that require inhuman levels of perfection among human actors. But even if the systems are different, they can still learn from each other.

Atul Gawande is a well known author The Checklist Manifesto. He has been advocating for many of the checklists and safety features that are standard in the aviation industry to be applied to medicine. He also wrote a piece about what medicine can learn from The Cheesecake Factory.

Some suggested areas for improvement include instituting redundancies, collapsing hierarchies and patient centered care.

One panel member was involved with error prevention at more of a business level. She mentioned the power of adding redundancies. Adding redundancies should be common practice and is common practice in other high stakes fields. Redundancies should be worked into routines and checks, although models of modern efficiency seem to be moving away from them. She also mentioned the powerful potential of dashboards and the importance of comparative information. One great example of the power of comparative information is “Solutions for Patient Safety” http://www.solutionsforpatientsafety.org/ . This is a group of 78 pediatric hospitals that share a common dashboard. Using the dashboard the hospitals can see how they stand in terms of infections and other errors compared to the rest of the network. It’s a teaching model- the best teach the rest about the measures they’re using to combat each problem. The panelist mentioned that we buy healthcare products without comparative information, but information on dashboards can really increase accountability.

Collapsing hierarchies would make it more culturally acceptable to report medical errors. This could also be augmented through multidisciplinary peer reviews, involving everyone from providers across medical specialties and training to janitors and other people present at the time of care.

One of the panelists wrote a patient bill of rights. An audience member commented on the need for patients to feel more powerful and have more power in medical situations. He noted that the playing field between doctor and patient is inherently unequal. As soon as you remove your clothes and put on the patient smock you begin to feel powerless. He noted that some medical providers will take advantage of that vulnerability. The foundation of patient centered care is informed consent. If you don’t understand your options, you cannot make an informed choice.

One specific example of an area where patients are unable to make informed decisions was off-label prescriptions. Prescriptions are often prescribed off-label, meaning that the patient is not part of the population base for which the drug was tested. This was the case for me when my first child was born, and I was induced with Cytotec. When she was born, a healthy 8 lb 3 ounce baby aspirated meconium and ended up in the NICU while I was treated for hemorrhage. I knew nothing of the drug or the potential consequences. In fact, I had chosen an unmedicated chidbirth and eschewed interventions altogether.

Another example of an area where patients can’t always make informed decisions is that of cost. There has been quite a bit of buzz lately about the ridiculous hospital bills patients receive upon discharge. I can’t tell you how paranoid I am about any supplies used on myself or my kids in the E.R. having seen some of those bills. A close friend of mine recently had an incident where an inexpensive scheduled dentist appointment turned into over $2000 in charges, due immediately. That incident led to an extensive series of phonecalls between myself and the dental office, debating consent.

An audience member spoke about the importance of patient advocates.  Apparently there is a growing business of professional patient advocates. I think that this is wonderful, because historically the only qualification necessary for a patient advocate was that they not be the patient. I’ve had the experience of reading transcripts of doctor patient visits that included advocates. Certainly not all advocates are built alike! This role is more deeply explored in the book “High Performance Healthcare”

Opportunities for Linguists

There are two main applications for linguistics that are most evident in this discussion. One is the potential for computational linguists and natural language processing experts to mine the textual data available in  electronic health records as they become increasingly available. The other is the opportunity for discourse analysts to conduct research on the actual communication between everyone involved. Discourse analysts can both develop and institute more structured protocols, such as the double verification before certain medications and procedures, and raise awareness regarding instances when less than optimal communication styles can lead to mix-ups or other mistakes. Discourse analysts who specialize in apologies could be particularly effective advisors in training medical professionals to talk with patients and their advocates and family following medical errors. This is a strong interest of mine, and I’m lucky enough to attend regular medical discourse discussion groups with the head of my graduate department, Heidi Hamilton. Her work is a real treasure trove of medical discourse, well worth investigating further.

On a personal note, it is also very healing for victims and survivors to build narratives around these incidents that help to give them a wider context and meaning. I wrote about that process here: https://freerangeresearch.com/2012/05/22/ot-on-loss-and-grief-and-the-power-of-storytelling/

You may notice that I decided at that point not to give the medical error a place in my mom’s story. That was an important decision for me that helped me to heal.

Moving on

The three panelists had all lost people due to medical errors. I’ve also been the victim of medical errors. We were able to find some healing in the process of going deeper into the errors and the medical system that enabled them. You have also probably suffered in some way as the result of a medical error. It is also important to note that all of us have also had our lives made better by medicine at some point, and we probably also all know people whose lives were saved by medicine. It is an imperfect system, but it is a system with a lot of strengths.

Storytelling about the Past and Predicting the Future: On People, Computers and Research in 2014 and Beyond

My Grandma was a force to be reckoned with. My grandfather was a writer, and he described her driving down the street amidst symphonies. She was beautiful and stubborn, strong willed and sharp. Once a young woman with the good looks of a model, she wore high heels and took daily trips to the gym well into her 90’s. At the age of 94 she managed to run across her house, turn off the water and stand with her hand on her hip in front of the shower before I returned from the next room over with the shampoo I forgot (lest I waste water).

My Grandma, looking amazing

My Grandma, looking amazing

A few years ago I visited her in Florida. She collected work for all of her visitors to do, and we were busy from the moment I arrived. To my surprise, many of the tasks she had gathered involved dealing with customer service and discovering the truth in advertisements. At one point she led me into the local pharmacy with a stack of papers and asked to see the manager. Once she found the manager she began to go through the papers one by one and ask about them. The first paper on the stack was about the Magic Jack. He showed her the package, and she questioned him in depth about how it worked. I was shocked. I’d never thought of a store manager in this role before.

After that trip I began to pay closer attention to the ways in which the people around me dealt with customer service, and I became a kind of customer service liaison for my family. My older family members had an expectation that any customer service agent be both extensively knowledgeable and dependably respectful, but the problems of customer service seemed to have grown beyond this small, personable level to a point where a large network of people with structurally different areas of knowledge act together to form a question answering system. The amount and structure of knowledge necessary has become the focus of the customer service problem, and people everywhere complain about the lack of knowledge, ability and pleasant attitude of the customer service agents they encounter.

This is a problem with many layers and levels to it, and it is a problem that reflects the developing data science industry well. In order to deliver good customer service a great deal of information has to be organized and structured in a meaningful way to allow for optimal extraction. But this layer cannot be everything. The customer service interaction itself needs to be set-up in such a way to allow customers to feel satisfied. People expect personalized, accurate interactions that are structured in a way that is intuitive to them. The customer service experience cannot be the domain of the data scientists. If it is automated, it requires usability experts to develop and test systems that are intuitive and easy to use. If it is done by people, the people need to have access to the expertise necessary for them to do their job and be trained in successful interpersonal interaction. I believe that this whole system could be integrated well under a single goal: to provide timely and direct answers to customer inquiries in 3 steps or less.

The past few years have brought a rapid increase in customization. We have learned to expect the information around us to be customized, curated and preprocessed. We expect customer service to know intuitively what our problems are and answer them with ease. We expect Facebook to know what we want to see and customize our streams appropriately. We expect news sites to be structured to reflect the way we use them. This increase in demand and expectations is the drive behind our hunger for data science, and it will fuel a boom in data and information science positions until we have a ubiquitous underlayer of organized information across all necessary domains.

But data and information science are new fields and not well understood. Our expectations as users exceed the abilities of this fast-evolving field. We attract pioneers who are willing to step into a field that is changing shape beneath their feet as they work. But we ask for too much of a result and expect too much of a result, because these pioneers can’t be everything across all fields. They are an important structural layer of our newly unfolding economy, but in each case, another layer of people are needed in order to achieve the end result.

Usability is an important step above the data and information science layer. Through usability studies, Facebook will eventually learn that people and goals are not constant across all visits. Sometimes I look at Facebook simply to see if I’ve missed any big developments in the lives of my friends and loved ones. Sometimes I want to catch news. Sometimes I’m bored and looking for ridiculous stuff to entertain me. Sometimes I have my daughter next to me and want to show her funny pet pictures that I normally wouldn’t look twice at. Through usability studies, Facebook will eventually learn that users need some control over the information presented to them when they visit.

Through usability studies newspapers will better understand the important practice of headline scanning and develop pay models that work with peoples reading habits. Through qualitative research newspapers will understand their importance as the originators of news about big events with few witnesses, like peace treaties and celebrity births and deaths and the real value of social media for events with large numbers of witnesses and points of view. News media sources are deep in a period of transition where they are learning to better understand dissemination, virality, clicks, page views, reader behavior and reader expectations, and the strengths and weaknesses of social media news sources.

There have been many blog posts (like this one) about Isaac Asimov’s predictions for the future, because he was so right about so many things. At this point we’re at a unique vantage point where his notions of machine programmers and machine tenders are taking deeper shape. This year we will continue to see these changes form and reform around us.

An Analytical person at the Nutcracker (or Research Methodology, Nutcracker Style)

Last night we attended a Russian Ballet performance of the Nutcracker. It was a great performance, and fun was had by all.

2013-12-17 18.38.03

Early in the performance I realized that although I have developed some understanding of the ballet, I hadn’t shared any of that knowledge with my kids. At this point, I started whispering to them quietly to explain what they were seeing. I whispered quick, helpful comments, such as “those are toys dancing” and “the kids have gone to sleep now, so this is just the adults dancing.” It wasn’t long into the performance that this dynamic began to change. I realized that their insights were much funnier than mine “wow, that guy should go on ‘So You Think You Can Dance!’ or ‘The Voice’ or something! “and that my comments were starting to be pretty off-base. My comments evolved into a mash-up of “The kids have gone to sleep now” “No, I guess the kids haven’t gone to sleep yet” “I really can’t tell if the kids are still up or not!” and “Those are the sugarplum fairies” “Wait, no, maybe these are the sugar plum fairies?” and “I don’t know, sweetie, just watch them dance!” By the end of the show I had no idea what was going on or why the Chuck.E.Cheese king was dancing around on stage (although one of the girls suspected this particular king was actually a bear?). The mom next to me told me she didn’t know what was going on either “and,” she added, “I go to the Nutcracker every year! Maybe that was what made it a Russian Nutcracker?” …And here I thought the Russian influences were the Matryoshka dolls and the Chinese dancers clothed in yellow (despite the awkward English conversation that the costumes prompted).

At the beginning of the show I was nervous to whisper with my kids, but I soon realized that there was a low hum all around me and throughout the concert hall of people whispering with their kids. This, I think, is what remix research methods should be all about- recording and interviewing many audience members to gain a picture of the many perspectives in their interpretations of the show. Here is a challenge question to my readers who are hipper to qualitative research methods: what research strategy could best capture many different interpretations of the same event?

Earlier this week I spoke with a qualitative researcher about the value of an outsider perspective when approaching a qualitative research project. Here is a good example of this dynamic at play: people clapped at various parts of the performance. I recognized that people were clapping at the end of solo or duo performances (like jazz). If I were to describe these dances, I would use the claps as a natural demarcation, but I probably would not think to make any note of the clapping itself. However, the kids in my crew hadn’t encountered clapping during a show before and assumed that clapping marked “something awesome or special.” Being preteens, the kids wanted to prove that they could clap before everyone else, and then revel in the wave of clapping that they seemingly started. At one point this went awry, and the preteens were the only audience members clapping. This awkward moment may have annoyed some of the people around us, but it really made the little sister’s day! From a research perspective, these kids would be more likely to thoroughly document and describe the clapping than I would, which would make for a much more thorough report. Similarly, from a kids-going-to-a-show perspective this was the first story they told to their Dad when they got home- and one that kicked off the rest of our report with uncontrollable laughter and tears.

As the show went on and appeared not to follow any of the recognizable plot points that I had expected (I expected a progressive journey through worlds experienced from the vantage of a sleigh but instead saw all of the worlds dancing together with some unrecognizable kids variously appearing on a sleigh and the main characters sometimes dancing in the mix or on their own), I began to search for other ways to make sense of the spectacle. I thought of a gymnast friend of mine and our dramatically different interpretations of gymnastics events (me: “Wow! Look what she did!” her: “Eh, she scratched the landing. There will be points off for that.” Which parts of the dancing should I be focusing on? I told my little one “Pay attention, so we can try these moves at home.” Barring any understanding of the technical competencies involved (but sure that laying your body at some of these amazing angles, or somehow spinning on one foot, or lifting another person into the air require tons of training, skills and knowledge) or any understanding of the plot as it was unfolding in front of me, I was left simply to marvel at it all. This is why research is an iterative process. In research, we may begin by marveling, but then we observe, note, and observe again. And who knows what amazing insights we will have developed once the process has run its course enough times for events to start making sense!

To be a researcher is not to understand, but rather to have the potential to understand- if you do the research.

The holiday season and the post-degree process

I haven’t blogged much this month.

Yesterday I didn’t blog because I was wandering around my neighborhood with my kids and my winter boots, looking for the ultimate sledding hill that wasn’t just mud.

 

2013-12-10 13.03.42

I did get this cool shot of the snow melting:

2013-12-10 14.11.15

This past weekend I didn’t blog because I was trying to get some holiday shopping done. Holiday shopping is a mess of contradictions. The music and festive spirit are relaxing and wonderful, but the task at hand is to reckon with our wants. My goal lately has been exactly the opposite of this- to appreciate what abundance already constitutes my life and not to focus on needing or wanting more. This is an important part of my post degree process.

At the time of my graduation I joked with a close friend about expecting life to be like a musical, with the people around me singing and dancing my accomplishments and those of my classmates. For those of you still in school I hate to break your bubble, but there will likely not be a musical in your honor, as deserving as you may be.

Graduation is not the end of your work as a student. Your work will extend beyond graduation and in to what I’ve come to think of as an extra semester of undetermined length. This is the time when we try to make all of our hard work pay off. We learn that the world will not recognize our accomplishments unless we learn how to be our own best advocates, and we learn how difficult it is to advocate for ourselves across lines of field and areas of practice.

This process involves a reckoning between the idealized notions of our future that motivated us through late-nighters and all-nighters and the realities of our post degree lives. It also involves a surprisingly long transition from the frenetic pace of student life to the appreciative pace of real life. We learn how to channel the energy that is no longer focused on school work but free to roam across a wide range of interests and responsibilities. We forge a new set of priorities. We realize that we will not find jobs that are as well rounded as we are. We see that we are not frozen in place after our degrees but will continue the lifelong process of learning. We begin to find peace in the knowledge that what we have is enough. We may not have the yacht and the private plane, but we have food on our plates and in our bellies. And what we have is enough.

Graduates (especially in today’s employment market) have to wrestle with the responsibilities of post-degree life, the lack of recognition of their academic accomplishments, and the transition [back?] into the swing of daily work life. We have to transition from the big dreams of school life to the small rewards of real life. For me this process involves a compacting. It involves tightening the family budget and saving for bigger goals. It involves family challenges to see how long we can go between trips to the grocery store and the fun set of culinary challenges that rise from the emptier cupboards (Have you seen those cooking shows where the contestants are challenged to invent a meal based on a small number of random ingredients?). It involves decluttering my house to get rid of extra stuff, appreciate what we have and lessen our responsibilities (less stuff to clean!), and it involves spoiling my family with the time and attention I couldn’t give them before.

This all seems to directly contradict the goals of holiday shopping. I wandered through aisle after aisle of stuff that I couldn’t imagine needing or wanting, thinking of needs and wants as a kind of black hole where needing and wanting can simply lead to more needing and wanting. I’m not sure how my holiday shopping process will shake out this year, but I do know that my happiness and the happiness of those I love can’t be found on any store shelves.

For you students, recent graduates and professional researchers and other readers, I wish you all the peace and gratitude of the season. May the new year bless you with continued curiosity. May we never stop learning and growing. May the process and daily rituals of our lives be reward enough. We can’t anticipate the challenges 2014 will bring, but let us be grateful that we have the tools that we will need to greet them with.

And most of all, I want to thank those of you who read my blog posts. Thank you for your time and attention and for encouraging me to continue to explore. I hope to reward you soon with a rundown of some particularly great events I’ve attended lately!

Great readings that might shake you to your academic core? I’m compiling a list

In the spirit of research readings that might shake you to your academic core, I’m compiling a list. Please reply to this thread with any suggestions you have to add. They can be anything from short blog posts (microblog?) to research articles to books. What’s on your ‘must read’ list?

Here are a couple of mine to kick us off:

 

Charles Goodwin’s Professional Vision paper

I don’t think I’ve referred to any paper as much as this one. It’s about the way our professional training shapes the way we see the things around us. Shortly after reading this paper I was in the gym thinking about commonalities between the weight stacks and survey scales. I expect myself to be a certain relative strength, and when that doesn’t correlate with the place where I need to place my pin I’m a little thrown off.

It also has a deep analysis of the Rodney King verdict.

 

Revitalizing Chinatown Into a Heterotopia by Jia Lou

This article is based on a geosemiotic analysis of DC’s Chinatown. It is one of the articles that helped me to see that data really can come in all forms

 

After method: Mess in Social Science Research by John Law

This is the book that inspired this list. It also inspired this blog post.

 

On Postapocalyptic Research Methods and Failures, Honesty and Progress in Research

I’m reading a book that I like to call “post-apocalyptic research methodology.” It’s ‘After Method: Mess in Social Science Research’ by John Law. At this point the book reads like a novel. I can’t quite imagine where he’ll take his premise, but I’m searching for clues and turning pages. In the meantime, I’ve been thinking quite a bit about failure, honesty, uncertainty and humility in research.

How is the current research environment like a utopian society?

The research process is often idealized in public spaces. Whether the goal of the researcher is to publish a paper based on their research, present to an audience of colleagues or stakeholders about their research, or market the product of their research, all researchers have a vested interest in the smoothness of the research process. We expect to approach a topic, perform a series of time-tested methods or develop innovative new methods with strong historical traditions, apply these methods as neatly as possible, and end up with a series of strong themes that describe the majority of our data. However, in Law’s words “Parts of the world are caught in our ethnographies, our histories and our statistics. But other parts are not, and if they are then this is because they have been distorted into clarity.” (p. 2) We think of methods as a neutral middle step and not a political process, and this way of thinking allows us to focus on reliability and validity as surface measures and not inherent questions. “Method, as we usually imagine it, is a system for offering more or less bankable guarantees.” (p. 9)

Law points out that research methods are, in practice, very limited in the social sciences “talk of method still tends to summon up a relatively limited repertoire of responses.” (p. 3) Law also points out that every research method is inherently political. Every research method involves a way of seeing or a way of looking at the data, and that perspective maps onto the findings it yields. Different perspectives yield different findings, whether they are subtly or dramatically different. Law’s central assertion is that methods don’t just describe social realities but also help to create them. Recognizing the footprint of our own methods is a step toward better understanding our data and results.

In practice, the results that we focus on are largely true. They describe a large portion of the data, ascribing the rest of the data to noise or natural variation. When more of our data is described in our results, we feel more confident about our data and our analysis.

Law argues that this smoothed version of reality is far enough from the natural world that it should perk our ears. Research works to create a world that is simple and falls into place neatly and resembles nothing we know, “’research methods’ passed down to us after a century of social science tend to work on the assumption that the world is properly to be understood as a set of fairly specific, determinate, and more or less identifiable processes.” (p. 5) He suggests instead that we should recognize the parts that don’t fit, the areas of uncertainty or chaos, and the areas where our methods fail. “While standard methods are often extremely good at what they do, they are badly adapted to the study of the ephemeral, the indefinite and the irregular.” (p. 4). “Regularities and standardizations are incredibly powerful tools, but they set limits.” (p. 6)

Is the Utopia starting to fall apart?

The current research environment is a bit different from that of the past. More people are able to publish research at any stage without peer review using media like blogs. Researchers are able to discuss their research while it is in progress using social media like Twitter. There is more room to fail publicly than there ever has been before, and this allows for public acknowledgment of some of the difficulties and challenges that researcher’s face.

Building from ashes

Law briefly introduces his vision on p. 11 “My hope is that we can learn to live in a way that is less dependent on the automatic. To live more in and through slow method, or vulnerable method, or quiet method. Multiple method. Modest method. Uncertain method. Diverse method.”

Many modern discussions of about management talk about the value of failure as an innovative tool. Some of the newer quality control measures in aviation and medicine hinge on the recognition of failure and the retooling necessary to prevent or limit the recurrences of specific types of events. The theory behind these measures is that failure is normal and natural, and we could never predict the many ways in which failure could happen. So, instead of exclusively trying to predict or prohibit failure, failures should be embraced as opportunities to learn.

Here we can ask: what can researchers learn from the failures of the methods?

The first lesson to accompany any failure is humility. Recognizing our mistakes entails recognizing areas where we fell short, where our efforts were not enough. Acknowledging that our research training cannot be universal, that applying research methods isn’t always straightforward and simple, and that we cannot be everything to everyone could be an important stage of professional development.

How could research methodology develop differently if it were to embrace the uncertain, the chaotic and the places where we fall short?

Another question: What opportunities to researchers have to be publicly humble? How can those spaces become places to learn and to innovate?

Note: This blog post is dedicated to Dr Jeffrey Keefer @ NYU, who introduced me to this very cool book and has done some great work to bring researchers together

Methodology will only get you so far

I’ve been working on a post about humility as an organizational strategy. This is not that post, but it is also about humility.

I like to think of myself as a research methodologist, because I’m more interested in research methods than any specific area of study. The versatility of methodology as a concentration is actually one of the biggest draws for me. I love that I’ve been able to study everything from fMRI subjects and brain surgery patients to physics majors and teachers, taxi drivers and internet activists. I’ve written a paper on Persepolis as an object of intercultural communication and a paper on natural language processing of survey responses, and I’m currently studying migration patterns and communication strategies.

But a little dose of humility is always a good thing.

Yesterday I hosted the second in a series of online research, offline lunches that I’ve been coordinating. The lunches are intended as a way to get people from different sectors and fields who are conducting research on the internet together to talk about their work across the artificial boundaries of field and sector. These lunches change character as the field and attendees change.

I’ve been following the field of online research for many years now, and it has changed dramatically and continually before my eyes. Just a year ago Seth Grimes Sentiment Analysis Symposia were at the forefront of the field, and now I wonder if he is thinking of changing the title and focus of his events. Two years ago tagging text corpora with grammatical units was a standard midstep in text analysis, and now machine algorithms are far more common and often much more effective, demonstrating that grammar in use is far enough afield from grammar in theory to generate a good deal of error. Ten years ago qualitative research was often more focused on the description of platforms than the behaviors specific to them, and now the specific innerworkings of platform are much more of an aside to a behavioral focus.

The Association of Internet Researchers is currently having their conference in Denver (#ir14), generating more than 1000 posts per day under the conference hashtag and probably moving the field far ahead of where it was earlier this week.

My interest and focus has been on the methodology of internet research. I’ve been learning everything from qualitative methods to natural language processing and social network analysis to bayesian methods. I’ve been advocating for a world where different kinds of methodologists work together, where qualitative research informs algorithms and linguists learn from the differences between theoretical grammar and machine learned grammar, a world where computer scentists work iteratively with qualitative researchers. But all of these methods fall short because there is an elephant in the methodological room. This elephant, ladies and gentleman, is made of content. Is it enough to be a methodological specialist, swinging from project to project, grazing on the top layer of content knowledge without ever taking anything down to its root?

As a methodologist, I am free to travel from topic area to topic area, but I can’t reach the root of anything without digging deeper.

At yesterday’s lunch we spoke a lot about data. We spoke about how the notion of data means such different things to different researchers. We spoke about the form and type of data that different researchers expect to work with, how they groom data into the forms they are most comfortable with, how the analyses are shaped by the data type, how data science is an amazing term because just about anything could be data. And I was struck by the wide-openness of what I was trying to do. It is one thing to talk about methodology within the context of survey research or any other specific strategy, but what happens when you go wider? What happens when you bring a bunch of methodologists of all stripes together to discuss methodology? You lack the depth that content brings. You introduce a vast tundra of topical space to cover. But can you achieve anything that way? What holds together this wide realm of “research?”

We speak a lot about the lack of generalizable theories in internet research. Part of the hope for qualitative research is that it will create generalizable findings that can drive better theories and improve algorithmic efforts. But that partnership has been slow, and the theories have been sparse and lightweight. Is it possible that the internet is a space where theory alone just doesn’t cut it? Could it be that methodologists need to embrace content knowledge to a greater degree in order to make any of the headway we so desperately want to make?

Maybe the missing piece of the puzzle is actually the picture painted on the pieces?

comic

The data Rorschach test, or what does your research say about you?

Sure, there is a certain abundance of personality tests: inkblot tests, standardized cognitive tests, magazine quizzes, etc. that we could participate in. But researchers participate in Rorschach tests of our own every day. There are a series of questions we ask as part of the research process, like:

What data do we want to collect or use? (What information is valuable to us? What do we call data?)

What format are we most comfortable with it in? (How clean does it have to be? How much error are we comfortable with? Does it have to resemble a spreadsheet? How will we reflect sources and transformations? What can we equate?)

What kind of analyses do we want to conduct? (This is usually a great time for our preexisting assumptions about our data to rear their heads. How often do we start by wondering if we can confirm our biases with data?!)

What results do we choose to report? To whom? How will we frame them?

If nothing else, our choices regarding our data reflect many of our values as well as our professional and academic experiences. If you’ve ever sat in on a research meeting, you know that “you want to do WHAT with which data?!” feeling that comes when someone suggests something that you had never considered.

Our choices also speak to the research methods that we are most comfortable with. Last night I attended a meetup event about Natural Language Processing, and it quickly became clear that the mathematician felt most comfortable when the data was transformed into numbers, the linguist felt most comfortable when the data was transformed into words and lexical units, and the programmer was most comfortable focusing on the program used to analyze the data. These three researchers confronted similar tasks, but their three different methods that will yield very different results.

As humans, we have a tendency to make assumptions about the people around us, either by assuming that they are very different or very much the same. Those of you who have seen or experienced a marriage or serious long-term partnership up close are probably familiar with the surprised feeling we get when we realize that one partner thinks differently about something that we had always assumed they would not differ on. I remember, for example, that small feeling that my world was upside down just a little bit when I opened a drawer in the kitchen and saw spoons and forks together in the utensil organizer. It had simply never occurred to me that anyone would mix the two, especially not my own husband!

My main point here is not about my husband’s organizational philosophy. It’s about the different perspectives inherently tied up in the research process. It can be hard to step outside our own perspective enough to see what pieces of ourselves we’ve imposed on our research. But that awareness is an important element in the quality control process. Once we can see what we’ve done, we can think much more carefully about the strengths and weaknesses of our process. If you believe there is only one way, it may be time to take a step back and gain a wider perspective.