Sentiment Analysis and Twitter
More and more, the internet is being mined and scratched for data – and, of course, at the moment our current focus is on forecasting the election. Tuesday night, for example, CNN will use “sentiment analysis” to track the themes that preoccupy voters.
Spontaneous and unsolicited information about the feelings voters have towards candidates and issues can be found on Twitter and other sites of social conversation. David Bohrman, CNN’s Washington bureau chief, says that the most attractive part about the data mining system they have developed to use that information is “its ability to monitor the conversations that are taking place independently, rather than having to frame the issues by asking direct questions. ‘We’re waiting to see what they’re going to say.’” (See “Nation’s Political Pulse, Taken Using Net Chatter,” in today’s New York Times.)
This promises to be free of the usual problems of sampling bias or unconscious preconceptions – a perennial dream of those who seek more accurate information about the behavior of voters or consumers or any other sector of the general public.
But it is not free of it own problems. Will the public express itself in ways that fit what pollsters need to hear or can even understand? Can sarcasm and irony be filtered out of the message? Or will the system backfire and exaggerate dangerous or dysfunctional trends?
These trends have been explored recently on a new website in the UK, Mindful Money, focused on financial behavior. Their research collaborator, Xiao-Jun Zeng at the University of Manchester, trying to move beyond the usual forms of information about market trends, realized “that something was missing from the parameters, namely moods, psychology. While news, data mining is quite easy, capturing mood information is more difficult.” And, so, they turned to Twitter, and “uncovered an accuracy of 87.6% in predicting the daily up and down changes in the closing values of the Dow.” (See “Tweeting Moods Key to Stock Market Forecasting.”)
While this research is still in its beginning phase, the correlation is astonishing. It promises to offer a perfectly legal alternative to the reliability that, previously, traders could only get through insider trading or stock manipulation.
But this, too, is not without its dangers. As Stephen Fitzpatrick, at Mindful Money says: “What we currently have in the online financial space is a nervous social system and there is a danger that twitter can increase the extent to which markets are correlated which is precisely what happened during the crash.”
So even if we can learn to read the social unconscious accurately, even if new technologies can mine Twitter to give us an unprecedented look into crowd behavior as it is happening, we still have to think about what we are doing with the information we get.
A better “nervous system” still requires a brain, and it demands that we be mindful.