subject: Can Twitter Predict the Future? [print this page] Can Twitter Predict the Future? Can Twitter Predict the Future?
For well over a century writers have expressed their visions of the future, and those visions have attracted fans in such great numbers that an entire genre has grown up around them: Science Fiction. In the 19th Century, Jules Verne and H.G. Wells wrote of robotics, world wars, warfare tactics including aerial bombing, use of tanks and chemical weapons, and nuclear power. In 1945, Arthur C. Clarke foreshadowed telecommuting, mobile phones and the internet. What is astounding about these predictions is that they came true.
Mathematical Modeling
Having a foreknowledge of events to come puts the "knower" in a very powerful position. Who wouldn't want to know the outcome of sporting events, or which stocks were about to take off? From a national security standpoint, knowing your adversary's plans what they intend to do and when they intend to do it is a defensive weapon more powerful than nuclear bombs and dtente.
Mathematical models have been created to predict the stock market, the spread of diseases, and presidential elections. Professor Bruce Bueno de Mesquita claims to have spent 25 years perfecting a computer model that can predict the outcome of any global conflict; provided, he says, that the "basic input is accurate." In the words of Shakespeare's Hamlet: ""aye, there's the rub." To make reasonably accurate predictions, quality input must be provided because garbage in = garbage out.
The CIA and Recorded Future
The CIA is especially interested in identifying threats to US security, and they have found an information source right out in the open Twitter. Former CIA Director Michael Hayden told a conference in 2008: "Secret information isn't always the brass ring in our professionthere's a real satisfaction in solving a problem or answering a tough question with information that someone was dumb enough to leave out in the open."
The CIA and Google have invested in a start-up called Recorded Future with the hope that they can provide the accurate raw data needed to predict the future. Recorded Future scans tens of thousands of websites, Twitter accounts, and blogs and notes the people, places, and activities they mention. They then examine when and where the events happened ("spatial and temporal analysis") and the tone of the document ("sentiment analysis"). An artificial intelligence algorithm looks for connections between people and events.
Recorded Future claims its temporal analytics engine "goes beyond search" by "looking at the invisible links between documents that talk about the same, or related, entities and events." The purpose is to determine for each incident who was involved, where it happened and when an incident might occur. Recorded Future then plots the online chatter, showing the momentum for any given event through a "momentum curve."
Recorded Future has had some public successes. In March of this year, Israeli President Shimon Peres accused the terrorist group Hezbollah of having "scud-like" weapons. Recorded Future scanned statements made by Hezbollah leader Hassan Nasrallah and found evidence to corroborate the accusation.
Perhaps the future is inevitable once events gain a certain momentum. If that is the case, the Arthur C. Clarke prediction of instant person-to-person contact anywhere in the world was inevitable once Comsat was a reality. Twitter, Yahoo and others already have in place a "momentum measuring device" in their "Trending" measurement. Recorded Future takes Twitter Trending to the next level and into the future.
