subject: Government Big Data Market Creating Big Opportunities For Business [print this page] Big Data is not, strictly speaking, a technology or even a market. It's a new opportunity to derive insights and generate benefits never possible before, from improving medical care to preventing terrorism, from reducing fraud to stopping traffic jams. Government agencies, along with enterprises in the private sector, are being inundated with information, in large part "unstructured data" not already in a spread sheet or formatted database, such as high-definition video feeds from drones and comments on social networking sites. 1.8 zetabytes of information were created globally in 2011, or the equivalent of 200 billion, two-hour HD movies, and that amount is expected to double every year (TechAmerica).
So what is Big Data? Various definitions have been proposed. A common one: "Big data is when the size of the data itself becomes part of the problem." IBM finds its definition in the characteristics of the data itself, often called the "3 V's.": "Volume (Terabytes -> Zetabytes), Variety (Structured -> Semi-structured -> Unstructured), and Velocity (Batch -> Streaming Data)." Others have proposed adding a fourth "V" to this equation to reflect a key challenge of social media: Veracity. One NIST expert proposed this definition: "Big data is where the data volume, acquisition velocity, or data representation limits the ability to perform effective analysis using traditional relational approaches or requires the use of significant horizontal scaling for efficient processing." One observer simply says: "What is Big Data? You know it when you see it .."
The value of Big Data was seen years ago by some government agencies. The IRS built sophisticated systems to detect tax cheating. CMS -- the Centers for Medicare & Medicaid Services, which administers Medicare and other government health programs) has 4,000 employees to handle more government spending than the Department of Defense. It has long used data mining to detect and prevent fraud. The already deep investment in technology by the Intelligence Community accelerated dramatically after 9/11; Big Data applications at the NSA, for example, exceed the best of the private sector in some areas. But many other agencies are behind. While government workers say they're swamped with information, including unstructured data, just 40% say they're using that data for strategic decisions, according to a 2012 study.
President Obama has launched a major push to get agencies into the game, adding $200 million for R&D to the estimated 160+ Big Data projects now underway by federal organizations. More and more agencies are also being goaded into action by the GPRA (The Government Performance and Results Act), updated in 2011, which requires agencies to set goals, measure results and report progress.
Vendor-addressable IT spending on federal Big Data initiatives is expected to grow from under $5 billion in FY12 to $7.2 billion by FY17, representing a compound annual growth rate (CAGR) of 8.2% (Deltek). This spending includes areas such as analytics, storage, networking and services, and is related to yet more spending on "the Cloud." These trends and parallel ones in the private sector are also driving M&A (IBM alone has spent billions in recent years buying analytics companies) and venture investment.
To be sure, there are constraints in the growth of the market. A fundamental one is the learning curve for government managers to understand the Big Data opportunity, create plans to harness it and get it into budgets. This process should drive demand for expert consulting services to government agencies over the near term. Other challenges include decelerating federal spending; inadequate human capital (McKinsey claims America will be short 190,000 data scientists by 2019) and a sclerotic contracting system. Public concern over privacy and civil liberties remains an important factor -- there are already an estimated 40 privacy-related laws.
Then there are technology challenges raised by Big Data, especially as social networking and agency appetite for unstructured data such as reconnaissance video grows. Networking infrastructure will need to move massive amounts of data to disparate users in scattered locations. Advances in cloud computing systems and High Performance Computing (HPC) hardware will enable the analytics that produce more and better insights.
That said, Big Data will likely remain a growth area in flat or declining federal IT budgets. That's because Big Data holds out one of the few solutions potentially capable of increasing government's efficiency and effectiveness. In short, many Big Data projects will offer the promise of paying for themselves in productivity gained or lives saved.
