It’s hard to avoid the term big data if you work in B2B, but ask anyone who isn’t an analytics geek what it means and you’ll get blank stares and “you know, a lot of data”. Like most buzzwords, big data has become so generic a phrase it’s meaningless. What do people mean when they say big data and what do you need to know? Here’s a primer for non-technical B2B executives and entrepreneurs.
Where did the term big data come from? This practically archaeological history from Forbes goes into the background of the phrase. As for the usage today: in a nutshell, there are dozens of different digital platforms in our lives that capture and generate an ever-increasing amount of structured and unstructured data. So much data that old storage and processing was too unwieldy; new ways to manage it had to be invented to make it usable. So much data was available then that cross-referencing, regression analysis, and other manipulation started to generate enormous returns. Individually datasets are useful; combined, or processed and associated differently, they can reveal extremely powerful, sometimes unnervingly prescient, patterns and insights.
How was it popularized? IBM has been working on managing big data for decades, and the IT community at large for years. The concept caught the attention of a wider business audience and to an extent the general public in 2012 when the New York Times published a story about big data and consumer shopping habits containing an anecdote about big data predicting a pregnant teen before her father knew about it. This tapped into something: an artificial intelligence drawing an accurate conclusion about something very human before an infinitely more qualified human could. It captured the imagination and the term soon was ubiquitous.
What does IBM’s Watson have to do with big data? Watson is both a product of big data and a platform on which it can be processed and now, delivered as a service. Watson is the first generation in production of what is known as cognitive computing. Watson can now ‘learn’ about an industry, then consume a company’s data and turn that into everything from customer-facing services like website Q&A and first-level technical support, to diagnostics.
What are structured and unstructured data? Basically, structured data comes with a schema or logic to the data built in; its properties are known. Unstructured data may have an unknown or unquantified number of variables or come in unpredictable formats. An example of structured data would be a symphony, and unstructured data the conversations of a pod of dolphins.
What is Hadoop? Like any other innovation, big data has an ecosystem of technology and platforms that support it. Hadoop has become so big Forrester says economics dictate it is no longer optional. It is an open source software (Apache) framework specifically developed for distributed processing of big data. Other big data friendly technologies include SAP’s HANA, capable of processing enormous amounts of data from different sources simultaneously.
Is AI big data? AI is a type of application of big data. It would be impossible to produce AI without the input of vast amounts of structured and unstructured data. AI could even be defined as the ability of systems to seamlessly process and act upon both structured and unstructured data.
What do B2B managers need to know? Big data type applications are becoming accessible and affordable for small businesses, especially through applications like Watson for small business and virtual agents such as Intelliresponse, recently acquired by 7. If you offer these services to customers, big data may already be making it easier and more affordable to do so.
What are the implications for job seekers and hiring managers? Data science is booming and qualified data professionals increasingly difficult to find. The Digital Analytics Association offers courses, education and more for an annual membership fee.