ecent research has found that “big data” is no longer a buzzword or fad. Some
62.5% of US companies are currently implementing or expanding their big data
projects. Another 30% are in the planning stage, getting ready to adopt it within
the coming 12 months. In 2016, this was a $130 billion market and it’s expected
to nearly double by 2020. All this is attributed to that increasing availability of
data, a new generation of technology and a cultural shift toward data-driven decision-making.
There is no formal definition of the term “big data.” Everyone seems to have their own definition. Generally, big data refers to data sets that are so voluminous and complex that traditional data processing methods, systems and software are not capable of dealing with them. The
problem is their volume or quantity, their velocity, i.e. the speed at which they are generated
and transferred, often in real-time, and their variety, consisting of text, images, audio, video, etc.
Also, these data sets are often unstructured—they’re not organized in any logical manner, and
are on the order of terabytes (1012), or petabytes (1015) or even exabytes (1018). Just to give you
some sort of a yardstick, eBay, Twitter and Google process about 100 petabytes (104,857,600
gigabytes) per day while Facebook does closer to 600 petabytes per day. In terms of retail enterprises, Wal-Mart handles about 2. 5 petabytes per hour. And this is just to gather and store the
data, in other words to create the dataset.
New tools to the rescue
The good news is that technological experts have begun to come up with new tools to solve
the speed and storage challenges, while the rest of us are ramping up our efforts to understand
and utilize them. As for searching and analyzing the contents of our newly acquired data, that’s
where our friends in the software branch of the data sciences have developed a whole new set
of tools and techniques for capturing, cleaning, searching and analyzing large data sets.
There’s a whole family of software programs devoted to data storage and management. They
are designed to provide an infrastructure on which you can run all your other analytics tools as
Harnessing the power of ‘big data’
BY EV COLLIER
34 Marine Electronics Journal May/June 2018
and circulating so
much data today that not only has the
explosion overloaded our wireless
technologies, it has begun to challenge our ability to capture, transfer,
store, search and extract meaning
from it. Just a couple of years back,
IBM reported that we were creating
some 2. 5 quintillion bytes annually and
that 90% of the world’s data was created in just two years! Large amounts
of data are coming in unstructured
and at such a high rate of speed that it
represents a real problem for all types
and sizes of businesses and organizations. But finding solutions is important because the data contains a gold
mine of information—for large and
small businesses alike.