Big Data’s Big Challenges

Big Data’s Big Challenges

The term “big data”, originally described by Doug Laney of Mega Group, is widely accepted to represent data sets where the three Vs—volume, velocity and variety—present specific challenges in managing these data sets.

Velocity refers to the speed with which data is created. Every 60 seconds, 48 hours of video are uploaded to YouTube, Twitter users send 100,000 tweets and Instagram users share 3,600 photos — that’s data velocity.

Variety is the result of all this activity. Obviously it’s not limited to these social media platforms, but it can include all the various ways we consume information and the formats by which we take them in or disseminate them. Additionally, organizations are collecting and storing more data so that they can get better insight into their business and customers in order to fully optimize productivity, customer engagement and revenue opportunities. And consider all of the “smart”, industry-specific technology used like electric grids or city traffic systems. All of this information demands innovative ways to access it  and store it. Old school databases aren’t able to adequately handle and process the information which has given way to powerful, robust new technologies.

Volume represents the size of all this data and is more difficult to define because it can constantly move and change.

Although big data is revolutionizing the way we live, conduct business and forecast for our futures, the abundance of information it provides that we can use to enhance everything from healthcare to education to advertising is not without its issues. A recent report in Washington Technology,  outlines some of the overall biggest challenges we face with big data:

• Data acquisition


• Storage


• Processing


• Data transport and dissemination


• Data management and curation


• Archiving


• Security


• Workforce with specialized skills


• Cost of all of the above


 

For companies specifically, the challenges center around the fact that the way we live and technological advancements in general require that this data be processed faster than ever before so that we can respond faster, and in many cases, be able to respond in real-time. Moreover, this information comes from  a wide array of channels. To mange it all, organizations must have:

Solid Technical Infrastructure

Big data requires a solid technical infrastructure including storage, bandwidth, CPU, etc. Organizations have to be prepared for the variation in workload it requires. The Cloud is certainly suited to accommodate big data needs — the challenge in this case is primarily associated with finding the right provider with the right, cost-effective solutions for your needs.

Strong Applications

Finding the right applications to support your organization’s big data needs can also be a challenge. You will need to consider integration with your existing tools, learning curves and having a powerful enough operating environment in which to manage all of the various pieces. Many of the applications currently on the market are new and have many kinks to be worked out.

The Right Knowledge-base and Skill sets

In addition to having the right application, you will need the right team to manage your data. You need people who understand the business side of big data, who know the right questions to ask vendors and understand and can communicate your needs. You also will have to identify the right technical staff to manage the infrastructure and applications. And you need “data scientists” that can understand the numbers and algorithms behind the data. This team also has to have the right attitude for experimentation and research that is tied to business objectives.

Your organization must also find a way for the big data team to work effectively with organizational data that is often inconsistent and siloed, making data difficult to capture and analyze.

Clear Valuations

Another challenge for companies is being able to ascribe clear value to outcomes so that you can prioritize activities. This is of extreme importance so that the investment and experimentation yields a return.

When your organization is able to build a strong infrastructure for big data manned by knowledgeable, flexible experts and skilled technicians who can help your organization see what information is valuable in reaching your business goals, you’ll be on your way to getting the big results you want from your big data.