Top reasons why 85% of big data projects are still a failure. This is based upon Gartner's recent study and various case studies shared over Internet.

1. Top Reasons why 85% of
BigData Projects are Still a
Failure

2. Introduction
We all know BigData revolution is Open Source and has huge cost-
benefit ratio over respective traditional platforms. What we also
know per Gartner’s recent Study, that there is about 85% failure
rate in deploying BigData platforms across the board despite
Industry vertical. This is true even with most of the top B2B/B2C,
Social/Professional Media, Cloud Storage, Streaming, Networking,
Retail, Products Professional Services & other firms having Data/BI
domain.

3. Introduction...contd
These firms are not even successful in adopting BigData in their own
Production space, then how can they make their internal or external
customers successful? If, we try to learn facts, we could find that most of
these Firms are letting their engineering resources to experiment on their
Customer’s space to get better, and such experiments costs those
customers huge time and cost without them knowing.
Advantage of such scenarios are often passed on to BigData Products &
Consulting firms, who also sells their Professional services.
Let’s dig deeper on the reasons why and maybe you would be surprised...

4. 85% of BigData Projects are Failing...Live Case
Recently, I came in contact with a highly published IT Cloud Firm out of the west
coast. After moving to the Big Data platform a couple years ago, their marketing
staff are still very frustrated as they received their desired analytics with over a
day's worth of latency. Moreover, their data quality is so poor, they are resorted to
making inferences based on financial book references and come up with some
range on justifiable numbers to work on. Their so called “data” team comprises of
all great python/sql developers but with no Architects and no proper QA
resources. Yes, they have great Data Scientists too. But, end result is almost
close to 3 on the scale of 1 to 10 (1 being trash / 10 being excellent). Doesn’t it
seem like that their Management lacks big picture, vision, prior “successful”
venture with “Data”, and as well as incompetence in hiring right “Architect” and
“QA” resources? These are the only factors which can be counted towards this
use case.

5. 85% BigData Projects are Failing...Live Case...contd
Another live case- Let’s talk about Single Source of Truth for KPIs. One of the
topmost IT Cloud online retail firm out of the west coast has failed miserably in
providing Single Source of Truth on most of their high-profile KPIs to their
Business partners despite adopting BigData. Each version of their BigData
platforms are so siloed, that business users chaired next to each other, receives
different KPIs numbers of the previous business day. Their so called “Data
Pipeline” concept is entirely broken. Some of their siloed BigData houses have
latencies more than 15+ hours. None of their team members have basic Data
skills, and greatly lacks data processing knowledge, yet they are apparently
proficient in python, java, and SQL. And, yes, they have so called Data Scientists
as well. Overall, their Management lacks basic data foundation principals and
vision on how they should be driving such effort.

6. So, Why are they failing….?
I researched several case studies over Internet including Gartner too. If, I
summarize all of them, they would look like below. If, you’re one of the victims,
you’re sure to find your own issues down below:
1) Herd Mentality (which is so true).
2) Challenges with the problems presented by legacy technologies in
Enterprise.
3) Pre-existing Corporate Biases.
4) Siloed Data (a huge bottleneck)
5) Management resistance, which is influenced by involved IT Consulting
Partner's for latter sake.

7. So, Why are they failing….?...contd...
6) Management Bureaucracy - Choosing the wrong Use Cases to please top
brass and then hung up on scaling further.
7) Failed to convince Management on Enterprise wide strategy.
8) Wrong Management Hires (truly truly true)
9) Lacks basic Data knowledge and completely unaware of what BigData is
there for.
10)Poor Communication among all layers.
11)Management Vision and their own Failures affecting BigData initiatives for
Enterprise.

8. So, Why are they failing….?...contd...
12)Lacks in building Trust in Data Products and algorithms in consulting with
Business Partners (huge factor).
13)Lack of Skilled “Architect” and “QA” (high potential factor)
14)Using the wrong tools just because influential Data Scientists/Data
Engineers or influential Consulting Partner wanted to enhance their skills
and experience.
15)Of-course that all being said, lacking poor planning by Management and
hence driving entire team on the wrong path or else forcing team members
with bad consequences.

9. In nutshell...Analysis Outcome….
Multiple Case Studies into such big data failures have revealed that many of the
challenges occur from human issues rather than technical failures. There are rarely
any lack of skills or experience on Technical front. It’s mostly Management which
blames their technical resources or their lack of, for their own failures. To summarize
top reasons, 99% of such failures can be bottomlined to:
1) Top Management (probably including C-Level Execs) inefficiencies in lacking big
picture, finite strategy and direction. They often creates broken Orgs on purpose.
2) Middle Management - who lacks knowledge of basic Data Foundation &
Principles, and they are the ones who have never been successful in their past
career even with traditional approaches. But, they survived due to obvious
reasons. :) and got hired by Top Management. And, they hire lessor equivalent
staff on purpose, and afraid to hire right skills for obvious reasons. :)

10. Such Management folks ends up hiring BigData products’ firms and their
professional services to get things in order. That results going over initially thought
Budget and all projected margins turns red into their accounting books. It ruins
entire cost-benefit ratio of BigData adoption.
Some of these IT Management folks involved in such decision makings came from
zero background of any successful “Data” ventures ever, or came from Operations
team, or from Business portfolio, or from Consulting firm related to top C-level
execs, or even from fashion industry or retail supply chain shipping background, or
a relative/friend of C-level exec, or even a project or even program manager
etc….:)
In nutshell...Analysis Outcome….contd...

11. Every Firm adopting BigData should carefully evaluate who they are hiring in the
Management, and scrutinize their past successes. Don’t just look for if they can
code in python, java or in SQL or how well connected they might be. Once right
person is out there, right skills and experiences on board would transparently
hired & followed on with success.
On resourcing aspects, ETL developer or IT Manager or Data Science staff can’t
be Architect and QA must not befriend to Architect and Developers. That’s how
one can build and deliver BigData platform successfully.
So, what’s in nutshell...Analysis Outcome….contd...

12. Thoughts?