Data that requires more processing resources to produce value than are available within the constraints of the current business problem.

1. EVOLUTION
OF BIG DATA
Shilpa krishna
Research Scholar

3. PATH TO BIG DATA

4. DATA MANAGEMENT
PROFESSIONAL

5. DATA ENGINEER

6. BUSINESS ANALYST

7. MACHINE LEARNING
RESEARCHER

8. DATA ORIENTED PROFESSIONAL

9. BIG DATA PIPELINE IN
DEPTH

10. Components of a big data pipeline are :
• The messaging system
• Message distribution support to various nodes for
further data processing.
• Data analysis system to derive decisions from data.
• Data storage system to store results and related
information.
• Data representation and reporting tools and alerts
system.

11. Important parameters that a big data pipeline
system must have -
• Compatible with big data
• Low latency
• Scalability
• Flexibility
• Economic
• A diversity that means it can handle various use cases.

12. In a Big Data pipeline system, the
two core processes are -
• The messaging system
• The data ingestion process

13. LAMDA ARCHITECTURE

14. Batch Layer
Serving Layer
Incoming
data Queries
Real time Layer
Data Storage Layer
Batch
Engine
Real Time
Engine
Serving
Backend
Historical data Result data

15. KAPPA ARCHITECTURE

16. Serving Layer
Real time Layer Query
Streaming data
Query
Stream
process
Real time
views
Real time
views

17. BIG DATA
VISUALIZATION

18. • Review large amounts of data
• Spot trends
• Identify correlations and unexpected relationships
• Present the data to others
Big data visualisation techniques offer a fast
and effective way to :

19. Various Big Data Visualisation examples
include :
1. Linear
2. 2D/Planar/Geospatial
3. 3D/Volumetric
4. Temporal
5. Multidimensional
6. Tree/hierarchical

20. BIG DATA PRIVACY

21. Big data privacy risks
1. Data breaches
2. Data brokerage
3. Data discrimination

22. Best practices for maintaining the
privacy of big data -
1. Employ real-time Monitoring
2. Implement homomorphic encryption
3. Avoid collecting too much data
4. Prevent internal threats

23. THANK YOU FOR WATCHING