Data Engineer's Lunch #63: Building a Cryptocurrency Data Catalogue

•

0 j'aime•129 vues

In Data Engineer’s Lunch #63, Travis Collins, founder of the open source project DataPM, will present DataPM, how to get access to cryptocurrency, and blockchain data. This is part 1 of a series with Decodable on processing real-time crypto transactions fed by DataPM. Accompanying YouTube: https://youtu.be/_YltwetuPK0 Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Data Engineer’s Lunch Weekly at 12 PM EST Every Monday: https://www.meetup.com/Data-Wranglers-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/ Join The Anant Team: https://www.careers.anant.us

Données & analyses

datapm
Build Communities Around Data That Matters

DataPM’s mission is to organize the world’s
data and make it universally accessible and
useful.

Open Source
https://github.com/big-armor/datapm

For Collaborating on Data Across Teams/Orgs
Used by Data Engineers, Analysts, Scientists,
and Researchers

Data Package Manager
DataPM.io -> Data
GitHub.com -> Code
PyPi.org -> Python
Maven.org -> Java
npmjs.com -> Javascript

Schema Change Management
Know when things break!

Host Your Own DataPM Registry
https://datapm.io/docs/private-registry/

You probably shouldn’t
put your data in DataPM
Keep using your existing publishing platforms,
add DataPM for easy distribution

Typical “datapm fetch” Flow
Source
Data
Repository
DataPM Registry
DataPM Client
Sink
Data
Repository
1
2
3

Future Roadmap
GUI Client
Data Governance
Querying
Buy Button

Next Week Part 2: Decodable
Stream Processing with DataPM
https://decodable.co

Join Our Slack Community
https://datapm.io

Contenu connexe

Similaire à Data Engineer's Lunch #63: Building a Cryptocurrency Data Catalogue

We can make a data mining to get the prediction about the future data, which is mined from an old data especially Big data using a machine learning algorithms based on two clusters. One is the intrinsic for managing the file system of Big data, which is called Hadoop. The other is essentially to make fast analysis of Big data which is called Apache Spark. In order to achieve this purpose we will use R based on Rstudio or Scala based on Zeppelin.

Analyzing Big data in R and Scala using Apache Spark 17-7-19

Ahmed Elsayed

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010

Bhupesh Bansal

Hadoop and Voldemort @ LinkedIn

Hadoop User Group

Enterprise guide to building a Data Mesh

Sion Smith

Organizing the Data Chaos of Scientists

Andreas Schreiber

Robin Bloor and Teradata Live Webcast on April 22, 2014 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=2e69345c0a6a4e5a8de6fc72652e3bc6 Can you replace the data warehouse with Hadoop? Is Hadoop an ideal ETL subsystem? And what is the real magic of Hadoop? Everyone is looking to capitalize on the insights that lie in the vast pools of big data. Generating the value of that data relies heavily on several factors, especially choosing the right solution for the right context. With so many options out there, how do organizations best integrate these new big data solutions with the existing data warehouse environment? Register for this episode of The Briefing Room to hear veteran analyst Dr. Robin Bloor as he explains where Hadoop fits into the information ecosystem. He’ll be briefed by Dan Graham of Teradata, who will offer perspective on how Hadoop can play a critical role in the analytic architecture. Bloor and Graham will interactively discuss big data in the big picture of the data center and will also seek to dispel several common misconceptions about Hadoop. Visit InsideAnlaysis.com for more information.

Hadoop and the Data Warehouse: Point/Counter Point

Inside Analysis

DataFinder: A Python Application for Scientific Data Management

Andreas Schreiber

ScyllaDB, along side some of the other major distributed real-time technologies gives businesses a unique opportunity to achieve enterprise consciousness - a business platform that delivers data to the people that need when they need it any time, anywhere. This talk covers how modern tools in the open data platform can help companies synchronize data across their applications using open source tools and technologies and more modern low-code ETL/ReverseETL tools. Topics: - Business Platform Challenges - What Enterprise Consciousness Solves - How ScyllaDB Empowers Enterprise Consciousness - What can ScyllaDB do for Big Companies - What can ScyllaDB do for smaller companies.

Developing Enterprise Consciousness: Building Modern Open Data Platforms

ScyllaDB

Starting with outlining the history of conventional version control before diving into explaining QoDs (Quantitative Oriented Developers) and the unique problems their ML systems pose from an operations perspective (MLOps). With the only status quo solutions being proprietary in-house pipelines (exclusive to Uber, Google, Facebook) and manual tracking/fragile "glue" code for everyone else. Datmo works to solve this issue by empowering QoDs in two ways: making MLOps manageable and simple (rather than completely abstracted away) as well as reducing the amount of glue code so to ensure more robust end-to-end pipelines. This goes through a simple example of using Datmo with an Iris classification dataset. Later workshops will expand to show how Datmo can work with other data pipelining tools.

Version Control in Machine Learning + AI (Stanford)

Anand Sampat

Building Your Own Facebook Real Time Analytics System with Cassandra and GigaSpaces. Facebook's real time analytics system is a good reference for those looking to build their real time analytics system for big data. The first part covers the lessons from Facebook's experience and the reason they chose HBase over Cassandra. In the second part of the session, we learn how we can build our own Real Time Analytics system, achieve better performance, gain real business insights, and business analytics on our big data, and make the deployment and scaling significantly simpler using the new version of Cassandra and GigaSpaces Cloudify.

Big Data Real Time Analytics - A Facebook Case Study

Nati Shalom

Agile, Continous Intergration, DevOps, Big data are not longer buzzwords but part of the day today process of everyone working in software development and delivery. To cope with applications that need to be deployed in production almost the same moment they were created, software development has changed, impacting the way of working for everyone in the team. In this talk, Roland will discuss the challenges performance testers face with Big Data applications and how Architecture, Agile, Continous Intergration and DevOps come together to create solutions.

Testistanbul 2016 - Keynote: "Performance Testing of Big Data" by Roland Leusden

Turkish Testing Board

Taboola Road To Scale With Apache Spark

tsliwowicz

Data orchestration is the lifeblood of any successful data analytics solution. Take a deep dive into Azure Data Factory's data movement and transformation activities, particularly its integration with Azure's Big Data PaaS offerings such as HDInsight, SQL Data warehouse, Data Lake, and AzureML. Participants will learn how to design, build and manage big data orchestration pipelines using Azure Data Factory and how it stacks up against similar Big Data orchestration tools such as Apache Oozie. Video of presentation: https://channel9.msdn.com/Events/Ignite/Australia-2017/DA332

Microsoft Ignite AU 2017 - Orchestrating Big Data Pipelines with Azure Data F...

Lace Lofranco

Greenplum versus redshift and actian vectorwise comparison

Dr. Syed Hassan Amin

Hadoop basics

Antonio Silveira

Real time analytics

Leandro Totino Pereira

Big Data and Hadoop

MaulikLakhani

MoPub, a Twitter company, provides monetization solutions for mobile app publishers and developers around the globe. MoPub receives over 33 Billion ad requests per day generating over 200TB of raw logs every day. We built MoPub Analytics as the analytics platform, using Druid + Imply for our end users who are Publishers, Demand side partners and Internal users. We will talk about the architecture of the analytics platform, our Druid cluster setup, hardware choices, monitoring, use cases, limiting factors, challenges with lookups and solutions we used. Watch video:https://imply.io/virtual-druid-summit/analytics-over-terabytes-of-data-at-twitter-apache-druid

Analytics over Terabytes of Data at Twitter

Imply

Watch Pablo's session from Fast Data Strategy on-demand here: https://goo.gl/1aEBo8 The tide is changing for analytics architectures. Traditional approaches, from the data warehouse to the data lake, implicitly assume that all relevant data can be stored in a single, centralized repository. But this approach is slow and expensive, and sometimes not even feasible, because some data sources are too big to be replicated, and data is often too distributed such as those found in cloud data sources to make a “full centralization” strategy successful. Watch this session to learn more about: • Modern data architectures • Why logical architectures are the best option when integrating big data • How Denodo’s parallel in-memory capabilities with dynamic query optimization redefine analytics architectures

Denodo Platform 7.0: Redefine Analytics with In-Memory Parallel Processing an...

Denodo

Empowering Transformational Science

Chelle Gentemann

Similaire à Data Engineer's Lunch #63: Building a Cryptocurrency Data Catalogue (20)

Analyzing Big data in R and Scala using Apache Spark 17-7-19

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010

Hadoop and Voldemort @ LinkedIn

Enterprise guide to building a Data Mesh

Organizing the Data Chaos of Scientists

Hadoop and the Data Warehouse: Point/Counter Point

DataFinder: A Python Application for Scientific Data Management

Developing Enterprise Consciousness: Building Modern Open Data Platforms

Version Control in Machine Learning + AI (Stanford)

Big Data Real Time Analytics - A Facebook Case Study

Testistanbul 2016 - Keynote: "Performance Testing of Big Data" by Roland Leusden

Taboola Road To Scale With Apache Spark

Microsoft Ignite AU 2017 - Orchestrating Big Data Pipelines with Azure Data F...

Greenplum versus redshift and actian vectorwise comparison

Hadoop basics

Real time analytics

Big Data and Hadoop

Analytics over Terabytes of Data at Twitter

Denodo Platform 7.0: Redefine Analytics with In-Memory Parallel Processing an...

Empowering Transformational Science

Plus de Anant Corporation

QLoRA Fine-Tuning on Cassandra Link Data Set (1/2) Cassandra Lunch 137

Data Engineer's Lunch #63: Building a Cryptocurrency Data Catalogue

Recommandé

Recommandé

Contenu connexe

Similaire à Data Engineer's Lunch #63: Building a Cryptocurrency Data Catalogue

Similaire à Data Engineer's Lunch #63: Building a Cryptocurrency Data Catalogue (20)

Plus de Anant Corporation

Plus de Anant Corporation (20)

Dernier

Dernier (20)

Data Engineer's Lunch #63: Building a Cryptocurrency Data Catalogue