Building a big data warehouse

•

3 j'aime•903 vues

Let's face it: data warehousing in the traditional sense has been tedious, lacking agility, and slow. Often designed and built around a static set of up front questions, they are basically a big database tailored to the application of populating dashboards. The fact that the volume of data that we need to deal with recently exploded by several order of magnitude isn't improving the situation. It's no surprise that we see a new class of data warehousing setups emerge, using big data technologies en NoSQL stores. A nice side effect is that these solutions are usually not only the domain of the BI crowd, but can also be developer friendly and allow development of more data driven apps. In this session I will present about experiences using Hadoop and other tools from the Hadoop ecosystem, such as Hive, Pig and bare MapReduce, to handle data that grows tens of GBs per day. We create a system where data is captured, stored and made available to different users and use cases, ranging from end users that write SQL queries to software developers that access the underlying data to create data driven products. I will cover topics like ETL, querying, development and deployment and reporting; using a fully open source stack, of course.

Technologie Business

GoDataDriven
PROUDLY PART OF THE XEBIA GROUP
@mids106
jorisbontje@godatadriven.com
Building a Big Data
Warehouse
Joris Bontje
Big Data Hacker

About Me
Big Data Hacker
Data Driven Solution Architect
HadoopTrainer

About GoDataDriven

Data Warehouse
Evolution

http://en.wikipedia.org/wiki/Data_warehouse
In computing, a data warehouse is a
database used for reporting and data
analysis.

Database Architecture (1.0)
Products)
Customers)
Orders)
Inventory)
Sales)DB)

Analytical Database (2.0)
Sales&
Inventory&Customers&
Products&
Orders&

Basic DWH Architecture
TX#
DB#
Analy+cal#
DB#
BI#ETL

Data Marts
TX#
DB#
DW#
Sales#
Mktg#
Prch#
BI#

Multiple Data-Sources
other&
Files&
TX&
DB&
DW&
Sales&
Mktg&
Prch&
BI&

Operational Data Store
DW#ODS#
other#
Files#
TX#
DB# Sales#
Mktg#
Prch#
BI#

Hadoop

GoDataDriven
No Hadoop
DW#ODS#
other#
Files#
TX#
DB# Sales#
Mktg#
Prch#
BI#

GoDataDriven
ETL Engine
other&
Files&
TX&
DB& Sales&
Mktg&
Prch&
DW
BI&

GoDataDriven
Tiered Data Warehouse
other&
Files&
TX&
DB& Sales&
Mktg&
Prch&
BI&

GoDataDriven
Analytical Query Engine
other&
Files&
TX&
DB&
BI&

Tools

Tools

Tools Applied

Tools Applied

Considerations

Considerations
Big Data is dirty
Automate everything
Monitoring and QA become the same thing

My Past Trends
Big Data Forum 2012

My PastTrends
Cloud / On-demand

My PastTrends
Hadoop Hardware

My PastTrends
Batch → Real-Time

New Trends
XebiCon 2013

Trends
Impala
Open Source, Real-time Query engine
for Hadoop

Trends
Defacto standard for Hadoop metadata

Simple Database Architecture
Products)
Customers)
Orders)
Inventory)
Sales)DB)

The future?
Products)
Customers)
Orders)
Inventory)
Sales)

GoDataDriven
We’re hiring / Questions? / Thank you!
@mids106
jorisbontje@godatadriven.com
Joris Bontje
Big Data Hacker

Contenu connexe

En vedette

Deep Learning

An introduction to Machine Learning

An introduction to Machine Learning

An introduction to Machine Learning

It’s long ago, approx. 30 years, since AI was not only a topic for Science-Fiction writers, but also a major research field surrounded with huge hopes and investments. But the over-inflated expectations ended in a subsequent crash and followed by a period of absent funding and interest – the so-called AI winter. However, the last 3 years changed everything – again. Deep learning, a machine learning technique inspired by the human brain, successfully crushed one benchmark after another and tech companies, like Google, Facebook and Microsoft, started to invest billions in AI research. “The pace of progress in artificial general intelligence is incredible fast” (Elon Musk – CEO Tesla & SpaceX) leading to an AI that “would be either the best or the worst thing ever to happen to humanity” (Stephen Hawking – Physicist). What sparked this new Hype? How is Deep Learning different from previous approaches? Are the advancing AI technologies really a threat for humanity? Let’s look behind the curtain and unravel the reality. This talk will explore why Sundar Pichai (CEO Google) recently announced that “machine learning is a core transformative way by which Google is rethinking everything they are doing” and explain why "Deep Learning is probably one of the most exciting things that is happening in the computer industry” (Jen-Hsun Huang – CEO NVIDIA). Either a new AI “winter is coming” (Ned Stark – House Stark) or this new wave of innovation might turn out as the “last invention humans ever need to make” (Nick Bostrom – AI Philosoph). Or maybe it’s just another great technology helping humans to achieve more.

Deep learning - A Visual Introduction

Deep learning - A Visual Introduction

Deep learning - A Visual Introduction

Machine Learning for Dummies

Machine Learning for Dummies

Machine Learning for Dummies

Venkata Reddy Konasani

Introduction to Machine Learning

Introduction to Machine Learning

Introduction to Machine Learning

Deep Learning in Computer Vision

Deep Learning in Computer Vision

Deep Learning in Computer Vision

Deep learning - Conceptual understanding and applications

Deep learning - Conceptual understanding and applications

Deep learning - Conceptual understanding and applications

Amazon Deep Learning

Amazon Deep Learning

Amazon Deep Learning

Amanda Mackay (she/her)

Deep Learning and the state of AI / 2016

Deep Learning and the state of AI / 2016

Deep Learning and the state of AI / 2016

Grigory Sapunov

Deep neural networks

Deep neural networks

Deep neural networks

Introduction to Big Data/Machine Learning

Introduction to Big Data/Machine Learning

Introduction to Big Data/Machine Learning

Lars Marius Garshol

Deep Learning - Convolutional Neural Networks

Deep Learning - Convolutional Neural Networks

Deep Learning - Convolutional Neural Networks

Christian Perone

Suggestions: 1) For best quality, download the PDF before viewing. 2) Open at least two windows: One for the Youtube video, one for the screencast (link below), and optionally one for the slides themselves. 3) The Youtube video is shown on the first page of the slide deck, for slides, just skip to page 2. Screencast: http://youtu.be/VoL7JKJmr2I Video recording: http://youtu.be/CJRvb8zxRdE (Thanks to Al Friedrich!) In this talk, we take Deep Learning to task with real world data puzzles to solve. Data: - Higgs binary classification dataset (10M rows, 29 cols) - MNIST 10-class dataset - Weather categorical dataset - eBay text classification dataset (8500 cols, 500k rows, 467 classes) - ECG heartbeat anomaly detection - Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai - To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata

Deep Learning through Examples

Deep Learning through Examples

Deep Learning through Examples

En vedette (13)

Deep Learning

An introduction to Machine Learning

An introduction to Machine Learning

An introduction to Machine Learning

Deep learning - A Visual Introduction

Deep learning - A Visual Introduction

Deep learning - A Visual Introduction

Machine Learning for Dummies

Machine Learning for Dummies

Machine Learning for Dummies

Introduction to Machine Learning

Introduction to Machine Learning

Introduction to Machine Learning

Deep Learning in Computer Vision

Deep Learning in Computer Vision

Deep Learning in Computer Vision

Deep learning - Conceptual understanding and applications

Deep learning - Conceptual understanding and applications

Deep learning - Conceptual understanding and applications

Amazon Deep Learning

Amazon Deep Learning

Amazon Deep Learning

Deep Learning and the state of AI / 2016

Deep Learning and the state of AI / 2016

Deep Learning and the state of AI / 2016

Deep neural networks

Deep neural networks

Deep neural networks

Introduction to Big Data/Machine Learning

Introduction to Big Data/Machine Learning

Introduction to Big Data/Machine Learning

Deep Learning - Convolutional Neural Networks

Deep Learning - Convolutional Neural Networks

Deep Learning - Convolutional Neural Networks

Deep Learning through Examples

Deep Learning through Examples

Deep Learning through Examples

Plus de GoDataDriven

Streamlining Data Science Workflows with a Feature Catalog

Streamlining Data Science Workflows with a Feature Catalog

Streamlining Data Science Workflows with a Feature Catalog

Visualizing Big Data in a Small Screen

Visualizing Big Data in a Small Screen

Visualizing Big Data in a Small Screen

Building a Scalable and reliable open source ML Platform with MLFlow

Building a Scalable and reliable open source ML Platform with MLFlow

Building a Scalable and reliable open source ML Platform with MLFlow

Training Taster: Leading the way to become a data-driven organization

Training Taster: Leading the way to become a data-driven organization

Training Taster: Leading the way to become a data-driven organization

My Path From Data Engineer to Analytics Engineer

My Path From Data Engineer to Analytics Engineer

My Path From Data Engineer to Analytics Engineer

dbt Python models - GoDataFest by Guillermo Sanchez

dbt Python models - GoDataFest by Guillermo Sanchez

dbt Python models - GoDataFest by Guillermo Sanchez

Workshop on Google Cloud Data Platform

Workshop on Google Cloud Data Platform

Workshop on Google Cloud Data Platform

Prevent mis-aligned environments between developers, onboard new-joiners faster, and reduce the time it takes to take your project to production. Sounds interesting? Devcontainers can help you with this. Devcontainers allow you to connect your IDE to a running Docker container and develop inside it. This gives you all the benefits of reproducibility that Docker is known for. In this talk, I will walk you through what Devcontainers are, why they might be useful for you, and how to create one for your Python project using VSCode.

How to create a Devcontainer for your Python project

How to create a Devcontainer for your Python project

How to create a Devcontainer for your Python project

Many machine learning models we use today have the core assumption that our data needs to be tabular, but how often is this truly the case? What if our data points are not independent? By ignoring the potential interrelatedness of our data, do we lose meaningful information that our models cannot leverage? In this talk, we shall explore graph neural networks and highlight how they can solve interesting problems in a way that is intractable when limiting ourselves to using tabular data. We will look at the limitations of common algorithms and highlight how some clever linear algebra enables us to incorporate more meaningful information into our models. Social network data is a popular example of where relationships are relevant but relationships exist in many types of data where it may not be so obvious. Whether it's e-commerce, logistics or molecular data, relationships within your data likely exist and making use of them can be incredibly powerful. This talk will hopefully spark your curiosity and provide you with a way of looking at problems from a new angle. It is intended for anyone with an interest in machine learning and will only lightly touch on some technical details.

Using Graph Neural Networks To Embrace The Dependency In Your Data by Usman Z...

Using Graph Neural Networks To Embrace The Dependency In Your Data by Usman Z...

Using Graph Neural Networks To Embrace The Dependency In Your Data by Usman Z...

Time-series data is all around us: from logistics to digital marketing, from pricing to stock markets - it’s hard to imagine a modern business that has no time series data to forecast. However, mastering such forecasting is not an easy task. For this talk, we have collected a list of common time series issues that digital fortune tellers commonly run into. You will learn how to identify, understand and resolve them better. This will include stabilising divergent time series, handling outliers without anomaly propagation, reducing the impact of noise and more.

Common Issues With Time Series by Vadim Nelidov - GoDataFest 2022

Common Issues With Time Series by Vadim Nelidov - GoDataFest 2022

Common Issues With Time Series by Vadim Nelidov - GoDataFest 2022

MLOps CodeBreakfast on AWS - GoDataFest 2022

MLOps CodeBreakfast on AWS - GoDataFest 2022

MLOps CodeBreakfast on AWS - GoDataFest 2022

MLOps CodeBreakfast on Azure - GoDataFest 2022

MLOps CodeBreakfast on Azure - GoDataFest 2022

MLOps CodeBreakfast on Azure - GoDataFest 2022

Tableau vs. Power BI by Juan Manuel Perafan - GoDataFest 2022

Tableau vs. Power BI by Juan Manuel Perafan - GoDataFest 2022

Tableau vs. Power BI by Juan Manuel Perafan - GoDataFest 2022

Deploying a Modern Data Stack by Lasse Benninga - GoDataFest 2022

Deploying a Modern Data Stack by Lasse Benninga - GoDataFest 2022

Deploying a Modern Data Stack by Lasse Benninga - GoDataFest 2022

AWS Well-Architected Webinar Security - Ben de Haan

AWS Well-Architected Webinar Security - Ben de Haan

AWS Well-Architected Webinar Security - Ben de Haan

1. Start searching use cases with value & impact: without use cases, nobody will want to draft a data strategy Where do you want to go? Draft a clear Customer Experience that you want to create and think about the organization & data strategy to get there! 2. Get Tech (data scientists and engineers) and Business (Product Management & Commercial) on the same table: create a solid foundation. 3. Start with communities of practice to learn & experiment together and build the capability. 4. Stop talking about data. Start experimenting and doing. 5. Product Management needs to get real about data. (start training these capabilities)

The 7 Habits of Effective Data Driven Companies

The 7 Habits of Effective Data Driven Companies

The 7 Habits of Effective Data Driven Companies

The typical organizational model is that teams are in constant flux, are created for work, are only responsible for the change and are not empowered, or lack trust, to run products. A high performance organization model allows teams to take full responsibility for cost, compliance and security, and lets them own their own incidents. This improves quality, change failure rates, lower costs and leads to more happy employees. DevOps is about creating with the end in mind, cross-functional autonomous teams and end-tn-end responsibility. You build it, you run it. You break it, you fix it. This means you want to automate everything in a CI/CD pipeline. Roll-forward, don't roll-back. DevOps principles play an important role in a data-driven maturity model. Continuous prototyping and a data mindset and skills for everybody. In a Data Science Workflow combining input data and deriving the model features usually requires the most of the work, and lots of iterations before its done. Implement features one-by-one. So, start with a baseline model and compare this against more complex models, to see if additional complexity is worth the performance gain. The result of a data scientist is a trained model. Such a model contains 4 components: input data, derived features, chosen model type and hyperparameters. A trained model is always the combination of data and the code. So where do you run this trained model? Model management is versioning code but not the data. A model management server stores hyperparameters, performance metrics, metadata, trained models. IN a data science pipeline, we have two components for deployment: the application and the trained model. So we split the pipeline into parts: a build pipeline, a train pipeline and a deploy pipeline. A complete pipeline mapped to azure components would look largely like this: An Azure DevOps Build pipeline, an Azure ML Training pipeline and an Azure DevOps Release pipeline.

DevOps for Data Science on Azure - Marcel de Vries (Xpirit) and Niels Zeilema...

DevOps for Data Science on Azure - Marcel de Vries (Xpirit) and Niels Zeilema...

DevOps for Data Science on Azure - Marcel de Vries (Xpirit) and Niels Zeilema...

At GoDataFest 2019, Guy Kfir presented how AI delivers a new experience to Formula 1 fans across the world. AWS fuels the analytics through machine learning. Did you know a Formula 1 race car contains 120 sensors and generated 3 GB of data every race at 1,500 data points per second? AWS developed several applications, including overtake possibility, pitstop advantage. How important is it for your company to invest in Machine Learning and AI? There are three scenario's for AI/ML success: Automation, Enrichment and Invention. So, what are you waiting for: create the loop, advance your data strategy and organize for succes. To get started identify AI/ML use cases, educate yourself, start with AI services and move to Amazon Sagemaker, engage with AWS, consider the partner eco system (like GoDataDriven or Binx).

Artificial intelligence in actions: delivering a new experience to Formula 1 ...

Artificial intelligence in actions: delivering a new experience to Formula 1 ...

Artificial intelligence in actions: delivering a new experience to Formula 1 ...

During GoDataFest 2019, Rens Weijers, manager data & strategy and Peter van ' t Hof, data engineer, share the story of how Vattenfall develops smart applications on Azure. Vattenfall has the ambition to transition to fossil-free living within one generation. But what about decentral energy solutions in the Customers & Solutions business unit? Data is key to help customers to reduce their CO2 footprint. Azure enables Vattenfall to be personal and relevant towards customers.

Smart application on Azure at Vattenfall - Rens Weijers & Peter van 't Hof

Smart application on Azure at Vattenfall - Rens Weijers & Peter van 't Hof

Smart application on Azure at Vattenfall - Rens Weijers & Peter van 't Hof

Every company today is talking about AI/ML, but when most companies talk about AI/ML in their transformation journey, you hear terms like Proof of Concept, Feasibility Study, Pilot, A/B Test. We are at the peak of AI's hype, but only 12% of enterprises have deployed AI in production. Google aims to make big data processing available for everyone, the possiblities of Big Query ML are endless: Marketing, retail, industrial and IoT, media, gaming, and so fort.

Democratizing AI/ML with GCP - Abishay Rao (Google) at GoDataFest 2019

Democratizing AI/ML with GCP - Abishay Rao (Google) at GoDataFest 2019

Democratizing AI/ML with GCP - Abishay Rao (Google) at GoDataFest 2019

Plus de GoDataDriven (20)

Streamlining Data Science Workflows with a Feature Catalog

Streamlining Data Science Workflows with a Feature Catalog

Streamlining Data Science Workflows with a Feature Catalog

Visualizing Big Data in a Small Screen

Visualizing Big Data in a Small Screen

Visualizing Big Data in a Small Screen

Building a Scalable and reliable open source ML Platform with MLFlow

Building a Scalable and reliable open source ML Platform with MLFlow

Building a Scalable and reliable open source ML Platform with MLFlow

Training Taster: Leading the way to become a data-driven organization

Training Taster: Leading the way to become a data-driven organization

Training Taster: Leading the way to become a data-driven organization

My Path From Data Engineer to Analytics Engineer

My Path From Data Engineer to Analytics Engineer

My Path From Data Engineer to Analytics Engineer

dbt Python models - GoDataFest by Guillermo Sanchez

dbt Python models - GoDataFest by Guillermo Sanchez

dbt Python models - GoDataFest by Guillermo Sanchez

Workshop on Google Cloud Data Platform

Workshop on Google Cloud Data Platform

Workshop on Google Cloud Data Platform

How to create a Devcontainer for your Python project

How to create a Devcontainer for your Python project

How to create a Devcontainer for your Python project

Using Graph Neural Networks To Embrace The Dependency In Your Data by Usman Z...

Using Graph Neural Networks To Embrace The Dependency In Your Data by Usman Z...

Using Graph Neural Networks To Embrace The Dependency In Your Data by Usman Z...

Common Issues With Time Series by Vadim Nelidov - GoDataFest 2022

Common Issues With Time Series by Vadim Nelidov - GoDataFest 2022

Common Issues With Time Series by Vadim Nelidov - GoDataFest 2022

MLOps CodeBreakfast on AWS - GoDataFest 2022

MLOps CodeBreakfast on AWS - GoDataFest 2022

MLOps CodeBreakfast on AWS - GoDataFest 2022

MLOps CodeBreakfast on Azure - GoDataFest 2022

MLOps CodeBreakfast on Azure - GoDataFest 2022

MLOps CodeBreakfast on Azure - GoDataFest 2022

Tableau vs. Power BI by Juan Manuel Perafan - GoDataFest 2022

Tableau vs. Power BI by Juan Manuel Perafan - GoDataFest 2022

Tableau vs. Power BI by Juan Manuel Perafan - GoDataFest 2022

Deploying a Modern Data Stack by Lasse Benninga - GoDataFest 2022

Deploying a Modern Data Stack by Lasse Benninga - GoDataFest 2022

Deploying a Modern Data Stack by Lasse Benninga - GoDataFest 2022

AWS Well-Architected Webinar Security - Ben de Haan

AWS Well-Architected Webinar Security - Ben de Haan

AWS Well-Architected Webinar Security - Ben de Haan

The 7 Habits of Effective Data Driven Companies

The 7 Habits of Effective Data Driven Companies

The 7 Habits of Effective Data Driven Companies

DevOps for Data Science on Azure - Marcel de Vries (Xpirit) and Niels Zeilema...

DevOps for Data Science on Azure - Marcel de Vries (Xpirit) and Niels Zeilema...

DevOps for Data Science on Azure - Marcel de Vries (Xpirit) and Niels Zeilema...

Artificial intelligence in actions: delivering a new experience to Formula 1 ...

Artificial intelligence in actions: delivering a new experience to Formula 1 ...

Artificial intelligence in actions: delivering a new experience to Formula 1 ...

Smart application on Azure at Vattenfall - Rens Weijers & Peter van 't Hof

Smart application on Azure at Vattenfall - Rens Weijers & Peter van 't Hof

Smart application on Azure at Vattenfall - Rens Weijers & Peter van 't Hof

Democratizing AI/ML with GCP - Abishay Rao (Google) at GoDataFest 2019

Democratizing AI/ML with GCP - Abishay Rao (Google) at GoDataFest 2019

Democratizing AI/ML with GCP - Abishay Rao (Google) at GoDataFest 2019

Dernier

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

Artificial Intelligence Chap.5 : Uncertainty

Artificial Intelligence Chap.5 : Uncertainty

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Real Time Object Detection Using Open CV

Real Time Object Detection Using Open CV

Real Time Object Detection Using Open CV

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

This presentation explores the impact of HTML injection attacks on web applications, detailing how attackers exploit vulnerabilities to inject malicious code into web pages. Learn about the potential consequences of such attacks and discover effective mitigation strategies to protect your web applications from HTML injection vulnerabilities. for more information visit https://bostoninstituteofanalytics.org/category/cyber-security-ethical-hacking/

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

Boston Institute of Analytics

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Scaling API-first – The story of a global engineering organization

Scaling API-first – The story of a global engineering organization

Scaling API-first – The story of a global engineering organization

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Dernier (20)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

Artificial Intelligence Chap.5 : Uncertainty

Artificial Intelligence Chap.5 : Uncertainty

Artificial Intelligence Chap.5 : Uncertainty

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Real Time Object Detection Using Open CV

Real Time Object Detection Using Open CV

Real Time Object Detection Using Open CV

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

HTML Injection Attacks: Impact and Mitigation Strategies

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Scaling API-first – The story of a global engineering organization

Scaling API-first – The story of a global engineering organization

Scaling API-first – The story of a global engineering organization

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Building a big data warehouse

1. GoDataDriven PROUDLY PART OF THE XEBIA GROUP @mids106 jorisbontje@godatadriven.com Building a Big Data Warehouse Joris Bontje Big Data Hacker

2. About Me Big Data Hacker Data Driven Solution Architect HadoopTrainer

3.

4. About GoDataDriven

5. Data Warehouse Evolution

6. http://en.wikipedia.org/wiki/Data_warehouse In computing, a data warehouse is a database used for reporting and data analysis.

7. Database Architecture (1.0) Products) Customers) Orders) Inventory) Sales)DB)

8. Analytical Database (2.0) Sales& Inventory&Customers& Products& Orders&

9. Basic DWH Architecture TX# DB# Analy+cal# DB# BI#ETL

10. Data Marts TX# DB# DW# Sales# Mktg# Prch# BI#

11. Multiple Data-Sources other& Files& TX& DB& DW& Sales& Mktg& Prch& BI&

12. Operational Data Store DW#ODS# other# Files# TX# DB# Sales# Mktg# Prch# BI#

14. GoDataDriven No Hadoop DW#ODS# other# Files# TX# DB# Sales# Mktg# Prch# BI#

15. GoDataDriven ETL Engine other& Files& TX& DB& Sales& Mktg& Prch& DW BI&

16. GoDataDriven Tiered Data Warehouse other& Files& TX& DB& Sales& Mktg& Prch& BI&

17. GoDataDriven Analytical Query Engine other& Files& TX& DB& BI&

20. Tools Applied

21. Tools Applied

22. Considerations

23. Considerations Big Data is dirty Automate everything Monitoring and QA become the same thing

24. My Past Trends Big Data Forum 2012

25. My PastTrends Cloud / On-demand

26. My PastTrends Hadoop Hardware

27. My PastTrends Batch → Real-Time

28. New Trends XebiCon 2013

29. Trends Impala Open Source, Real-time Query engine for Hadoop

30. Trends Defacto standard for Hadoop metadata

31. Simple Database Architecture Products) Customers) Orders) Inventory) Sales)DB)

32. The future? Products) Customers) Orders) Inventory) Sales)

33. GoDataDriven We’re hiring / Questions? / Thank you! @mids106 jorisbontje@godatadriven.com Joris Bontje Big Data Hacker