Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guillaume Torch, Big Data Engineer - GumGum

•

1 j'aime•507 vues

GumGum uses Druid to ingest more than 30 billion events every day, which can be queried almost as soon as they happen with a very low response time. This is a tell-all talk about GumGum's love story with Druid, how Druid works and how GumGum leverages Druid's capabilities.

Technologie

Real Time Analytics at Scale with
Guillaume Torche
Saturday 9th July
Big Data LA 2016

${"id":"165a0b33-fd2d-4f45-83e4-a373cc4e6333","t":1435633205023,"cl":"js","ua":"Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) GSA/6.0.51363 Mobile/12F70 Safari/600.1.4","ip":"70.109.46.30","cc":"US","rg":"CA","ct":"Santa Maria","pc":"93455","mc":855,"isp":"Verizon Internet Services","bf":"6c52d0492fa92bd912d3a53a3c864e5a47008779","vst":"62784988-5d30-40a2-a334- 43e456499ac6"},{"v":"1.1","e":"view","si":632,"t":"56c89c04","ab":25465,"pv":"6cff666d-92c5- 42ac-9f94-39a40f9f297d","pu":"http://www.nytimes.com/2016/01/08/business/international/a-new- economic-era-for-china-goes-off-the- rails.html?hp&action=click&pgtype=Homepage&clickSource=story-heading&module=first-column- region&region=top-news&WT.nav=top- news&_r=0","af":false,"rpm":0.65999997,"pc":0.25,"nc":0,"dsp":27,"dai":"pqdmnbu","dci":"rpz224 3","do":"*.nytimes.com","vi":3,"pi":1134,"pri":3,"uti":16,"adi":300,"ei":3,"spi":172,"atmi":20 8,"ct":"rtb","cli":3,"cid":1256}$

What does Real Time mean?
Response time
Data freshness
Response Time

About me
Guillaume Torche
Born and raised in France
First experience in Big Data at Aubay in Paris
Started working for GumGum 2 years ago
About one year and a half of experience with Druid

Invented In Image advertising in 2008
http://gumgum.com/gallery
Processing 2.6B image impressions / month
10B impressions / month
10B events / day
2000 premium publishers
132 employees - 37.5% YOY growth

Interactive data exploration
Fast results
Access fresh data

Before we met Big Data
MySQL for Real Time reporting
Data aggregation with Storm
Batch updates of MySQL metrics

Brief of Druid
Built by another Ad tech company
Used by top tech companies

Key features
Arbitrary Slice-N-Dice data
Fast aggregations on time series data
Highly available
Real Time
Distributed

Queries
Broker
Realtime
Historical
Stream
Deep storage

Queries
Broker
Realtime
Historical
Stream
Deep storage
Coordinator

Queries
Broker
Realtime
Historical
Stream
Deep storage
Coordinator
Batch
Data
Ingestion
Ingestion

Column compression - Dictionary encoding
Timestamp Product Vertical Ad impressions Clicks Revenue
2015-01-01T00:01:35Z In-Image Gaming 15 3 10
2015-01-04T00:03:48Z In-Screen Gaming 5 1 3
2015-01-03T00:08:05Z Native Fashion 2 0 0.5
2015-01-02T00:02:35Z In-Image Fashion 25 15 50
● Create internal ids:
○ Gaming => 0, Fashion => 1
● Store column:
○ Vertical => [ 0 0 1 1 ]

Inverted Index
Row Vertical
0 Gaming
1 Gaming
2 Fashion
3 Fashion
Gaming
[1, 1, 0, 0]
Fashion
[0, 0, 1, 1]
Gaming
OR
Fashion
[1, 1, 1, 1]

How does Druid achieve low
latency ingestion?

Write optimized
Read optimized
Flush
S1
S2
S3
S4 S5Final
Segment
Historical

https://www.linkedin.com/in/guillaume-torche-4a209790
http://imply.io/
http://druid.io
http://gumgum.com/careers

Contenu connexe

En vedette

Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...

Data Con LA

Prototypes are typically re-implemented in another language due to compatibility issues with R in the enterprise, but TIBCO Enterprise Runtime for R (TERR) allows the language to be run on several platforms. Enterprise-level scalability has been brought to the R language, enabling rapid iteration without the need to recode, re-implement and test. This presentation will delve further into these topics, highlighting specific use cases and the true value that can be gained from utilizing R. The session will be followed by a lively, open Q&A discussion.

Big Data Day LA 2016/ Big Data Track - Apply R in Enterprise Applications, Lo...

Data Con LA

Druid @ branch

Biswajit Das

Abstract:- Tracking user events as they happen can challenge anyone providing real time user interaction. It can demand both huge scale and a lot of processing to support dynamic adjustment to targeting products and services. As the operational data store Couchbase data services are capable of processing tens of millions of updates a day. Streaming through systems such as Apache Spark and Kafka into Hadoop, information about these key events can be turned into deeper knowledge. We will review Lambda architectures deployed at sites like PayPal, Live Person and LinkedIn that leverage a Couchbase Data Pipeline. Bio:- Justin Michaels. With over 20 years experience in deploying mission critical systems, Justin Michaels industry experience covers capacity planning, architecture and industry vertical experience. Justin brings his passion for architecting, implementing and improving Couchbase to the community as a Solution Architect. His expertise involves both conventional application platforms as well as distributed data management systems. He regularly engages with existing and new Couchbase customers in performance reviews, architecture planning and best practice guidance.

Stream your Operational Data with Apache Spark & Kafka into Hadoop using Couc...

Data Con LA

This talk provides an overview of the open source Storm system for processing Big Data in realtime. The talk starts with an overview of the technology, including key components: Nimbus, Zookeeper, Topology, Tuple, Trident. It looks at integration with Hadoop through YARN and recent improvements. The presentation then dives into the complex Big Data architecture in which Storm can be integrated . The result is a compelling stack of technologies including integrated Hadoop clusters, MPP, and NoSQL databases. After this, we look at example use cases for Storm: real-time advertising statistics, updating a Machine Learned model for content popularity predictions, and financial compliance monitoring.

Real time big data analytics with Storm by Ron Bodkin of Think Big Analytics

Data Con LA

How do you run analytics queries on a dataset that grows by billions of entries a day? What if you need to be able to drill into it, filter it or aggregate it by any available dimension? On demand, without precomputing, and with sub second latency? Oh and this isn't for an internal dashboard. The interface is customer-facing and is going to be accessed by thousands of clients. In this talk, I'll tell you how Criteo, one of the biggest ad tech firms in the world, uses Druid to build its new analytics platform. Druid is an open-source, real-time data store designed to power interactive applications at scale. I’ll walk you through its architecture, explain how it scales and how data is stored on disk and in memory to serve queries faster than you can blink.

Interactive analytics at scale with druid

Julien Lavigne du Cadet

Since Doug Cutting invented Hadoop and Amazon Web Services released S3 ten years ago, we've seen quite a bit of innovation in large-scale data storage and processing. These innovations have enabled engineers to build data infrastructure at scale, many of them fail to fill their scalable systems with useful data, struggling to unify data silos or failing to collect logs from thousands of servers and millions of containers. Fluentd and Embulk are two projects that I've been involved to solve the unsexy yet critical problem of data collection and transport. In this talk, I will give an overview of Fluentd and Embulk and give a survey of how they are used at companies like Microsoft and Atlassian or in projects like Docker and Kubernetes.

Big Data Day LA 2016/ Big Data Track - Fluentd and Embulk: Collect More Data,...

Data Con LA

In this interactive panel discussion, you will hear from these Spark experts as to why they chose to go "all-in" on Spark, leveraging the rich core capabilities that make Spark so exciting, and committing to significant IP that turns Spark into a world-class enterprise data preparation engine. Raymond and David will explain specific cases where capabilities were built on top of core Spark to provide a true interactive data prep application experience. Innovations such as creating a Domain Specific Language (DSL), an optimizing compiler, a persistent columnar caching layer, application specific Resilient Distributed Datasets (RDDs), on-line aggregation operators to solve the core memory, pipelining and shuffling obstacles to produce a highly interactive application with the core user and data volume scale-out benefits of Spark.

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Panel - Interactive Applic...

Data Con LA

Big Data Day LA 2016/ NoSQL track - Analytics at the Speed of Light with Redi...

Data Con LA

For decades the entertainment industry has relied on small data (e.g., test audiences, focus groups, etc.) when making decisions about programming and developing new projects. With the rise of online content distribution and internet TV, companies now have access to massive and unprecedented amounts of viewing data. This creates a new challenge: How to marry the power of big data and big analytics with the creative force of the entertainment industry to make both better. I will talk about how Netflix has tackled this issue, and in doing so, created a dynamic new force in Hollywood.

Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...

Data Con LA

Big Data Day LA 2016/ Data Science Track - Affinity Marketing Leveraging Crow...

Data Con LA

This talk explores how Netflix equips its engineers with the freedom to find and introduce the right software for the job - even if it isn't used anywhere else in-house. Examples include how Netflix has enabled analysts to fluidly switch between MPP RDBMS and an auto-scaling Presto cluster, how Spark + NoSQL stores are used when deploying data sets to internal web apps, and how data scientists are enabled to work in the ML framework of their choosing and deploy models as a service.

Big Data Day LA 2016/ Big Data Track - Rapid Analytics @ Netflix LA (Updated ...

Data Con LA

Big Data Day LA 2016 Keynote - Reynold Xin/ Databricks

Data Con LA

Apache HBase™ is the Hadoop database, a distributed, scalable, big data store. Use Apache HBase when you need random, realtime read/write access to your Big Data. This project's goal is the hosting of very large tables -- billions of rows X millions of columns -- atop clusters of commodity hardware. Apache HBase is an open-source, distributed, versioned, non-relational database modeled after Google's Bigtable This talk will introduce to Apache HBase and will give you an overview of Columnar databases. We will also talk about how Facebook is using HBase currently. We will talk about HBase security, Apache Phoenix and Apache Slider

Apache HBase - Introduction & Use Cases

Data Con LA

Real-time Analytics with Apache Flink and Druid

Jan Graßegger

The goal of this talk to lay out a framework for what algorithms work best in which situations, and why. Drawing on results of hundreds of crowd-sourced predictive modeling contests, this talk shows examples of how structure informs a choice in algorithm. As an illustration of these concepts, ZestFinance's work with China's retail giant, JD.com is used to describe how the right algorithms were applied to the right datasets to turn shopping data into credit data -- creating credit scores from scratch.

Big Data Day LA 2016/ Data Science Track - The Right Tool for the Job: Guidel...

Data Con LA

Real-time analytics with Druid at Appsflyer

Michael Spector

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Data Provenance Support in...

Data Con LA

The case study is about ViralGains - a US based video marketing platform. The presentation was delivered by me (Salil Kalia) at Great Indian Developer Summit (GIDS) 2016. This is a piece of a great work that we have done at TO THE NEW Digital with our customer, ViralGains. Here, I show-cased Druid (http://druid.io) and the supporting technologies (Kafka/Zookeeper) to demonstrate how it helped us in building a stable realtime analytics system, in capturing hundreds of millions of analytics events per day. When it comes to Ad industry - it becomes very important to be precise or close to precision because money is involved at every step (even for a single ad impression). The case study included a demo and a short talk on their journey of moving from Redis to Cassandra and finally ending up on Druid with an outstanding performance.

Case Study: Realtime Analytics with Druid

Salil Kalia

Big Data Day LA 2016 Keynote - Andy Feng/ Yahoo

Data Con LA

En vedette (20)

Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...

Big Data Day LA 2016/ Big Data Track - Apply R in Enterprise Applications, Lo...

Druid @ branch

Stream your Operational Data with Apache Spark & Kafka into Hadoop using Couc...

Real time big data analytics with Storm by Ron Bodkin of Think Big Analytics

Interactive analytics at scale with druid

Big Data Day LA 2016/ Big Data Track - Fluentd and Embulk: Collect More Data,...

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Panel - Interactive Applic...

Big Data Day LA 2016/ NoSQL track - Analytics at the Speed of Light with Redi...

Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...

Big Data Day LA 2016/ Data Science Track - Affinity Marketing Leveraging Crow...

Big Data Day LA 2016/ Big Data Track - Rapid Analytics @ Netflix LA (Updated ...

Big Data Day LA 2016 Keynote - Reynold Xin/ Databricks

Apache HBase - Introduction & Use Cases

Real-time Analytics with Apache Flink and Druid

Big Data Day LA 2016/ Data Science Track - The Right Tool for the Job: Guidel...

Real-time analytics with Druid at Appsflyer

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Data Provenance Support in...

Case Study: Realtime Analytics with Druid

Big Data Day LA 2016 Keynote - Andy Feng/ Yahoo

Similaire à Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guillaume Torch, Big Data Engineer - GumGum

Nicholas Brown, Director of Software Architecture, Vivint Four years ago Vivint set out to mainstream the Smart Home industry with a new purpose-built platform. Within a year, this platform would need to be ready for the integration of millions of devices. Performance was a must. But, perhaps even more importantly, it would need to offer unprecedented extensibility to meet the needs of a nascent industry with little tolerance for rigidity, still being defined even four years later. MongoDB would provide the backbone of this ground-breaking IoT Platform. Today, we will explore the challenges Vivint faced in design and scale, and what we learned (the hard way) about building an IoT platform. https://www.mongodb.com/

Building and Scaling the Internet of Things with MongoDB at Vivint

MongoDB

[WTMC 2019] Detecting malicious campaigns in obfuscated JavaScript with scala...

Oleksii Starov

Intro to Big Data - Orlando Code Camp 2014

John Ternent

Javascript Object Signing & Encryption

Aaron Zauner

Dynamic Apps with WebSockets and MQTT - IBM Impact 2014

Bryan Boyd

インフラ野郎 Azureチーム博多夏祭り

Toru Makabe

Model-driven Telemetry: The Foundation of Big Data Analytics

Cisco Canada

How We Learned To Love The Data Center Operating System

saulius_vl

10 Key MongoDB Performance Indicators

iammutex

Drone Data Flowing Through Apache NiFi

Timothy Spann

sf bay area dfir meetup (2016-04-30) - OsxCollector

Rishi Bhargava

This talk will be about difficulties they have met on the road, and how they overcame some of them. It will reveal both - technical and management side. Alberts Bušinskis (Garuda Dev Giri) - Yoga & Meditation teacher, IT enthusiast, software developer. Many years ago he had left a very good job in Switzerland and went to Indian Himalayas to practice Yoga. After he realized that Software Development is not different from Yoga, he came back to homeland trying to be helpful and inspiring.

"Traffic Speed Control System in the Cloud using Machine Learning" by Albert ...

DevClub_lv

We will examine how IoT and Serverless come together so you can build scalable systems in a fast modular way. How small, single purpose functions can fit into an IoT architecture enabling you to be flexible as you design the system. We will briefly look at features Azure has to offer for IoT and then look at how to extend the system using Azure Functions to provide functionality like alerts, device management, and Cloud to Device communication. By the end of the session, you will have seen real use cases from projects that have been built at Microsoft and know how to add leverage serverless in IoT scenarios.

Serverless and IoT

James Sturtevant

Mongo at Sailthru (MongoNYC 2011)

ibwhite

Model driven telemetry

Cisco Canada

Even for a big incident response team handling all of the repetitive tasks related to malware infections is a tedious task. Our malware analysts (we love to call them "ninjas" as they are superfast!) have spent a lot of their precious time staring at the digital forensics collected from potentially infected macOS systems. Early on, we have automated some parts of the collection (taking advantage of our open source OSXCollector) and analysis (with OSXCollector Output Filters), augmenting the initial set of digital forensics with the information gathered from the threat intelligence APIs. This helped us in taking the full advantage of the additional information on potentially suspicious domains, URLs and file hashes. But our approach to the analysis still required a certain degree of configuration and manual maintenance that was consuming lots of attention from malware responders. So we have turned our OSXCollector toolkit into AMIRA: Automated Malware Incident Response and Analysis service. It's like a ninja sword and a chainsaw combined into one. AMIRA turns the forensic information gathered by OSXCollector into an actionable response plan, suggesting the infection source as well as suspicious files and domains requiring a closer look. Furthermore, we integrated AMIRA with our incident response platform, making sure that as little interaction as necessary is required from the analyst to trigger the collection of the forensic artifacts.

Slicing Apples with Ninja Sword: Fighting Malware at the Corporate Level (OWA...

Jakub "Kuba" Sendor

Windows azure mobile services

Richard Conway

2016 02-04 howard look tidepool attd 2016 v2

Tidepool

Cracking the nut, solving edge ai with apache tools and frameworks

Timothy Spann

Liquid Stream Processing Across Web Browsers and Web Servers

Masiar Babazadeh

Similaire à Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guillaume Torch, Big Data Engineer - GumGum (20)

Building and Scaling the Internet of Things with MongoDB at Vivint

[WTMC 2019] Detecting malicious campaigns in obfuscated JavaScript with scala...

Intro to Big Data - Orlando Code Camp 2014

Javascript Object Signing & Encryption

Dynamic Apps with WebSockets and MQTT - IBM Impact 2014

インフラ野郎 Azureチーム博多夏祭り

Model-driven Telemetry: The Foundation of Big Data Analytics

How We Learned To Love The Data Center Operating System

10 Key MongoDB Performance Indicators

Drone Data Flowing Through Apache NiFi

sf bay area dfir meetup (2016-04-30) - OsxCollector

"Traffic Speed Control System in the Cloud using Machine Learning" by Albert ...

Serverless and IoT

Mongo at Sailthru (MongoNYC 2011)

Model driven telemetry

Slicing Apples with Ninja Sword: Fighting Malware at the Corporate Level (OWA...

Windows azure mobile services

2016 02-04 howard look tidepool attd 2016 v2

Cracking the nut, solving edge ai with apache tools and frameworks

Liquid Stream Processing Across Web Browsers and Web Servers

Plus de Data Con LA

Data Con LA 2022 Keynotes

Data Con LA

Data Con LA 2022 Keynotes

Data Con LA

Data Con LA 2022 Keynote

Data Con LA

Data Con LA 2022 - Startup Showcase

Data Con LA

Data Con LA 2022 Keynote

Data Con LA

Mike Limcaco, Analytics Specialist / Customer Engineer at Google Measure trends in a particular topic or search term across Google Search across the US down to the city-level. Integrate these data signals into analytic pipelines to drive product, retail, media (video, audio, digital content) recommendations tailored to your audience segment. We'll discuss how Google unique datasets can be used with Google Cloud smart analytic services to process, enrich and surface the most relevant product or content that matches the ever-changing interests of your local customer segment.

Data Con LA 2022 - Using Google trends data to build product recommendations

Data Con LA

Melinda Thielbar, Data Science Practice Lead and Director of Data Science at Fidelity Investments From corporations to governments to private individuals, most of the AI community has recognized the growing need to incorporate ethics into the development and maintenance of AI models. Much of the current discussion, though, is meant for leaders and managers. This talk is directed to data scientists, data engineers, ML Ops specialists, and anyone else who is responsible for the hands-on, day-to-day of work building, productionalizing, and maintaining AI models. We'll give a short overview of the business case for why technical AI expertise is critical to developing an AI Ethics strategy. Then we'll discuss the technical problems that cause AI models to behave unethically, how to detect problems at all phases of model development, and the tools and techniques that are available to support technical teams in Ethical AI development.

Data Con LA 2022 - AI Ethics

Data Con LA

Antje Barth, Principal Developer Advocate, AI/ML at AWS & Chris Fregly, Principal Engineer, AI & ML at AWS The frequency and severity of natural disasters are increasing. In response, governments, businesses, nonprofits, and international organizations are placing more emphasis on disaster preparedness and response. Many organizations are accelerating their efforts to make their data publicly available for others to use. Repositories such as the Registry of Open Data on AWS and Humanitarian Data Exchange contain troves of data available for use by developers, data scientists, and machine learning practitioners. In this session, see how a community of developers came together though the AWS Disaster Response hackathon to build models to support natural disaster preparedness and response.

Data Con LA 2022 - Improving disaster response with machine learning

Data Con LA

Sig Narvaez, Executive Solution Architect at MongoDB MongoDB is now a Developer Data Platform. Come learn whatï¿½s new in the 6.0 release and Atlas following all the recent announcements made at MongoDB World 2022. Topics will include - Atlas Search which combines 3 systems into one (database, search engine, and sync mechanisms) letting you focus on your product's differentiation. - Atlas Data Federation to seamlessly query, transform, and aggregate data from one or more MongoDB Atlas databases, Atlas Data Lake and AWS S3 buckets - Queryable Encryption lets you run expressive queries on fully randomized encrypted data to meet the most stringent security requirements - Relational Migrator which analyzes your existing relational schemas and helps you design a new MongoDB schema. - And more!

Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas

Data Con LA

Jaysen Gillespie, Head of Analytics and Data Science at RTB House 1. Shopkick has over 30M downloads, but the userbase is very heterogeneous. Anecdotal evidence indicated a wide variety of users for whom the app holds long-term appeal. 2. Marketing and other teams challenged Analytics to get beyond basic summary statistics and develop a holistic segmentation of the userbase. 3. Shopkick's data science team used SQL and python to gather data, clean data, and then perform a data-driven segmentation using a k-means algorithm. 4. Interpreting the results is more work -- and more fun -- than running the algo itself. We'll discuss how we transform from ""segment 1"", ""segment 2"", etc. to something that non-analytics users (Marketing, Operations, etc.) could actually benefit from. 5. So what? How did team across Shopkick change their approach given what Analytics had discovered.

Data Con LA 2022 - Real world consumer segmentation

Data Con LA

Ravi Pillala, Chief Data Architect & Distinguished Engineer at Intuit TurboTax is one of the well known consumer software brand which at its peak serves 385K+ concurrent users. In this session, We start with looking at how user behavioral data & tax domain events are captured in real time using the event bus and analyzed to drive real time personalization with various TurboTax data pipelines. We will also look at solutions performing analytics which make use of these events, with the help of Kafka, Apache Flink, Apache Beam, Spark, Amazon S3, Amazon EMR, Redshift, Athena and Amazon lambda functions. Finally, we look at how SageMaker is used to create the TurboTax model to predict if a customer is at risk or needs help.

Data Con LA 2022 - Modernizing Analytics & AI for today's needs: Intuit Turbo...

Data Con LA

Data Con LA 2022 - Moving Data at Scale to AWS

Data Con LA

Anand Ranganathan, Chief AI Officer at Unscrambl Conversational AI is getting more and more widely used for customer support and employee support use-cases. In this session, I'm going to talk about how it can be extended for data analysis and data science use-cases ... i.e., how users can interact with a bot to ask analytical questions on data in relational databases. This allows users to explore complex datasets using a combination of text and voice questions, in natural language, and then get back results in a combination of natural language and visualizations. Furthermore, it allows collaborative exploration of data by a group of users in a channel in platforms like Microsoft Teams, Slack or Google Chat. For example, a group of users in a channel can ask questions to a bot in plain English like ""How many cases of Covid were there in the last 2 months by state and gender"" or ""Why did the number of deaths from Covid increase in May 2022"", and jointly look at the results that come back. This facilitates data awareness, data-driven collaboration and joint decision making among teams in enterprises and outside. In this talk, I'll describe how we can bring together various features including natural-language understanding, NL-to-SQL translation, dialog management, data story-telling, semantic modeling of data and augmented analytics to facilitate collaborate exploration of data using conversational AI.

Data Con LA 2022 - Collaborative Data Exploration using Conversational AI

Data Con LA

Anil Inamdar, VP & Head of Data Solutions at Instaclustr The most modernized enterprises utilize polyglot architecture, applying the best-suited database technologies to each of their organization's particular use cases. To successfully implement such an architecture, though, you need a thorough knowledge of the expansive NoSQL data technologies now available. Attendees of this Data Con LA presentation will come away with: -- A solid understanding of the decision-making process that should go into vetting NoSQL technologies and how to plan out their data modernization initiatives and migrations. -- They will learn the types of functionality that best match the strengths of NoSQL key-value stores, graph databases, columnar databases, document-type databases, time-series databases, and more. -- Attendees will also understand how to navigate database technology licensing concerns, and to recognize the types of vendors they'll encounter across the NoSQL ecosystem. This includes sniffing out open-core vendors that may advertise as â€œopen source,"" but are driven by a business model that hinges on achieving proprietary lock-in. -- Attendees will also learn to determine if vendors offer open-code solutions that apply restrictive licensing, or if they support true open source technologies like Hadoop, Cassandra, Kafka, OpenSearch, Redis, Spark, and many more that offer total portability and true freedom of use.

Data Con LA 2022 - Why Database Modernization Makes Your Data Decisions More ...

Data Con LA

Zia Khan, Computer Systems Analyst and Data Scientist at LearningFuze Data Science tutorial is designed for people who are new to Data Science. This is a beginner level session so no prior coding or technical knowledge is required. Just bring your laptop with WiFi capability. The session starts with a review of what is data science, the amount of data we generate and how companies are using that data to get insight. We will pick a business use case, define the data science process, followed by hands-on lab using python and Jupyter notebook. During the hands-on portion we will work with pandas, numpy, matplotlib and sklearn modules and use a machine learning algorithm to approach the business use case.

Data Con LA 2022 - Intro to Data Science

Data Con LA

Data Con LA 2022 - How are NFTs and DeFi Changing Entertainment

Data Con LA

Curtis ODell, Global Director Data Integrity at Tricentis Join me to learn about a new end-to-end data testing approach designed for modern data pipelines that fills dangerous gaps left by traditional data management toolsâ€”one designed to handle structured and unstructured data from any source. You'll hear how you can use unique automation technology to reach up to 90 percent test coverage rates and deliver trustworthy analytical and operational data at scale. Several real world use cases from major banks/finance, insurance, health analytics, and Snowflake examples will be presented. Key Learning Objective 1. Data journeys are complex and you have to ensure integrity of the data end to end across this journey from source to end reporting for compliance 2. Data Management tools do not test data, they profile and monitor at best, and leave serious gaps in your data testing coverage 3. Automation with integration to DevOps and DataOps' CI/CD processes are key to solving this. 4. How this approach has impact in your vertical

Data Con LA 2022 - Why Data Quality vigilance requires an End-to-End, Automat...

Data Con LA

Arif Ansari, Professor at University of Southern California Super Bowl Ad cost $7 million and each year a few Super Bowl ads go viral. The traditional A/B testing does not predict virality. Some highly shared ones reach over 60 million organic views, which can be more valuable than views on TV. Not only are these voluntary, but they are typically without distraction, and win viewer engagement in the form of likes, comments, or shares. A Super Bowl ad that wins 69 million views on YouTube (e.g., Alexa Mind Reader) costs less than 10 cents per quality view! However, the challenge is triggering virality. We developed a method to predict virality and engineer virality into Ads. 1. Prof. Gerard J. Tellis and co-authors recommended that advertisers use YouTube to tease, test, and tweak (TTT) their ads to maximize sharing and viewing. 2022 saw that maxim put into practice. 2. We developed viral Ads prediction using two scientific models: a. Prof. Gerard Tellis et al.'s model for viral prediction b. Deep Learning viral prediction using social media effect 3. The model was able to identify all the top 15 Viral Ads it performed better than the traditional agencies. 4. New proposed method is Tease, Test, Tweak, Target and Spots Ad.

Data Con LA 2022-Perfect Viral Ad prediction of Superbowl 2022 using Tease, T...

Data Con LA

Jai Bansal, Senior Manager, Data Science at Aetna This talk describes an internal data product called Member Embeddings that facilitates modeling of member medical journeys with machine learning. Medical claims are the key data source we use to understand health journeys at Aetna. Claims are the data artifacts that result from our members' interactions with the healthcare system. Claims contain data like the amount the provider billed, the place of service, and provider specialty. The primary medical information in a claim is represented in codes that indicate the diagnoses, procedures, or drugs for which a member was billed. These codes give us a semi-structured view into the medical reason for each claim and so contain rich information about members' health journeys. However, since the codes themselves are categorical and high-dimensional (10K cardinality), it's challenging to extract insight or predictive power directly from the raw codes on a claim. To transform claim codes into a more useful format for machine learning, we turned to the concept of embeddings. Word embeddings are widely used in natural language processing to provide numeric vector representations of individual words. We use a similar approach with our claims data. We treat each claim code as a word or token and use embedding algorithms to learn lower-dimensional vector representations that preserve the original high-dimensional semantic meaning. This process converts the categorical features into dense numeric representations. In our case, we use sequences of anonymized member claim diagnosis, procedure, and drug codes as training data. We tested a variety of algorithms to learn embeddings for each type of claim code. We found that the trained embeddings showed relationships between codes that were reasonable from the point of view of subject matter experts. In addition, using the embeddings to predict future healthcare-related events outperformed other basic features, making this tool an easy way to improve predictive model performance and save data scientist time.

Data Con LA 2022- Embedding medical journeys with machine learning to improve...

Data Con LA

Jie Chen, Manager Advisory, KPMG Data is the new oil. However, many organizations have fragmented data in siloed line of businesses. In this topic, we will focus on identifying the legacy patterns and their limitations and introducing the new patterns packed by Kafka's core design ideas. The goal is to tirelessly pursue better solutions for organizations to overcome the bottleneck in data pipelines and modernize the digital assets for ready to scale their businesses. In summary, we will walk through three uses cases, recommend Dos and Donts, Take aways for Data Engineers, Data Scientist, Data architect in developing forefront data oriented skills.

Data Con LA 2022 - Data Streaming with Kafka

Data Con LA

Plus de Data Con LA (20)

Data Con LA 2022 Keynotes

Data Con LA 2022 Keynote

Data Con LA 2022 - Startup Showcase

Data Con LA 2022 Keynote

Data Con LA 2022 - Using Google trends data to build product recommendations

Data Con LA 2022 - AI Ethics

Data Con LA 2022 - Improving disaster response with machine learning

Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas

Data Con LA 2022 - Real world consumer segmentation

Data Con LA 2022 - Modernizing Analytics & AI for today's needs: Intuit Turbo...

Data Con LA 2022 - Moving Data at Scale to AWS

Data Con LA 2022 - Collaborative Data Exploration using Conversational AI

Data Con LA 2022 - Why Database Modernization Makes Your Data Decisions More ...

Data Con LA 2022 - Intro to Data Science

Data Con LA 2022 - How are NFTs and DeFi Changing Entertainment

Data Con LA 2022 - Why Data Quality vigilance requires an End-to-End, Automat...

Data Con LA 2022-Perfect Viral Ad prediction of Superbowl 2022 using Tease, T...

Data Con LA 2022- Embedding medical journeys with machine learning to improve...

Data Con LA 2022 - Data Streaming with Kafka

Dernier

Finology Group – Insurtech Innovation Award 2024

The Digital Insurer

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Friends Colony Women Seeking Men

Delhi Call girls

[2024]Digital Global Overview Report 2024 Meltwater.pdf

hans926745

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

What are drone anti-jamming systems? The drone anti-jamming systems and anti-spoof technology protect against interference, jamming, and spoofing of the UAVs. To protect their security, countries are beginning to research drone anti-jamming systems, also known as drone strike weapons. The anti-jam and anti-spoof technology protects against interference, jamming and spoofing. A drone strike weapon is a drone attack weapon that can attack and destroy enemy drones. So what is so unique about this amazing system?

What Are The Drone Anti-jamming Systems Technology?

Antenna Manufacturer Coco

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Delhi Call girls

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Heather Hedden, Senior Consultant at Enterprise Knowledge, presented “The Role of Taxonomy and Ontology in Semantic Layers” at a webinar hosted by Progress Semaphore on April 16, 2024. Taxonomies at their core enable effective tagging and retrieval of content, and combined with ontologies they extend to the management and understanding of related data. There are even greater benefits of taxonomies and ontologies to enhance your enterprise information architecture when applying them to a semantic layer. A survey by DBP-Institute found that enterprises using a semantic layer see their business outcomes improve by four times, while reducing their data and analytics costs. Extending taxonomies to a semantic layer can be a game-changing solution, allowing you to connect information silos, alleviate knowledge gaps, and derive new insights. Hedden, who specializes in taxonomy design and implementation, presented how the value of taxonomies shouldn’t reside in silos but be integrated with ontologies into a semantic layer. Learn about: - The essence and purpose of taxonomies and ontologies in information and knowledge management; - Advantages of semantic layers leveraging organizational taxonomies; and - Components and approaches to creating a semantic layer, including the integration of taxonomies and ontologies

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Enterprise Knowledge

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

HampshireHUG

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

GenCyber Cyber Security Day Presentation

Michael W. Hawkins

Dernier (20)

Finology Group – Insurtech Innovation Award 2024

Partners Life - Insurer Innovation Award 2024

Artificial Intelligence: Facts and Myths

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

08448380779 Call Girls In Friends Colony Women Seeking Men

[2024]Digital Global Overview Report 2024 Meltwater.pdf

GenAI Risks & Security Meetup 01052024.pdf

What Are The Drone Anti-jamming Systems Technology?

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Boost Fertility New Invention Ups Success Rates.pdf

A Domino Admins Adventures (Engage 2024)

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

Handwritten Text Recognition for manuscripts and early printed texts

Driving Behavioral Change for Information Management through Data-Driven Gree...

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

2024: Domino Containers - The Next Step. News from the Domino Container commu...

GenCyber Cyber Security Day Presentation

Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guillaume Torch, Big Data Engineer - GumGum

1. Real Time Analytics at Scale with Guillaume Torche Saturday 9th July Big Data LA 2016

2. What is analytics at scale ?

3. {"id":"165a0b33-fd2d-4f45-83e4-a373cc4e6333","t":1435633205023,"cl":"js","ua":"Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) GSA/6.0.51363 Mobile/12F70 Safari/600.1.4","ip":"70.109.46.30","cc":"US","rg":"CA","ct":"Santa Maria","pc":"93455","mc":855,"isp":"Verizon Internet Services","bf":"6c52d0492fa92bd912d3a53a3c864e5a47008779","vst":"62784988-5d30-40a2-a334- 43e456499ac6"},{"v":"1.1","e":"view","si":632,"t":"56c89c04","ab":25465,"pv":"6cff666d-92c5- 42ac-9f94-39a40f9f297d","pu":"http://www.nytimes.com/2016/01/08/business/international/a-new- economic-era-for-china-goes-off-the- rails.html?hp&action=click&pgtype=Homepage&clickSource=story-heading&module=first-column- region&region=top-news&WT.nav=top- news&_r=0","af":false,"rpm":0.65999997,"pc":0.25,"nc":0,"dsp":27,"dai":"pqdmnbu","dci":"rpz224 3","do":"*.nytimes.com","vi":3,"pi":1134,"pri":3,"uti":16,"adi":300,"ei":3,"spi":172,"atmi":20 8,"ct":"rtb","cli":3,"cid":1256}

4. What does Real Time mean?

5. What does Real Time mean? Response time Data freshness Response Time

6. Data Freshness

7. About me Guillaume Torche Born and raised in France First experience in Big Data at Aubay in Paris Started working for GumGum 2 years ago About one year and a half of experience with Druid

8. Invented In Image advertising in 2008 http://gumgum.com/gallery Processing 2.6B image impressions / month 10B impressions / month 10B events / day 2000 premium publishers 132 employees - 37.5% YOY growth

9. Interactive data exploration Fast results Access fresh data

10. Before we met Big Data MySQL for Real Time reporting Data aggregation with Storm Batch updates of MySQL metrics

11.

12. Brief of Druid Built by another Ad tech company Used by top tech companies

13. Key features Arbitrary Slice-N-Dice data Fast aggregations on time series data Highly available Real Time Distributed

14. Overall Architecture

15. Queries Broker Realtime Historical Stream Deep storage

16. Data Balance & Replication

17. Queries Broker Realtime Historical Stream Deep storage Coordinator

18. Data Ingestion

19. Queries Broker Realtime Historical Stream Deep storage Coordinator Batch Data Ingestion Ingestion

20. Why is Druid fast?

21. Column compression - Dictionary encoding Timestamp Product Vertical Ad impressions Clicks Revenue 2015-01-01T00:01:35Z In-Image Gaming 15 3 10 2015-01-04T00:03:48Z In-Screen Gaming 5 1 3 2015-01-03T00:08:05Z Native Fashion 2 0 0.5 2015-01-02T00:02:35Z In-Image Fashion 25 15 50 ● Create internal ids: ○ Gaming => 0, Fashion => 1 ● Store column: ○ Vertical => [ 0 0 1 1 ]

22. Inverted Index Row Vertical 0 Gaming 1 Gaming 2 Fashion 3 Fashion Gaming [1, 1, 0, 0] Fashion [0, 0, 1, 1] Gaming OR Fashion [1, 1, 1, 1]

23. How does Druid achieve low latency ingestion?

24. Realtime Stream

25. Write optimized Read optimized Flush S1 S2 S3 S4 S5Final Segment Historical

26.

27.

28. https://www.linkedin.com/in/guillaume-torche-4a209790 http://imply.io/ http://druid.io http://gumgum.com/careers

Notes de l'éditeur

http://pivot.druid.va.ggops.com:9090/
http://pivot.druid.va.ggops.com:9090/

Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guillaume Torch, Big Data Engineer - GumGum

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (20)

Similaire à Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guillaume Torch, Big Data Engineer - GumGum

Similaire à Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guillaume Torch, Big Data Engineer - GumGum (20)

Plus de Data Con LA

Plus de Data Con LA (20)

Dernier

Dernier (20)

Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guillaume Torch, Big Data Engineer - GumGum

Notes de l'éditeur