Briefing room 20160913-ep0016-sap-anomalies-or-alerts-streaming-analytics-to-the-rescue-dez-slides

•Télécharger en tant que PPT, PDF•

0 j'aime•177 vues

Dez Blanchfield

Briefing room 20160913 - ep0016- s ap- nomalies or alerts streaming analytics to the rescue dez slides

Technologie

@dez_blanchfield
FAST vs REAL TIME
enter streaming analytics

@dez_blanchfield
Analytics.. From batch to real-time..
My top five use cases for analytics.. all of which are screaming out for real time
capabilities streaming analytics can bring about a fundamental transformation in
ways we’re still coming to grips with..
1. Fraud Detecting & Security
2. Compliance & Regulatory Reporting
3. Customer Segmentation
4. Operational Risk Management
5. Personalised Product Offering
The challenge now for organisations of all size and scale, is to select the best
Streaming Analytics tools & platforms to meet the needs of both their staff and
their customers

@dez_blanchfield
3 x forms of analytics focus
past
present
predictive

@dez_blanchfield
Aviation.. One of my favorite use cases..

@dez_blanchfield
When life was simpler..

@dez_blanchfield
Analytics was simpler..

@dez_blanchfield
the last decade has seen the
worlds technology develop at
a pace difficult to comprehend

@dez_blanchfield
Beyond ‘simple’ scale & complexity..

@dez_blanchfield
it’s been reported that the
Boeing 787 Dreamliner has
1,000’s of sensors per wing

@dez_blanchfield
Present.. Life the Universe & Everything..
The Boeing Dreamliner 787 is reported to produce 500 Gigabytes of data per
flight, from thousands of sensors in every part of the aircraft, front back left and
right..
• 500 Gigabytes of new data generated per flight ( let’s call it half a Terabyte )
• 87,400 flights per day in the domestic airspace over the USA alone ( *NOAA )
• 87,400 x 0.5 Terabytes = 43,700 Terabytes approx.
• Let’s round that down to 42 Petabytes approx for easy math
• That’s 42 Petabyes of data per day! The meaning of Life, the Universe & Everything
• Traditional big data analytics simply can’t scale to cope with these volumes
* The USA’s National Oceanic & Atmospheric Administration estimate that on any given day, some 87,400 flights are in the skies in the United States

@dez_blanchfield
Why is Streaming so different?
The transition from Batch Mode processing to Stream Analytics is still a real
challenge for most organizations to grasp and put into effect, as it’s a paradigm
shift few have seen before outside the likes of Telecoms or Military industries..
• Batch Analytics
• High latency applications
• Static Data / Databases
• Store then Process
• Batch processes
• Stream Analytics
• Low latency applications
• High volume Event streams
• Real time Sense & respond
• Plug-in Stream processors
• In stream processing

@dez_blanchfield
Streaming Analytics is the new Black..
When we applied the 3x V’s of Volume, Variety and Velocity at what’s referred to as
‘WebScale’ we got Big Data.. Now we need to address the explosion of IoT coming
at us like a freight train, and without Streaming Analytics we are destined to fail..

@dez_blanchfield
organizations should move “full
stream ahead” to adopt streaming
analytics solutions if they wish to
offer their staff a bona fide ability to
meet the need for real-time data
and insights in decision making

Contenu connexe

Tendances

Polyglot Processing - An Introduction 1.0

Dr. Mohan K. Bavirisetty

There are many modern techniques for identifying anomalies in datasets. There are fewer that work as online algorithms suitable for application to real-time streaming data. What’s worse? Most of these methodologies require a deep understanding of the data itself. In this talk, we tour what the options are for identifying anomalies in real-time data and discuss how much we really need to know before hand to guess at the ever-useful question: is this normal?

Is this normal?

Theo Schlossnagle

Although both disciplines are unique in their own ways, Software Engineering and Data Science make heavy use of programing languages to do their respective jobs. Data Science is a relatively new discipline and many of its practitioners have not previously been professional software engineers. There are a few techniques that Data Scientists can leverage from Software Engineering in order to make their tooling and environments, faster to design, more easily debugged and most importantly, clearer to read. This talk will be going over some practical tips that anyone can use to help better understand their code; give clarity around cloud environments, their uses and drawbacks and finally briefly touching on the Software Development Lifecycle.

Software Engineering for Data Scientists

Domino Data Lab

DataHub

Aditya Parameswaran

Principles of System Observability

Janis Orlovs

American Diabetic association states that 29.1 million Americans and 300+ million people all over the world have diabetes. Diabetic medication management is always challenging. Based on doctor’s prescription, patients take insulin dosage one hour before breakfast, lunch or dinner. But the real world scenario insulin intake can be changed based on the blood glucose level, calorie intake on a specific day, etc. This talk explains how a real-time Big Data pipeline recommendation engine can be used to suggest insulin intake for diabetic patients in near real time. Based on calorie intake and blood glucose level from patients as well as generated dataset, insulin dosage can be recommended which will help patients to avoid over/under dosage. Designing medication recommender system is a need for the Healthcare industry. There is a growing trend for the applications to help doctors by recommending medication based on patient’s historic data. This also helps facilitate a doctor friendly and hospital free atmosphere for all users all over the world. This talk would delve into a Diabetes medication recommender system using Databricks and Spark. Databricks supports HIPAA compliant deployment for processing PHI data. This talk would cover building a secure pipeline with encrypted data and the end-to-end recommendation system using Structured streaming and IoT data flowing from sensor.

Building Real-Time Data Pipeline for Diabetes Medication Recommender System U...

Databricks

The data science process seeks to transform and empower organizations by finding and exploiting market inefficiencies and potentially hidden opportunities, but this is often an expensive, tedious process. However, many steps can be automated to provide a streamlined experience for data scientists. Eduardo Arino de la Rubia explores the tools being created by the open source community to free data scientists from tedium, enabling them to work on the high-value aspects of insight creation and impact validation. The promise of the automated statistician is almost as old as statistics itself. From the creations of vast tables, which saved the labor of calculation, to modern tools which automatically mine datasets for correlations, there has been a considerable amount of advancement in this field. Eduardo compares and contrasts a number of open source tools, including TPOT and auto-sklearn for automated model generation and scikit-feature for feature generation and other aspects of the data science workflow, evaluates their results, and discusses their place in the modern data science workflow. Along the way, Eduardo outlines the pitfalls of automated data science and applications of the “no free lunch” theorem and dives into alternate approaches, such as end-to-end deep learning, which seek to leverage massive-scale computing and architectures to handle automatic generation of features and advanced models.

Leveraging Open Source Automated Data Science Tools

Domino Data Lab

Presented at the Global Pharma R&D Informatics Congress. To find out more, visit: www.global-engage.com Analytical data is at the heart of pharmaceutical research, yet many organisations struggle with the variety of different formats, instrument vendors, and search and retrieval of data. In this presentation, Hans de Bie from ACD/Labs discusses automated capture, exchange formats, integrity, and next generation management systems.

The challenges of Analytical Data Management in R&D

Laura Berry

Do Agile Data in Just 5 Shocking Steps!

DataKitchen

Tools for Unstructured Data Analytics

Ravi Teja

Dataiku - for Data Geek Paris@Criteo - Close the Data Circle

Dataiku

Fraud is prevalent in every industry, and growing at an increasing rate, as the volume of transactions increases with automation. The National Healthcare Anti-Fraud Association estimates $350B of fraudulent spending. Forbes estimates $25B spending by US banks on anti-money laundering compliance. At the same time as fraud and anomaly detection use cases are booming, the skills gap of expert data scientists available to perform fraud detection is widening. The Kavi Global team will present a cloud native, wizard-driven AI anomaly detection solution, enabling Citizen Data Scientists to easily create anomaly detection models to automatically flag Collective, Contextual, and Point anomalies, at the transaction level, as well as collusion between actors. Unsupervised methods (Distribution, Clustering, Association, Sequencing, Historical Occurrence, Custom Rules) and supervised (Random Forest, Neural Network) models are executed in Apache Spark on Databricks. An innovative aggregation framework converts probabilistic fraud scores and their probabilities into a meaningful and actionable prioritized list of suspicious (a statistical outlier) and potentially fraudulent transaction to be investigated from a business point of view. The AI Anomaly Detection models improve over time using Human-in-the-Loop feedback methods to label data for supervised modeling. Finally, The Kavi team overviews the Anomaly Lifecycle: from statistical outlier to validated business fraud for reclaim and business process changes to long term prevention strategies using proactive audits upstream at the time of estimate to prevent revenue leakage. Two client success stories will be presented acros Pharmaceutical Rx and Transportation industries.

Wizard Driven AI Anomaly Detection with Databricks in Azure

Databricks

Big data

Pietro Nardone

Artificial Intelligence and Data-centric businesses by Óscar Méndez at Big Da...

Big Data Spain

The rapid expansion of mobile phone usage in low-income and middle-income countries has created unprecedented opportunities for applying AI to improve individual and population health. In benshi.ai, a non-profit funded by the Bill and Melinda Gates Foundation, the goal is to transform health outcomes in resource-poor countries through advanced AI applications. We aim to do so by providing personalized predictions and recommendations to support diagnosis to medical care teams and frontline workers, as well as to nudge patients through personalized incentives towards an improvement in disease treatment management and general wellness. To this end, we have built an operational machine learning platform that provides personalized content and interventions real-time. Multiple engineering and machine learning decisions have been made to overcome different challenges and to build an experimentation engine and a centralized data and model management system for global health. Databricks served as a cornerstone upon which all our data/ML services were built. In particular, MLflow and dbx (an opensource tool from Databricks) have been crucial for the training, tracking and management of our end-to-end model pipelines. From the data science perspective, our challenges involved causal inference analysis, behavioral time series forecasting, micro-randomized trials, and contextual bandits-based experimentation at the individual level. This talk will focus on how we overcome the technical challenges to build a state-of-the-art machine learning platform that serves to improve global health outcomes.

Towards Personalization in Global Digital Health

Databricks

6 levels of big data analytics applications

panoratio

Tendances (16)

Polyglot Processing - An Introduction 1.0

Is this normal?

Software Engineering for Data Scientists

DataHub

Principles of System Observability

Building Real-Time Data Pipeline for Diabetes Medication Recommender System U...

Leveraging Open Source Automated Data Science Tools

The challenges of Analytical Data Management in R&D

Do Agile Data in Just 5 Shocking Steps!

Tools for Unstructured Data Analytics

Dataiku - for Data Geek Paris@Criteo - Close the Data Circle

Wizard Driven AI Anomaly Detection with Databricks in Azure

Big data

Artificial Intelligence and Data-centric businesses by Óscar Méndez at Big Da...

Towards Personalization in Global Digital Health

6 levels of big data analytics applications

En vedette

'Bases genómicas del cáncer. hacia una medicina personalizada del cáncer de m...

Real Academia de Medicina y Ciencias Afines de la Comunidad Valenciana

OpenStack Australia Government Day 2016 - Dez Blanchfield

Dez Blanchfield

Fence Education Academy offer overseas educational consulting services for students who are interested to study medicine in the Philippines. Fence Education Academy was founded on 2006 and has fulfilled dreams of more than 1000 students. With our well qualified and experienced group of Professionals, we ease of the complexity in getting admission for students and their parents to ensure that their vision are aligned with the purpose of education they look for. As experts in medical counselling for Philippines education, our professional counselling team ensures that you get absolute clarity in all aspects and have great study experience. We have partnered with various universities in Philippines which offer exciting courses and excellent facilities and faculties with reasonable fees. Our counsellors can guide students and parents through the entire process of arranging studies abroad from selecting a course and ensure that you arrive safely at your chosen destination. Our organization has a single-minded focus, to guide students to the best possible Institutions, thus enabling them to build their future. Keywords : Top Medical University, Best Medical Education, MD Admission Program Link : https://mbbsinphilippinesperpe... Youtube : https://www.youtube.com/watch?v=Ydrd4PdlKs4

Fence education academy

STUDY DOCTOR OF MEDICINE

SAP MM Online Training Course

Venkat reddy

Андреја Николовска VII-2

Marija Piskacheva

Telemedicine, despite being an old subject, is presently receiving a huge push from government to address the healthcare inadequacy in India. The speciality health infrastructure is a need of the hour and presents an opportunity for telecom vendors, healthcare providers and policy makers to provide healthcare to masses. This document identifies the opportunity in telemedicine and indicates the efforts so far.

Telemedicine: An opportunity in Healthcare in India

Amit Bhargava

En vedette (6)

'Bases genómicas del cáncer. hacia una medicina personalizada del cáncer de m...

OpenStack Australia Government Day 2016 - Dez Blanchfield

Fence education academy

SAP MM Online Training Course

Андреја Николовска VII-2

Telemedicine: An opportunity in Healthcare in India

Similaire à Briefing room 20160913-ep0016-sap-anomalies-or-alerts-streaming-analytics-to-the-rescue-dez-slides

The Briefing Room with Dr. Robin Bloor and WebAction Live Webcast on July 23, 2014 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=360d371d3a49ad256942f55350aa0a8b The waiting used to be the hardest part, but not anymore. Today’s cutting-edge enterprises can seize opportunities faster than ever, thanks to an array of technologies that enable real-time responsiveness across the spectrum of business processes. Early adopters are solving critical business challenges by enabling the rapid-fire design, development and production of very specific applications. Functionality can range from improved customer engagement to dynamic machine-to-machine interactions. Register for this episode of The Briefing Room to learn from veteran Analyst Dr. Robin Bloor, who will tout a new era in data-driven organizations, and why a data flow architecture will soon be critical for industry leaders. He’ll be briefed by Sami Akbay of WebAction, who will showcase his company’s real-time data management platform, which combines all the component parts needed to access, process and leverage data big and small. He’ll explain how this new approach can provide game-changing power to organizations of all types and sizes. Visit InsideAnlaysis.com for more information.

Take Action: The New Reality of Data-Driven Business

Inside Analysis

Big Data LDN 2017: The New Dominant Companies Are Running on Data

Matt Stubbs

Big Data LDN 2017: The New Dominant Companies Are Running on Data

Matt Stubbs

The cost of Digital Transformation is dropping rapidly. The technologies and methodologies are evolving to open up new opportunities for new and established corporations to drive business. We will examine specific examples of how and why a combination of robust infrastructure, cloud first and machine learning can take your company to the next level of value and efficiency. Rich Dill, SnapLogic's enterprise solutions architect, at Big Data LDN 2017.

The new dominant companies are running on data

SnapLogic

The Briefing Room with David Loshin and Datawatch Live Webcast Feb. 17, 2015 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=4a053043c45cf0c2f6453dfb8577c72a Patience may be a virtue, but when it comes to streaming analytics, waiting is no option. Between Big Data and the Internet of Things, businesses are faced with more data and greater complexity than ever before. Traditional information architectures simply cannot support the kind of processing necessary to make use of this fast-moving resource. The modern context requires a shorter path to analytics, one that narrows the gap between governance and discovery Register for this episode of The Briefing Room to hear veteran Analyst David Loshin as he explains how the prevalence of streaming data is changing business pace and processes. He’ll be briefed by Dan Potter of Datawatch, who will tout his company’s real-time data discovery platform for data in motion. He will show how self-service data preparation can lead to faster insights, ultimately fostering the ability to make precise decisions at the right time. Visit InsideAnalysis.com for more information.

Moving Targets: Harnessing Real-time Value from Data in Motion

Inside Analysis

In-Memory Computing Webcast. Market Predictions 2017

SingleStore

Going Cloud First at the FT

Amazon Web Services

Big Data made easy in the era of the Cloud - Demi Ben-Ari

Demi Ben-Ari

IBM's Watson is a machine-learning platform that’s been built to mirror the same learning process that humans have: Observe, Interpret, Evaluate and Decide. Through the use of this cognitive framework, Watson can search through a database of information and pull out key insights to bridge gaps in human knowledge. It’s expertise scaling for enterprise. Watson has already helped businesses across a variety of industries increase their customer engagement, data discovery and informed decision making abilities. Is your business next?

Just ask Watson Seminar

Certus Solutions

SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"

MDS ap

The Briefing Room with Mark Madsen and WebAction Live Webcast Feb. 10, 2015 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=fa83c6283de99dfb6f38b9d7199cb452 In our increasingly interconnected world, the windows of opportunity for meaningful action are shrinking. Where hours once sufficed, minutes are now the norm. For some transactions, seconds make all the difference, even sub-seconds. Meeting these demands requires a new approach to information architecture, one that embraces the many innovations that are fundamentally changing the data-driven economy. Register for this episode of The Briefing Room to hear veteran Analyst Mark Madsen of Third Nature as he explains how a confluence of advances are changing the nature of data management. He'll be briefed by Sami Akbay of WebAction, who will showcase his company's real-time data platform, designed from the ground up to meet the challenges of leveraging Big Data in concert with all manner of operational enterprise systems. Visit InsideAnalysis.com for more information.

Time Difference: How Tomorrow's Companies Will Outpace Today's

Inside Analysis

Watch: https://bit.ly/2DYsUhD Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python and Scala put advanced techniques at the fingertips of the data scientists. However, these data scientists spent most of their time looking for the right data and massaging it into a usable format. Data virtualization offers a new alternative to address these issues in a more efficient and agile way. Attend this webinar and learn: - How data virtualization can accelerate data acquisition and massaging, providing the data scientist with a powerful tool to complement their practice - How popular tools from the data science ecosystem: Spark, Python, Zeppelin, Jupyter, etc. integrate with Denodo - How you can use the Denodo Platform with large data volumes in an efficient way - How Prologis accelerated their use of Machine Learning with data virtualization

Advanced Analytics and Machine Learning with Data Virtualization

Denodo

Invited Talk at Modern Data Management Systems Summit on August 29-30, 2014 at Tsinghua University in Beijing, China. http://ise.thss.tsinghua.edu.cn/MDMS/English/program.jsp Abstract: Modern enterprises are increasingly relying on complex analyses on large data sets to drive business decisions. Tasks such as root cause analysis from system logs and lead generation based on social media, customer retention and digital marketing are rapidly gaining importance. These applications generally consist of three major analytic phases: text analytics, semi-structured data processing (joins, group-by, aggregation), and statistical/predictive modeling. The size of the datasets in conjunction with the complexity of the analysis necessitates large-scale distributed processing of the analytical algorithms. At IBM we are building tools and technologies based on declarative languages to support each of these analytic phases. The declarative nature of the language abstracts away the need for programmer-optimization. Furthermore, the syntax of these languages is designed to appeal to the corresponding communities. As an example for statistical modeling, we expose a high-level language with syntax similar to R -- a very popular statistical processing language. In this talk I will give an overview of some real-world big data applications we are currently working on and use that to motivate the need for declarative analytics consisting of the three major phases discussed above. I will then describe, in some detail, declarative systems for text analytics along with a discussion on speeds, feeds and comparisons.

The Power of Declarative Analytics

Yunyao Li

The Briefing Room with Dean Abbott and Tableau Software Live Webcast July 23, 2013 http://www.insideanalysis.com Today’s desire for analytics extends well beyond the traditional domain of Business Intelligence. That’s partly because business users are realizing the value of mixing and matching all kinds of data, from all kinds of sources. One emerging market driver is Cloud-based data, and the desire companies have to analyze this data cohesively with their on-premise data sets. Register for this episode of The Briefing Room to learn from Analyst Dean Abbott, who will explain how the ability to access data in the cloud can play a critical role for generating business value from analytics. He’ll be briefed by Ellie Fields of Tableau Software who will tout Tableau’s latest release, which includes native connectors to cloud-based applications like Salesforce.com, Amazon Redshift, Google Analytics and BigQuery. She’ll also demonstrate how Tableau can combine cloud data with other data sources, including spreadsheets, databases, cubes and even Big Data.

Bridging the Gap: Analyzing Data in and Below the Cloud

Inside Analysis

The data revolution is well underway. Regardless of the industry or department you manage, working with data will soon be an essential part of all of our jobs, if it isn’t already. This could take the form of basic data analytics, data science, machine learning or artificial intelligence. This can be overwhelming: what do all these terms mean and how can they be leveraged to impact your employees’ work, whether that be in finance, healthcare, tech or the public sector, among many others? This webinar will give you a primer for understanding how data can impact your employees’ work, what they need to know and how to go about educating them on it.

What your employees need to learn to work with data in the 21 st century

Human Capital Media

Big data and APIs for PHP developers - SXSW 2011

Eli White

Qo Introduction V2

Joe_F

Tim Estes - Information Systems in an Entity Centric World

Digital Reasoning

From IoT to IoTA

Striim

Harness the Power of Big Data with Oracle

Sai Janakiram Penumuru

Similaire à Briefing room 20160913-ep0016-sap-anomalies-or-alerts-streaming-analytics-to-the-rescue-dez-slides (20)

Take Action: The New Reality of Data-Driven Business

Big Data LDN 2017: The New Dominant Companies Are Running on Data

The new dominant companies are running on data

Moving Targets: Harnessing Real-time Value from Data in Motion

In-Memory Computing Webcast. Market Predictions 2017

Going Cloud First at the FT

Big Data made easy in the era of the Cloud - Demi Ben-Ari

Just ask Watson Seminar

SAP Forum Ankara 2017 - "Verinin Merkezine Seyahat"

Time Difference: How Tomorrow's Companies Will Outpace Today's

Advanced Analytics and Machine Learning with Data Virtualization

The Power of Declarative Analytics

Bridging the Gap: Analyzing Data in and Below the Cloud

What your employees need to learn to work with data in the 21 st century

Big data and APIs for PHP developers - SXSW 2011

Qo Introduction V2

Tim Estes - Information Systems in an Entity Centric World

From IoT to IoTA

Harness the Power of Big Data with Oracle

Plus de Dez Blanchfield

Hot tech 20170329-idera - health check - maintaining enterprise bi-dez-slides

Dez Blanchfield

CDO Summit 2017 - Sydney - 20170315 - Dez Blanchfield

Dez Blanchfield

Briefing Room 20161213 - ep019 - Red Hat - Modern Business Storage

Dez Blanchfield

Young it - digital transformation on a personal level

Dez Blanchfield

Hot tech 20161221 - ep0022 - IDERA - an ounce of prevention - Forging Healthy BI

Dez Blanchfield

Smart Cities Expo - World Forum - Sydney - 2016 - Dez Blanchfield

Dez Blanchfield

Hot tech 20161207 - ep0021 - IDERA - Protect your database - High availabilit...

Dez Blanchfield

Hot tech 20161116-ep0019-idera - data modeling in an agile environment-dez-sl...

Dez Blanchfield

Hot tech 20160602-ep008-idera - forward momentum - moving relational beyond t...

Dez Blanchfield

Hot tech 20161005-ep0016-idera - index insanity - how to avoid database chaos...

Dez Blanchfield

Hot tech 20160510-ep005-magnitude software-the_biggest_picture-knowing_your_c...

Dez Blanchfield

Hot tech 20160922-ep0015-dell statistica - edge analytics - the io_t economy ...

Dez Blanchfield

Briefing room 20160920-ep017-striim - a real-time version of the truth-dez-sl...

Dez Blanchfield

Hot tech 20161102 - ep0018 - idera - application acceleration - faster perfor...

Dez Blanchfield

Health Data Management - Clear Data - 5 reasons hospital CIOs are extending t...

Dez Blanchfield

Hot tech 20160602 - ep007 - sync sort - big iron meet big data - liberating m...

Dez Blanchfield

Hot tech 20160914-ep0014-idera - who what where and how - why you want to kno...

Dez Blanchfield

Hot tech 20160825-ep0012-dell statistica-embed-analytics-everywhere-enabling-...

Dez Blanchfield

Hot tech 20160824 - ep0011 - idera - the art of visibility - enabling multi p...

Dez Blanchfield

Equinix Insurance - 2016Round Table Series - 20160728

Dez Blanchfield

Plus de Dez Blanchfield (20)