Big data and messaging with Spring

•Télécharger en tant que PPTX, PDF•

5 j'aime•1,636 vues

The document discusses messaging and big data applications. It introduces Spring Integration and Spring XD, which are frameworks for building distributed applications and processing streaming data. Spring XD provides a unified platform for ingesting, processing, and exporting big data through streams and jobs. It uses messaging and is distributed, extensible, and works with various data stores and analytics. The document concludes with information on learning more about related Spring projects.

Technologie

Big Data & Messaging

with Artem Bilan
by Pivotal
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Artem Bilan
Spring Integration Team

abilan@gopivotal.com

Spring Framework
Spring AMQP
Spring XD
Reactor

https://spring.io/team/artembilan

http://stackoverflow.com/users/2756547

http://www.linkedin.com/in/cleric

https://github.com/artembilan

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Messaging and why do I care?

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Messaging is very simple

Headers

Payload

public static void main(String[] args) {
...
}
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Messaging in Spring IO

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Spring Integration
Written
Uses

Meet IoC!

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

The heart of Spring Integration
–
–
–

Endpoints (Filters) connected through
Channels (Pipes) exchanging
Message

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Spring Integration Message Flow

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

How does it work?

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

What is a Big Data application?
 Not just writing a few MR or Hive/Pig jobs.
 The full lifecycle involves
• Ingestion
• Stream Processing
• Workflow Orchestration
• Enterprise Integration
• Export
• Horizontal scalable deployment
 How do you write one of these?
• Status quo is to combine different projects – not ideal
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Spring XD
• Unified Platform
• Developer Productivity
• Modular Extensibility
• Distributed Architecture
• Portable Runtime
• Hadoop Distribution Agnostic
• Proven Foundation
• XD = ‘eXtreme Data’
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Spring XD
Files

Sensors

Mobile

Social

Spring XD Shell
Spring XD Runtime

Taps

Jobs

Compute

Workflow

Redis

Ingest

Streams

Export

Export

RDBMS

Gemfire

HDFS

NoSQL

R, SAS

Predictive modeling
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Stream Processing Model

How can
http |we make this|easier?
filter
file

Non-linear stream definitions also supported
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Runtimes
XD Admin

http | filter | file

CLUSTERED NODE

CLUSTERED NODE

CLUSTERED NODE

SINGLE
NODE

HTTP
Module

Filter
Module

File
Module

All
Modules

Rabbit, Redis, (Pluggable)
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

In Memory
Transport

http | filter | file

Streams
MessageStore

HTTP
Tail
File
Mail
Twitter
Gemfire
Syslog
TCP
JMS
RabbitMQ
MQTT

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Filter
Transformer
Splitter
Aggregator
Groovy Script
Counters
HTTP
JSON
Java Code

File
HDFS
JDBC
TCP
Mail
RabbitMQ
Gemfire
Splunk
MQTT
Dynamic Router

Taps
 “Listen” to data from another stream
• Other stream is unaffected by the tap and unaware of its presence
• EAI ‘Wiretap’

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Analytics
• Simple Counter
• Field Value Counter
– Count occurrences of named
fields

• Aggregate Counter
– Pre-aggregate counts in time
buckets

• Gauge
– last value

• Rich Gauge
– Last value, running
average, min/max
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

• Abstract API
• Implementations for
– Memory
– Redis

• Looking at
– Pattern
– JPMML

Jobs
 Jobs are a directed graph of steps
 Steps
• Copy or Process data
• Files, Databases, MR, Pig, Hive, Cascading

 Step executions are persisted
• Checkpointing with restart
• Rich error handling capabilities
 Single node or distributed with data partitioning
 Jobs can be Triggered from streams
 Executing jobs generate a stream of event data

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

SPRING XD

Demo

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Learn More. Stay Connected.
•
•
•
•
•
•
•
•

Spring IO: https://spring.io/platform
GitHub: https://github.com/spring-projects
Spring Integration: http://projects.spring.io/spring-integration
Spring XD: http://projects.spring.io/spring-xd
Reactor: https://github.com/reactor
EIP: http://www.eaipatterns.com
Spring Batch: http://projects.spring.io/spring-batch
Spring for Hadoop: http://projects.spring.io/spring-hadoop

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

ありがとう
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Recommandé

Spring & messaging

Artem Bilan

SpringPeople Introduction to HTML5 & CSS3

SpringPeople

BeAPI API Framework

Owen Rubel

LEVERAGING BIG DATA FOR PUBLIC SECTOR VALUE, PRODUCTIVITY, TRANSFORMATION & I...

TMC

Relying on Data for Strategic Decision-Making--Financial Services Experience

Cloudera, Inc.

This document discusses how financial services companies can leverage big data and machine learning to drive strategic decision making through improved customer insights and risk management. It outlines key data challenges around data silos, volume, and costs. Example use cases are provided for building customer 360 profiles and improving fraud detection. The presentation argues that an enterprise data hub on Cloudera can help integrate diverse data sources, power real-time analytics at scale, and enable new business opportunities in areas like customer experience and risk compliance.

Get Results, Build Your Own Big Data Beast : Greenplum + Dell

skahler

One Hadoop, Multiple Clouds

Cloudera, Inc.

The document discusses Cloudera Director, a tool for deploying and managing Hadoop clusters across multiple public clouds. It begins with an introduction of the speaker and outlines some common architectures for running Hadoop in the cloud. Key points covered include cluster lifecycle management, elasticity, high availability, backup/disaster recovery, and a vision for the future of portable, self-service Hadoop experiences across clouds.

Is your big data journey stalling? Take the Leap with Capgemini and Cloudera

Cloudera, Inc.

Recommandé

Spring & messaging

Artem Bilan

SpringPeople Introduction to HTML5 & CSS3

SpringPeople

BeAPI API Framework

Owen Rubel

LEVERAGING BIG DATA FOR PUBLIC SECTOR VALUE, PRODUCTIVITY, TRANSFORMATION & I...

TMC

Relying on Data for Strategic Decision-Making--Financial Services Experience

Cloudera, Inc.

Get Results, Build Your Own Big Data Beast : Greenplum + Dell

skahler

One Hadoop, Multiple Clouds

Cloudera, Inc.

Is your big data journey stalling? Take the Leap with Capgemini and Cloudera

Cloudera, Inc.

Presentation a pivotal overview

xKinAnx

Guest Lecture on Big Data in Business,

saravana krishnamurthy

Pivotal OSS meetup - MADlib and PivotalR

go-pivotal

With the explosion of big data, the need for fast and inexpensive analytics solutions has become a key basis of competition in many industries. Extracting the value of big data with analytics can be complex, and requires advanced skills. At Pivotal, we are building open-source solutions (MADlib, PivotalR, PyMadlib) to simplify this process for the user, while maintaining the efficiency necessary for big data analysis. This talk will provide information about MADlib, an open source library of SQL-based algorithms for machine learning, data mining and statistics that run at large scale within a database engine, with no need for data import/export to other tools. It provides an overview of the library’s architecture and compares various statistical methods with those available in Apache Mahout. We also introduce, PivotalR, a R-based wrapper for MADlib that allows data scientists and programmers to access power of MADlib along with the ease of use of R.

Scale Out Your Big Data Apps: The Latest on Pivotal GemFire and GemFire XD

VMware Tanzu

Companies across all industries and sizes are investing in strategic custom applications to enhance their competitive advantages. Developing these applications requires continuous improvement, based on insights gleaned from collecting and analyzing the data that they generate. Big Data for high-performing, scalable and reliable applications requires a new set of tools and technologies. Pivotal GemFire is a distributed in-memory NoSQL data management solution for creating high-scale custom applications. Pivotal GemFire XD supports structured data as part the industry’s first Hadoop-based platform for creating closed loop analytics solutions – enabling businesses to continuously optimize real-time automation in their applications.

Accelerating Operational Excellence in 2015: Calculating the ROI of Real-Time...

Catavolt, Inc.

This document discusses how real-time data access on mobile devices can help manufacturing organizations improve operational excellence. It provides examples of how real-time data can streamline maintenance processes, quality assurance, safety inspections, and reporting. The document argues that replacing paper-based processes with digital tools allows manufacturing teams to address issues more quickly and make real-time decisions that improve productivity, customer satisfaction and ROI.

Pivotal Digital Transformation Forum: Data Science Technical Overview

VMware Tanzu

This document provides an overview of Pivotal's data science capabilities and tools. It discusses how Pivotal uses an Agile approach to data science projects, focusing on frequent interactions with customers. Pivotal's software stack is designed to enable data science work by supporting real-time, interactive, and batch operations on data through tools like Spring XD, Pivotal HD, and GemFire. Examples are provided of how Pivotal has used these tools for applications like connected cars and scalable video analytics.

Operationalizing Data Analytics

VMware Tanzu

Hadoop is regarded as a key capability for implementing Big Data initiatives in the enterprise, but organizations have yet to realize its full business benefits. In this webinar, Pivotal and guest Forrester Research, Inc. Identify the use cases driving Hadoop adoption, and explore what is needed to transform initial investments into results. Learn about: Challenges Hadoop introduces, and how the right tools and platforms can help address them Shifts in the industry with regards to SQL and NoSQL systems and their implications to Big Data analytics Applying in-memory technologies for data management systems, data analytics, transactional processing and operational databases Watch the on-demand webinar here: http://www.pivotal.io/big-data/pivotal-forrester-operationalizing-data-analytics-webinar Learn how to maximize business value from all of your data here: http://www.pivotal.io/big-data/pivotal-hd

Pivotal the new_pivotal_big_data_suite_-_revolutionary_foundation_to_leverage...

EMC

The document discusses Pivotal's big data suite and business data lake offerings. It provides an overview of the components of a business data lake, including storage, ingestion, distillation, processing, unified data management, and action components. It also defines various data processing approaches like streaming, micro-batching, batch, and real-time response. The goal is to help organizations build analytics and transactional applications on big data to drive business insights and revenue.

Why Are Digital Disruptors Successful And How Can You Become One?

VMware Tanzu

Data as the New Oil: Producing Value in the Oil and Gas Industry

VMware Tanzu

Oil and gas exploration and production activities generate large amounts of data from sensors, logistics, business operations and more. Given the data volume, variety and velocity, gaining actionable and relevant insights from the data is challenging. Learn about these challenges and how to address them by leveraging big data technologies in this webinar. During the webinar we will dive deep into approaches for predicting drilling equipment function and failure, a key step towards zero unplanned downtime. In the process of drilling wells, non-productive time due to drilling equipment failure can be expensive. We will highlight how the Pivotal Data Labs team uses big data technologies to build models for predicting drilling equipment function and failure. Models such as these can be used to build essential early warning systems to reduce costs and minimize unplanned downtime. Panelist: Rashmi Raghu, Senior Data Scientist, Pivotal Hosted by: Tim Matteson, Co-Founder -- Data Science Central Video replay is available to watch here: http://youtu.be/dhT-tjHCr9E

Modern Big Data Analytics Tools: An Overview

Great Wide Open

This document provides an overview of modern big data analytics tools. It begins with background on the author and a brief history of Hadoop. It then discusses the growth of the Hadoop ecosystem from early projects like HDFS and MapReduce to a large number of Apache projects and commercial tools. It provides examples of companies and organizations using Hadoop. It also outlines concepts like SQL on Hadoop, in-database analytics using MADLib, and the evolution of Hadoop beyond MapReduce with the introduction of YARN. Finally, it discusses new frameworks being built on top of YARN for interactive, streaming, graph and other types of processing.

Women who wrote the analytics book final

metabrown

Pivotal Data Lake Architecture & its role in security analytics

EMC

The document discusses using data science and a data lake approach for information security analytics. It begins with an agenda that includes information security use cases, data lakes, and extracting value from data using data science. It then discusses challenges with existing security analytics approaches and how a data lake can provide a unified environment for diverse security data sources. It provides examples of security analytics use cases like malware detection. It also discusses tools and techniques for data science, like MADlib, and how data science drives more value from a security data lake than traditional approaches.

Pivotal Digital Transformation Forum: Data Science Bridging the Gap

VMware Tanzu

This document discusses how data science can bridge the gap between data generation and comprehension. It provides examples of smart apps that combine and link data from different sources and domains to infer patterns, identify root causes, and potentially improve outcomes in real-time. The document advocates adding smart capabilities to apps by leveraging data science and emphasizes collaborating across teams like product management and engineering rather than having isolated data science efforts.

Building a Modern Analytic Database with Cloudera 5.8

Cloudera, Inc.

This document discusses building a modern analytic database with Cloudera. It outlines Marketing Associates' evaluation of solutions to address challenges around managing massive and diverse data volumes. They selected Cloudera Enterprise to enable self-service BI and real-time analytics at lower costs than traditional databases. The solution has provided scalability, cost savings of over 90%, and improved security and compliance. Future roadmaps for Cloudera's analytic database include faster SQL, improved multitenancy, and deeper BI tool integration.

Realizing the Promise of Big Data with Hadoop - Cloudera Summer Webinar Serie...

Cloudera, Inc.

Apache Hadoop, an open-source platform, is increasingly gaining adoption within organizations trying to draw insight from all the big data being generated. Hadoop, and a handful of open-source tools that complement it, are promising to make gigantic and diverse datasets easily and economically available for quick analysis. A burgeoning partner ecosystem is also essential to helping organizations turn big data into business value.

2014 Big_Data_Forum_Pivotal

COMPUTEX TAIPEI

Customer Spotlight: How WellCare Accelerated Big Data Delivery to Improve Ana...

VMware Tanzu

WellCare uses Attunity Replicate to offload data quickly and easily from its SQL Server and Oracle systems into Pivotal Greenplum Database. With the help of Attunity and Pivotal, WellCare has successfully enabled real-time reporting and analytics to gain a competitive edge. Additional information: https://pivotal.io/big-data/webinar/healthcare-success-story-wellcare Webinar recording: https://youtu.be/ZYiwRUxlY2s

Modernizing Architecture for a Complete Data Strategy

Cloudera, Inc.

The document outlines a presentation about modernizing data strategies. It discusses how companies' relationships with data are changing and the business drivers for adopting big data and analytics. It then provides guidance on building a modern data strategy, emphasizing the importance of people, process, and technology. Specifically, it recommends starting with high-impact use cases, staying agile, and evolving capabilities over time to maximize value from data. The presentation also covers how Hadoop is being used for different workloads in both on-premise and cloud environments.

Becoming Data-Driven Through Cultural Change

Cloudera, Inc.

We've arrived at a crossroads. Big data is an initiative every business knows they should take on in order to evolve their business, but no one knows how to tackle the project. This is the first in a series of webinars that describe how to break down the challenge into three major pieces: People, Process, and Technology. We'll discuss the industry trends around big data projects, the pitfalls with adopting a modern data strategy, and how to avoid them by building a culture of data-driven teams.

Institutionalizing Open Source - Puneet Sachdev - Nasscom Tech Series - June ...

Puneet Sachdev

This document discusses strategies for adopting open source software in enterprises. It identifies three types of enterprise adopters - IT service vendors, product vendors, and end user enterprises. It outlines barriers to open source adoption like too many options and diverse licensing. The document recommends a structured evaluation process using a reference framework, prescriptive stacks, and centralized lifecycle management. It also discusses risks associated with different development and usage scenarios and the need for governance policies to ensure compliance.

Building Highly Scalable Spring Applications using In-Memory Data Grids

John Blum

Contenu connexe

En vedette

Presentation a pivotal overview

xKinAnx

Guest Lecture on Big Data in Business,

saravana krishnamurthy

Pivotal OSS meetup - MADlib and PivotalR

go-pivotal

Scale Out Your Big Data Apps: The Latest on Pivotal GemFire and GemFire XD

VMware Tanzu

Accelerating Operational Excellence in 2015: Calculating the ROI of Real-Time...

Catavolt, Inc.

Pivotal Digital Transformation Forum: Data Science Technical Overview

VMware Tanzu

Operationalizing Data Analytics

VMware Tanzu

Pivotal the new_pivotal_big_data_suite_-_revolutionary_foundation_to_leverage...

EMC

Why Are Digital Disruptors Successful And How Can You Become One?

VMware Tanzu

Data as the New Oil: Producing Value in the Oil and Gas Industry

VMware Tanzu

Modern Big Data Analytics Tools: An Overview

Great Wide Open

Women who wrote the analytics book final

metabrown

Pivotal Data Lake Architecture & its role in security analytics

EMC

Pivotal Digital Transformation Forum: Data Science Bridging the Gap

VMware Tanzu

Building a Modern Analytic Database with Cloudera 5.8

Cloudera, Inc.

Realizing the Promise of Big Data with Hadoop - Cloudera Summer Webinar Serie...

Cloudera, Inc.

2014 Big_Data_Forum_Pivotal

COMPUTEX TAIPEI

Customer Spotlight: How WellCare Accelerated Big Data Delivery to Improve Ana...

VMware Tanzu

Modernizing Architecture for a Complete Data Strategy

Cloudera, Inc.

Becoming Data-Driven Through Cultural Change

Cloudera, Inc.

En vedette (20)

Presentation a pivotal overview

Guest Lecture on Big Data in Business,

Pivotal OSS meetup - MADlib and PivotalR

Scale Out Your Big Data Apps: The Latest on Pivotal GemFire and GemFire XD

Accelerating Operational Excellence in 2015: Calculating the ROI of Real-Time...

Pivotal Digital Transformation Forum: Data Science Technical Overview

Operationalizing Data Analytics

Pivotal the new_pivotal_big_data_suite_-_revolutionary_foundation_to_leverage...

Why Are Digital Disruptors Successful And How Can You Become One?

Data as the New Oil: Producing Value in the Oil and Gas Industry

Modern Big Data Analytics Tools: An Overview

Women who wrote the analytics book final

Pivotal Data Lake Architecture & its role in security analytics

Pivotal Digital Transformation Forum: Data Science Bridging the Gap

Building a Modern Analytic Database with Cloudera 5.8

Realizing the Promise of Big Data with Hadoop - Cloudera Summer Webinar Serie...

2014 Big_Data_Forum_Pivotal

Customer Spotlight: How WellCare Accelerated Big Data Delivery to Improve Ana...

Modernizing Architecture for a Complete Data Strategy

Becoming Data-Driven Through Cultural Change

Similaire à Big data and messaging with Spring

Institutionalizing Open Source - Puneet Sachdev - Nasscom Tech Series - June ...

Puneet Sachdev

Building Highly Scalable Spring Applications using In-Memory Data Grids

John Blum

Asynchronous Event Streams – when java.util.stream met org.osgi.util.promise!...

mfrancis

OSGi Community Event 2015 Asynchronous and event-driven programming models are great for processing data, whether it's sensors in homes, or metrics from a web server farm, event streaming is a tool for analyzing data that can be produced at any time.</p> Whilst Java 8 Streams have begun to teach developers about streaming, it's still stuck with a pull model. How do you cope with push-based events without blocking and timeouts? Rx Java Observables, and the Actor pattern provide a different abstraction, but wouldn't it be great to keep the simplicity of Streams with the power of asynchronous push-based events? This talk will describe the prototyping work happening in OSGi's Distributed Eventing RFC, using streams and promises to build event processing pipelines. The OSGi service platform has existed as a modular micro-service runtime for well over a decade, but more recently it has embraced asynchronous programming as a core part of the OSGi toolkit. With the introduction of the Async Service, OSGi promises it is now easier than ever to build asynchronous applications. The focus so far has been on asynchronous requests, but what about streaming systems, ones that need to process or filter thousands, millions, or billions of events before an answer is reached, or perhaps there is no end to the stream at all! Processing distributed events is the purpose of the Distributed Eventing RFC being discussed by the OSGi Enterprise Expert Group. As the lead author of the RFC, and the lead of the Asynchronous Service specification, Tim has a detailed understanding of the complexities of Asynchronous programming models. Tim will be able to describe the current prototyping around OSGi's Asynchronous Streaming model, and how it fits with the other popular asynchronous standards.

IoT architecture

Sumit Sharma

How do APIs and IoT relate? The answer is not as simple as merely adding an API on top of a dumb device, but rather about understanding the architectural patterns for implementing an IoT fabric. There are typically two or three trends: Exposing the device to a management framework Exposing that management framework to a business centric logic Exposing that business layer and data to end users. This last trend is the IoT stack, which involves a new shift in the separation of what stuff happens, where data lives and where the interface lies. For instance, it's a mix of architectural styles between cloud, APIs and native hardware/software configurations.

Get to know the browser better and write faster web apps

Lior Bar-On

The document discusses techniques for optimizing web page performance. It covers strategies related to the server, network, and browser. Regarding the network, it recommends reducing roundtrips, lazy loading resources, caching, and domain sharding. For the browser, it suggests postponing script execution, adding styles before elements, and offloading work to other threads. The document provides examples and recommends using tools like Chrome Dev Tools to analyze performance bottlenecks.

S2DS London 2015 - Hadoop Real World

Sean Roberts

Data Mesh Part 4 Monolith to Mesh

Jeffrey T. Pollock

This is Part 4 of the GoldenGate series on Data Mesh - a series of webinars helping customers understand how to move off of old-fashioned monolithic data integration architecture and get ready for more agile, cost-effective, event-driven solutions. The Data Mesh is a kind of Data Fabric that emphasizes business-led data products running on event-driven streaming architectures, serverless, and microservices based platforms. These emerging solutions are essential for enterprises that run data-driven services on multi-cloud, multi-vendor ecosystems. Join this session to get a fresh look at Data Mesh; we'll start with core architecture principles (vendor agnostic) and transition into detailed examples of how Oracle's GoldenGate platform is providing capabilities today. We will discuss essential technical characteristics of a Data Mesh solution, and the benefits that business owners can expect by moving IT in this direction. For more background on Data Mesh, Part 1, 2, and 3 are on the GoldenGate YouTube channel: https://www.youtube.com/playlist?list=PLbqmhpwYrlZJ-583p3KQGDAd6038i1ywe Webinar Speaker: Jeff Pollock, VP Product (https://www.linkedin.com/in/jtpollock/) Mr. Pollock is an expert technology leader for data platforms, big data, data integration and governance. Jeff has been CTO at California startups and a senior exec at Fortune 100 tech vendors. He is currently Oracle VP of Products and Cloud Services for Data Replication, Streaming Data and Database Migrations. While at IBM, he was head of all Information Integration, Replication and Governance products, and previously Jeff was an independent architect for US Defense Department, VP of Technology at Cerebra and CTO of Modulant – he has been engineering artificial intelligence based data platforms since 2001. As a business consultant, Mr. Pollock was a Head Architect at Ernst & Young’s Center for Technology Enablement. Jeff is also the author of “Semantic Web for Dummies” and "Adaptive Information,” a frequent keynote at industry conferences, author for books and industry journals, formerly a contributing member of W3C and OASIS, and an engineering instructor with UC Berkeley’s Extension for object-oriented systems, software development process and enterprise architecture.

Open APIs - Risks and Rewards (Øredev 2013)

Nordic APIs

Introducing Open APIs and the security risks involved and the great rewards that can be reaped. Going through the advantages of using and publishing APIs and how to get started, how to handle security risks with a "neo-security" stack and how Twitters API has been used to analyse Twitter use in Sweden. Lightning talk from Øredev 7 november 2013 in Malmö Sweden. Presented by Andreas Krohn, Travis Spencer and Hampus Brynolf. More information at http://nordicapis.com/oredev2013.

Node summit workshop

Shubhra Kar

HTTP/2 and a Faster Web

C4Media

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1yKnuxS. Omer Shapira introduces HTTP/2 (and SPDY), exploring the impact the protocol has on application design, and telling the story of LinkedIn adopting SPDY on its network infrastructure. Filmed at qconsf.com. Omer Shapira is an engineering manager at LinkedIn. He and his team are building scalable, low-latency traffic infrastructure to keep LinkedIn site fast.

A Pipeline for Distributed Topic and Sentiment Analysis of Tweets on Pivotal ...

Srivatsan Ramanujam

Unstructured data is everywhere - in the form of posts, status updates, bloglets or news feeds in social media or in the form of customer interactions Call Center CRM. While many organizations study and monitor social media for tracking brand value and targeting specific customer segments, in our experience blending the unstructured data with the structured data in supplementing data science models has been far more effective than working with it independently. In this talk we will show case an end-to-end topic and sentiment analysis pipeline we've built on the Pivotal Greenplum Database platform for Twitter feeds from GNIP, using open source tools like MADlib and PL/Python. We've used this pipeline to build regression models to predict commodity futures from tweets and in enhancing churn models for telecom through topic and sentiment analysis of call center transcripts. All of this was possible because of the flexibility and extensibility of the platform we worked with.

Operational Machine Learning: Using Microsoft Technologies for Applied Data S...

Khalid Salama

This document provides an overview of Microsoft technologies for operational machine learning and data science. It introduces concepts of data science and machine learning, then discusses how to move from experimental to operational machine learning. It outlines various Microsoft technologies for advanced analytics including Azure Machine Learning, Microsoft R Server, SQL Server R Services, SQL Server Analysis Services, Azure Cognitive Services, and Spark ML on HDInsight. For each technology, it provides brief descriptions and examples of capabilities. The document aims to help users understand how to apply these Microsoft technologies for data science and machine learning projects.

Web Architecture - Mechanism and Threats

Sumedt Jitpukdebodin

This document discusses web architecture, mechanisms, and threats. It describes the basic components of web architecture including HTML, HTTP, URIs, cookies, and the three-tier architecture. It also outlines some common web attacks like injection, XSS, and CSRF. Finally, it discusses security controls at the application layer through input validation, authentication, and sessions management and at the network layer using firewalls, IDS, WAF, and centralized logging.

Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)

Rittman Analytics

Oracle Data Integration Platform is a cornerstone for big data solutions that provides five core capabilities: business continuity, data movement, data transformation, data governance, and streaming data handling. It includes eight core products that can operate in the cloud or on-premise, and is considered the most innovative in areas like real-time/streaming integration and extract-load-transform capabilities with big data technologies. The platform offers a comprehensive architecture covering key areas like data ingestion, preparation, streaming integration, parallel connectivity, and governance.

What We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerce

VMware Tanzu

The document discusses Dell's adoption of Spring Boot and Pivotal Cloud Foundry (PCF) for its e-commerce platform. It describes how Dell transformed its technology foundation to be cloud-native using microservices, Spring, and PCF. It also details Dell's evaluation of different technologies like Spring Cloud Dataflow and Apache Camel for integration and highlights tips for adopting Spring and leveraging PCF.

gRPC, GraphQL, REST - Which API Tech to use - API Conference Berlin oct 20

Phil Wilkins

The document discusses different API technologies including gRPC, GraphQL, and REST. It provides overviews of each technology, describing their origins, key concepts, pros, and cons. gRPC was developed by Google and uses protocol buffers for messages and HTTP/2 for transport. GraphQL was created by Facebook and uses a query language for clients to specify the exact data they need. REST is the more established standard based on HTTP and uses URIs for identification of resources.

Building a Stock Prediction system with Machine Learning using Geode, SpringX...

William Markito Oliveira

Automation + dev ops summit hail hydrate! from stream to lake

Timothy Spann

Modernizing an Existing SOA-based Architecture with APIs

Apigee | Google Cloud

How to build unified Batch & Streaming Pipelines with Apache Beam and Dataflow

Daniel Zivkovic

Apache Beam is a beautiful framework that blurs the line between Batch and Streaming, so check out this interactive tutorial by Patrick Lecuyer - Head of Specialist Customer Engineering at Google Canada. His examples run on GCP Dataflow, but what you'll learn will be portable across clouds, and distributed processing engines like Apache Flink, Apache Samza, Apache Spark, IBM Streams... regardless of where you do your Big Data processing! The meetup recording with TOC for easy navigation is at https://youtu.be/7pUYKX40RfA. P.S. For more interactive lectures like this, go to http://youtube.serverlesstoronto.org/ or sign up for our upcoming live events at https://www.meetup.com/Serverless-Toronto/events/

Similaire à Big data and messaging with Spring (20)

Institutionalizing Open Source - Puneet Sachdev - Nasscom Tech Series - June ...

Building Highly Scalable Spring Applications using In-Memory Data Grids

Asynchronous Event Streams – when java.util.stream met org.osgi.util.promise!...

IoT architecture

Get to know the browser better and write faster web apps

S2DS London 2015 - Hadoop Real World

Data Mesh Part 4 Monolith to Mesh

Open APIs - Risks and Rewards (Øredev 2013)

Node summit workshop

HTTP/2 and a Faster Web

A Pipeline for Distributed Topic and Sentiment Analysis of Tweets on Pivotal ...

Operational Machine Learning: Using Microsoft Technologies for Applied Data S...

Web Architecture - Mechanism and Threats

Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)

What We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerce

gRPC, GraphQL, REST - Which API Tech to use - API Conference Berlin oct 20

Building a Stock Prediction system with Machine Learning using Geode, SpringX...

Automation + dev ops summit hail hydrate! from stream to lake

Modernizing an Existing SOA-based Architecture with APIs

How to build unified Batch & Streaming Pipelines with Apache Beam and Dataflow

Dernier

HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU

panagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/ DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen! Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell. Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten. Diese Themen werden behandelt - Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten - Wie funktionieren CCB- und CCX-Lizenzen wirklich? - Verstehen des DLAU-Tools und wie man es am besten nutzt - Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw. - Praxisbeispiele und Best Practices zum sofortigen Umsetzen

HCL Notes and Domino License Cost Reduction in the World of DLAU

panagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/ The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this! We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model. Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward. These topics will be covered - Reducing license cost by finding and fixing misconfigurations and superfluous accounts - How do CCB and CCX licenses really work? - Understanding the DLAU tool and how to best utilize it - Tips for common problem areas, like team mailboxes, functional/test users, etc - Practical examples and best practices to implement right away

Project Management Semester Long Project - Acuity

jpupo2018

Acuity is an innovative learning app designed to transform the way you engage with knowledge. Powered by AI technology, Acuity takes complex topics and distills them into concise, interactive summaries that are easy to read & understand. Whether you're exploring the depths of quantum mechanics or seeking insight into historical events, Acuity provides the key information you need without the burden of lengthy texts.

20240609 QFM020 Irresponsible AI Reading List May 2024

Matthew Sinclair

How to Get CNIC Information System with Paksim Ga.pptx

danishmna97

National Security Agency - NSA mobile device best practices

Quotidiano Piemontese

How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf

Chart Kalyan

GenAI Pilot Implementation in the organizations

kumardaparthi1024

Mariano G Tinti - Decoding SpaceX

Mariano Tinti

WeTestAthens: Postman's AI & Automation Techniques

Postman

Skybuffer SAM4U tool for SAP license adoption

Tatiana Kojar

Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool. SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.

Serial Arm Control in Real Time Presentation

tolgahangng

Columbus Data & Analytics Wednesdays - June 2024

Jason Packer

Taking AI to the Next Level in Manufacturing.pdf

ssuserfac0301

Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as: 1. How quickly AI is being implemented in manufacturing. 2. Which barriers stand in the way of AI adoption. 3. How data quality and governance form the backbone of AI. 4. Organizational processes and structures that may inhibit effective AI adoption. 6. Ideas and approaches to help build your organization's AI strategy.

みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...

名前です男

Programming Foundation Models with DSPy - Meetup Slides

Zilliz

GraphRAG for Life Science to increase LLM accuracy

Tomaz Bratanic

Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...

saastr

UiPath Test Automation using UiPath Test Suite series, part 6

DianaGray10

Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI. UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities. Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes. What will you get from this session? 1. Insights into integrating generative AI. 2. Understanding how this integration enhances test automation within the UiPath platform 3. Practical demonstrations 4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath Topics covered: What is generative AI Test Automation with generative AI and Open AI. UiPath integration with generative AI Speaker: Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP

Ocean lotus Threat actors project by John Sitima 2024 (1).pptx

SitimaJohn

Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.

Dernier (20)

HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU

HCL Notes and Domino License Cost Reduction in the World of DLAU

Project Management Semester Long Project - Acuity

20240609 QFM020 Irresponsible AI Reading List May 2024

How to Get CNIC Information System with Paksim Ga.pptx

National Security Agency - NSA mobile device best practices

How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf

GenAI Pilot Implementation in the organizations

Mariano G Tinti - Decoding SpaceX

WeTestAthens: Postman's AI & Automation Techniques

Skybuffer SAM4U tool for SAP license adoption

Serial Arm Control in Real Time Presentation

Columbus Data & Analytics Wednesdays - June 2024

Taking AI to the Next Level in Manufacturing.pdf

Programming Foundation Models with DSPy - Meetup Slides

GraphRAG for Life Science to increase LLM accuracy

Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...

UiPath Test Automation using UiPath Test Suite series, part 6

Ocean lotus Threat actors project by John Sitima 2024 (1).pptx

Big data and messaging with Spring

2. Artem Bilan Spring Integration Team abilan@gopivotal.com Spring Framework Spring AMQP Spring XD Reactor https://spring.io/team/artembilan http://stackoverflow.com/users/2756547 http://www.linkedin.com/in/cleric https://github.com/artembilan © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

10. What is a Big Data application?  Not just writing a few MR or Hive/Pig jobs.  The full lifecycle involves • Ingestion • Stream Processing • Workflow Orchestration • Enterprise Integration • Export • Horizontal scalable deployment  How do you write one of these? • Status quo is to combine different projects – not ideal © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

11. Spring XD • Unified Platform • Developer Productivity • Modular Extensibility • Distributed Architecture • Portable Runtime • Hadoop Distribution Agnostic • Proven Foundation • XD = ‘eXtreme Data’ © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

12. Spring XD Files Sensors Mobile Social Spring XD Shell Spring XD Runtime Taps Jobs Compute Workflow Redis Ingest Streams Export Export RDBMS Gemfire HDFS NoSQL R, SAS Predictive modeling © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

14. Runtimes XD Admin http | filter | file CLUSTERED NODE CLUSTERED NODE CLUSTERED NODE SINGLE NODE HTTP Module Filter Module File Module All Modules Rabbit, Redis, (Pluggable) © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission. In Memory Transport http | filter | file

15. Streams MessageStore HTTP Tail File Mail Twitter Gemfire Syslog TCP JMS RabbitMQ MQTT © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission. Filter Transformer Splitter Aggregator Groovy Script Counters HTTP JSON Java Code File HDFS JDBC TCP Mail RabbitMQ Gemfire Splunk MQTT Dynamic Router

16. Taps  “Listen” to data from another stream • Other stream is unaffected by the tap and unaware of its presence • EAI ‘Wiretap’ © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

17. Analytics • Simple Counter • Field Value Counter – Count occurrences of named fields • Aggregate Counter – Pre-aggregate counts in time buckets • Gauge – last value • Rich Gauge – Last value, running average, min/max © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission. • Abstract API • Implementations for – Memory – Redis • Looking at – Pattern – JPMML

18. Jobs  Jobs are a directed graph of steps  Steps • Copy or Process data • Files, Databases, MR, Pig, Hive, Cascading  Step executions are persisted • Checkpointing with restart • Rich error handling capabilities  Single node or distributed with data partitioning  Jobs can be Triggered from streams  Executing jobs generate a stream of event data © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

20. Learn More. Stay Connected. • • • • • • • • Spring IO: https://spring.io/platform GitHub: https://github.com/spring-projects Spring Integration: http://projects.spring.io/spring-integration Spring XD: http://projects.spring.io/spring-xd Reactor: https://github.com/reactor EIP: http://www.eaipatterns.com Spring Batch: http://projects.spring.io/spring-batch Spring for Hadoop: http://projects.spring.io/spring-hadoop © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.

Notes de l'éditeur

Unified platform across big data domainsStream processing: ingestion & analyticsBatch processing workflow orchestration & exportProductivityHigh level DSL for managing streams and jobsProven foundation Built on existing assets: Spring Batch, Integration, DataExtensibleDI, Test friendly…DistributedA | B | C - Pluggable transports: Rabbit, Redis, …Portable Runtime Standalone – Simplicity and testabilityYARN –Fault Tolerance and ScalabilityIn-memory data grids – co-location of reference data.PAASHadoop Distribution Agnostic
Now have a unified platform that handle the whole thing.TODO – get rid of mobile?....XD runtime address both stream and batch processing in a unfired manner.XD runtime address both stream and batch processing in a unfired manner.
Distributed mode, process boundaries
Now have a unified platform that handle the whole thing.TODO – get rid of mobile?....XD runtime address both stream and batch processing in a unfired manner.XD runtime address both stream and batch processing in a unfired manner.
Now have a unified platform that handle the whole thing.TODO – get rid of mobile?....XD runtime address both stream and batch processing in a unfired manner.XD runtime address both stream and batch processing in a unfired manner.
Now have a unified platform that handle the whole thing.TODO – get rid of mobile?....XD runtime address both stream and batch processing in a unfired manner.XD runtime address both stream and batch processing in a unfired manner.
http | filehttp | hdfstwittersearch | hdfsaggregatecounter on hashtags