Breaking The Clustering Limits @ AlphaCSP JavaEdge 2007
•Télécharger en tant que PPT, PDF•
1 j'aime•727 vues
The document discusses the evolution of clustering in Java applications. It describes how early clustering solutions focused on replicating state between nodes for scalability and failover. More modern approaches use data grids, computational grids, and clustered virtual machines to distribute data and processing across nodes for improved performance and resource utilization. A variety of open source and commercial implementations are presented, including EHCache, GlassFish Shoal, Oracle Coherence, JBoss POJO Cache, GigaSpaces, and Terracotta.
Signaler
Partager
Signaler
Partager
Recommandé
From I/O To RAM
IO is slow, memory is fast. For many applications, the main performance and scalability bottleneck is disk and network access. This session will cover strategies that can help you utilize your RAM efficiently even in a distributed environment.
Discover how a clustering solution like Terracotta can help you reduce overall application latency.
TheEdge10 : Big Data is Here - Hadoop to the Rescue
This document discusses big data and Hadoop. It begins by explaining how data is growing exponentially and defining what big data is. It then introduces Hadoop as an open-source framework for storing and processing big data across clusters of commodity hardware. The rest of the document provides details on the key components of Hadoop, including HDFS for distributed storage, MapReduce for distributed processing, and various related projects like Pig, Hive and HBase that build on Hadoop.
Persisting Your Objects In The Database World @ AlphaCSP Professional OSS Con...
The document summarizes key concepts and solutions related to object-relational mapping (O-R mapping) and persisting Java objects in a relational database. It introduces Hibernate as a popular open-source framework that provides a natural programming model and handles O-R impedance mismatch. The document then covers Hibernate concepts, configuration, querying, relationships and annotations. It includes a demo of mapping Java classes to database tables using Hibernate.
REST and JAX-RS
The document discusses JAX-RS (JSR-311), which is a Java API for RESTful web services. It aims to make it easy to build RESTful web services and clients using plain Java objects and annotations. Key points covered include:
- JAX-RS uses annotations to map Java methods to HTTP methods and URI paths.
- It supports common features like URI templates, content negotiation, cookies and headers.
- Providers are used to bridge between HTTP requests/responses and Java objects.
- Annotations like @Produces specify the media types a resource can generate.
Clustering In The Wild
The document discusses clustering the issue tracking application Jira using Terracotta. It identifies common challenges in clustering Java applications, such as files, caches, threads and HTTP sessions. It describes how Terracotta was used to cluster Lucene indexes, home-grown caches and services in Jira. Distributed and shared services were implemented using Terracotta APIs. The session state was left unclustered for simplicity. External plugins were made cluster-aware through new APIs. In summary, clustering Jira required decisions around distributing data vs behavior and providing thread safety while leveraging Terracotta for transparency.
Improving PySpark performance: Spark Performance Beyond the JVM
This talk covers a number of important topics for making scalable Apache Spark programs - from RDD re-use to considerations for working with Key/Value data, why avoiding groupByKey is important and more. We also include Python specific considerations, like the difference between DataFrames/Datasets and traditional RDDs with Python. We also explore some tricks to intermix Python and JVM code for cases where the performance overhead is too high.
Ray and Its Growing Ecosystem
Ray (https://github.com/ray-project/ray) is a framework developed at UC Berkeley and maintained by Anyscale for building distributed AI applications. Over the last year, the broader machine learning ecosystem has been rapidly adopting Ray as the primary framework for distributed execution. In this talk, we will overview how libraries such as Horovod (https://horovod.ai/), XGBoost, and Hugging Face Transformers, have integrated with Ray. We will then showcase how Uber leverages Ray and these ecosystem integrations to simplify critical production workloads at Uber. This is a joint talk between Anyscale and Uber.
Smart Data Conference: DL4J and DataVec
This document discusses using DL4J and DataVec to build production-ready deep learning workflows for time series and text data. It provides an example of modeling sensor data with recurrent neural networks (RNNs) and character-level text generation with LSTMs. Key points include:
- DL4J is a deep learning framework for Java that runs on Spark and supports CPU/GPU. DataVec is a tool for data preprocessing.
- The document demonstrates loading and transforming sensor time series data with DataVec and training an RNN on the data with DL4J.
- It also shows vectorizing character-level text data from beer reviews with DataVec and using an LSTM in DL4J to generate new
Recommandé
From I/O To RAM
IO is slow, memory is fast. For many applications, the main performance and scalability bottleneck is disk and network access. This session will cover strategies that can help you utilize your RAM efficiently even in a distributed environment.
Discover how a clustering solution like Terracotta can help you reduce overall application latency.
TheEdge10 : Big Data is Here - Hadoop to the Rescue
This document discusses big data and Hadoop. It begins by explaining how data is growing exponentially and defining what big data is. It then introduces Hadoop as an open-source framework for storing and processing big data across clusters of commodity hardware. The rest of the document provides details on the key components of Hadoop, including HDFS for distributed storage, MapReduce for distributed processing, and various related projects like Pig, Hive and HBase that build on Hadoop.
Persisting Your Objects In The Database World @ AlphaCSP Professional OSS Con...
The document summarizes key concepts and solutions related to object-relational mapping (O-R mapping) and persisting Java objects in a relational database. It introduces Hibernate as a popular open-source framework that provides a natural programming model and handles O-R impedance mismatch. The document then covers Hibernate concepts, configuration, querying, relationships and annotations. It includes a demo of mapping Java classes to database tables using Hibernate.
REST and JAX-RS
The document discusses JAX-RS (JSR-311), which is a Java API for RESTful web services. It aims to make it easy to build RESTful web services and clients using plain Java objects and annotations. Key points covered include:
- JAX-RS uses annotations to map Java methods to HTTP methods and URI paths.
- It supports common features like URI templates, content negotiation, cookies and headers.
- Providers are used to bridge between HTTP requests/responses and Java objects.
- Annotations like @Produces specify the media types a resource can generate.
Clustering In The Wild
The document discusses clustering the issue tracking application Jira using Terracotta. It identifies common challenges in clustering Java applications, such as files, caches, threads and HTTP sessions. It describes how Terracotta was used to cluster Lucene indexes, home-grown caches and services in Jira. Distributed and shared services were implemented using Terracotta APIs. The session state was left unclustered for simplicity. External plugins were made cluster-aware through new APIs. In summary, clustering Jira required decisions around distributing data vs behavior and providing thread safety while leveraging Terracotta for transparency.
Improving PySpark performance: Spark Performance Beyond the JVM
This talk covers a number of important topics for making scalable Apache Spark programs - from RDD re-use to considerations for working with Key/Value data, why avoiding groupByKey is important and more. We also include Python specific considerations, like the difference between DataFrames/Datasets and traditional RDDs with Python. We also explore some tricks to intermix Python and JVM code for cases where the performance overhead is too high.
Ray and Its Growing Ecosystem
Ray (https://github.com/ray-project/ray) is a framework developed at UC Berkeley and maintained by Anyscale for building distributed AI applications. Over the last year, the broader machine learning ecosystem has been rapidly adopting Ray as the primary framework for distributed execution. In this talk, we will overview how libraries such as Horovod (https://horovod.ai/), XGBoost, and Hugging Face Transformers, have integrated with Ray. We will then showcase how Uber leverages Ray and these ecosystem integrations to simplify critical production workloads at Uber. This is a joint talk between Anyscale and Uber.
Smart Data Conference: DL4J and DataVec
This document discusses using DL4J and DataVec to build production-ready deep learning workflows for time series and text data. It provides an example of modeling sensor data with recurrent neural networks (RNNs) and character-level text generation with LSTMs. Key points include:
- DL4J is a deep learning framework for Java that runs on Spark and supports CPU/GPU. DataVec is a tool for data preprocessing.
- The document demonstrates loading and transforming sensor time series data with DataVec and training an RNN on the data with DL4J.
- It also shows vectorizing character-level text data from beer reviews with DataVec and using an LSTM in DL4J to generate new
A real-time architecture using Hadoop & Storm - Nathan Bijnens & Geert Van La...
The document discusses a real-time architecture using Hadoop and Storm. It proposes a layered architecture with a batch layer, speed layer, and serving layer. The batch layer uses Hadoop for batch processing and view generation. The speed layer uses Storm for stream processing and real-time views. The serving layer queries both the batch and real-time views to provide merged results. This architecture is known as the Lambda architecture and allows discarding and recomputing views from the immutable raw data as needed.
Integrating Deep Learning Libraries with Apache Spark
The combination of deep learning with Apache Spark has the potential to make a huge impact. Joseph Bradley and Xiangrui Meng share best practices for integrating popular deep learning libraries with Apache Spark. Rather than comparing deep learning systems or specific optimizations, Joseph and Xiangrui focus on issues that are common to many deep learning frameworks when running on a Spark cluster, such as optimizing cluster setup and data ingest (clusters can be configured to avoid task conflicts on GPUs and to allow using multiple GPUs per worker), configuring the cluster (setting up pipelines for efficient data ingest improves job throughput), and monitoring long-running jobs (interactive monitoring facilitates both the work of configuration and checking the stability of deep learning jobs). Joseph and Xiangrui then demonstrate the techniques using Google’s popular TensorFlow library.
Apache Spark: What's under the hood
This document provides an overview of Apache Spark's architectural components through the life of simple Spark jobs. It begins with a simple Spark application analyzing airline on-time arrival data, then covers Resilient Distributed Datasets (RDDs), the cluster architecture, job execution through Spark components like tasks and scheduling, and techniques for writing better Spark applications like optimizing partitioning and reducing shuffle size.
Deep Dive: Memory Management in Apache Spark
Memory management is at the heart of any data-intensive system. Spark, in particular, must arbitrate memory allocation between two main use cases: buffering intermediate data for processing (execution) and caching user data (storage). This talk will take a deep dive through the memory management designs adopted in Spark since its inception and discuss their performance and usability implications for the end user.
Introduction to Spark ML Pipelines Workshop
Introduction to Spark ML Pipelines Workshop slides - companion IJupyter notebooks in Python & Scala are available from my github at https://github.com/holdenk/spark-intro-ml-pipeline-workshop
Writing Continuous Applications with Structured Streaming Python APIs in Apac...
Description:
We are amidst the Big Data Zeitgeist era in which data comes at us fast, in myriad forms and formats at intermittent intervals or in a continuous stream, and we need to respond to streaming data immediately. This need has created a notion of writing a streaming application that’s continuous, reacts and interacts with data in real-time. We call this continuous application, which we will discuss.
Abstract:
We are amidst the Big Data Zeitgeist era in which data comes at us fast, in myriad forms and formats at intermittent intervals or in a continuous stream, and we need to respond to streaming data immediately. This need has created a notion of writing a streaming application that’s continuous, reacts and interacts with data in real-time. We call this continuous application.
In this talk we will explore the concepts and motivations behind the continuous application, how Structured Streaming Python APIs in Apache Spark 2.x enables writing continuous applications, examine the programming model behind Structured Streaming, and look at the APIs that support them.
Through a short demo and code examples, I will demonstrate how to write an end-to-end Structured Streaming application that reacts and interacts with both real-time and historical data to perform advanced analytics using Spark SQL, DataFrames and Datasets APIs.
You’ll walk away with an understanding of what’s a continuous application, appreciate the easy-to-use Structured Streaming APIs, and why Structured Streaming in Apache Spark 2.x is a step forward in developing new kinds of streaming applications.
How Machine Learning and AI Can Support the Fight Against COVID-19
In this session, we show how to leverage CORD dataset, containing more than 400000 scientific papers on COVID and related topics, and recent advances in natural language processing and other AI techniques to generate new insights in support of the ongoing fight against this infectious disease.
The idea explored in our talk is to apply modern NLP methods, such and named entity recognition (NER) and relation extraction to article’s abstracts (and, possibly, full text), to extract some meaningful insights from the text, and to enable semantically rich search over the paper corpus. We first investigate how to train NER model using Medical NER dataset from Kaggle, and specialized version of BERT (PubMedBERT) as a feature extractor, to allow automatic extraction of such entities as medical condition names, medicine names and pathogens. Entity extraction alone can provide us with some interesting findings, such as how approaches to COVID treatment evolved with time, in terms of mentioned medicines. We demonstrate how to use Azure Machine Learning for training the model.
To take this investigation one step further, we also investigate the usage of pre-trained medical models, available as Text Analytics for Health service on the Microsoft Azure cloud. In addition to many entity types, it can also extract relations (such as the dosage of medicine provisioned), entity negation, and entity mapping to some well-known medical ontologies. We investigate the best way to use Azure ML at scale to score large paper collection, and to store the results.
Inferno Scalable Deep Learning on Spark
This document summarizes a presentation on Inferno, a system for scalable deep learning on Apache Spark. Inferno allows deep learning models built with Blaze, La Trobe University's deep learning system, to be trained faster using a Spark cluster. It coordinates distributed training of Blaze models across worker nodes, with optimized communication of weights and hyperparameters. Evaluation shows Inferno can train ResNet models on ImageNet up to 4-5 times faster than a single GPU. The presentation provides an overview of deep learning and Spark, demonstrates how Blaze allows easy model building, and explains Inferno's architecture for distributed deep learning training on Spark.
Packed Objects: Fast Talking Java Meets Native Code - Steve Poole (IBM)
Presented at JAX London 2013
Worried about the future of Java? Want to see it keep moving forward? Don't be concerned. The transformation of Java is already underway. Driven by new technologies and new opportunities Java and the JVM are entering uncharted worlds and challenging old approaches. In this session learn about one such expedition in the form of an introductory talk to technology being developed by IBM. This experimental technology is exploring a new way to share data between the JVM and other runtimes.
A really really fast introduction to PySpark - lightning fast cluster computi...
Apache Spark is a fast and general engine for distributed computing & big data processing with APIs in Scala, Java, Python, and R. This tutorial will briefly introduce PySpark (the Python API for Spark) with some hands-on-exercises combined with a quick introduction to Spark's core concepts. We will cover the obligatory wordcount example which comes in with every big-data tutorial, as well as discuss Spark's unique methods for handling node failure and other relevant internals. Then we will briefly look at how to access some of Spark's libraries (like Spark SQL & Spark ML) from Python. While Spark is available in a variety of languages this workshop will be focused on using Spark and Python together.
Beyond Shuffling - Effective Tips and Tricks for Scaling Spark (Vancouver Sp...
This document provides a summary of a presentation on scaling Apache Spark. It discusses techniques for reusing RDDs through caching, persistence levels and checkpointing. It also covers best practices for working with key-value data to avoid problems from groupByKey, and using Spark SQL and accumulators. Finally, it previews bringing code generation to Spark ML to improve performance.
Elasticsearch And Apache Lucene For Apache Spark And MLlib
This document summarizes a presentation about using Elasticsearch and Lucene for text processing and machine learning pipelines in Apache Spark. Some key points:
- Elasticsearch provides text analysis capabilities through Lucene and can be used to clean, tokenize, and vectorize text for machine learning tasks.
- Elasticsearch integrates natively with Spark through Java/Scala APIs and allows indexing and querying data from Spark.
- A typical machine learning pipeline for text classification in Spark involves tokenization, feature extraction (e.g. hashing), and a classifier like logistic regression.
- The presentation proposes preparing text analysis specifications in Elasticsearch once and reusing them across multiple Spark pipelines to simplify the workflows and avoid data movement between systems
Apache Spark Fundamentals Meetup Talk
The document provides an overview of Apache Spark fundamentals including what Spark is, its ecosystem and terminology, how to create RDDs and use different operations like transformations and actions, RDD lineage and evolution from RDDs to DataFrames and DataSets. It also discusses concepts like job lifecycle, persistency, and running Spark on a YARN cluster. Code samples are shown to demonstrate different Spark features. The presenter has a computer engineering background and currently works on data analytics and transformations using Spark.
Spark and spark streaming internals
Spark and Spark Streaming internals allow for low latency, fault tolerance, and diverse workloads. Spark uses a Resilient Distributed Dataset (RDD) model where data is partitioned across a cluster. A directed acyclic graph (DAG) is used to schedule tasks across stages in an optimized way. Spark Streaming runs streaming computations as small deterministic batch jobs by chopping live streams into batches and processing them using RDD transformations and actions.
Beyond parallelize and collect - Spark Summit East 2016
As Spark jobs are used for more mission critical tasks, beyond exploration, it is important to have effective tools for testing. This talk expands on “Effective Testing For Spark Programs” (not required to have been seen) to discuss how to create large scale test jobs without depending on collect & parallelize which limit the sizes of datasets we can work with. Testing Spark Streaming jobs can be especially challenging, as the normal techniques for loading test data don’t work and additional work must be done to collect the results and stop streaming. We will explore the difficulties with testing Streaming Programs, options for setting up integration testing, beyond just local mode, with Spark, and also examine best practices for acceptance tests.
Problem Solving Recipes Learned from Supporting Spark: Spark Summit East talk...
Due to Spark, writing big data applications has never been easier…at least until they stop being easy! At Lightbend we’ve helped our customers out of a number of hidden Spark pitfalls. Some crop up often; the ever-persistent OutOfMemoryError, the confusing NoSuchMethodError, shuffle and partition management, etc. Others occur less frequently; an obscure configuration affecting SQL broadcasts, struggles with speculating, a failing stream recovery due to RDD joins, S3 file reading leading to hangs, etc. All are intriguing! In this session we will provide insights into their origins and show how you can avoid making the same mistakes. Whether you are a seasoned Spark developer or a novice, you should learn some new tips and tricks that could save you hours or even days of debugging.
Deep Learning: DL4J and DataVec
This document discusses using DL4J and DataVec to build deep learning workflows for modeling time series sensor data with recurrent neural networks. It provides an example of loading and transforming sensor data with DataVec, configuring an RNN with DL4J, and training the model both locally and distributed on Spark. The overall workflow involves extracting, transforming, and loading data with DataVec, vectorizing it, modeling with DL4J, evaluating performance, and deploying trained models for execution on Spark/Hadoop platforms.
Internals
we will see internal architecture of spark cluster i.e what is driver, worker,executer and cluster manager, how spark program will be run on cluster and what are jobs,stages and task.
Spark as the Gateway Drug to Typed Functional Programming: Spark Summit East ...
This document discusses how Spark can serve as a gateway to typed functional programming by introducing developers to Scala and functional programming concepts. It covers how Spark leverages concepts like lazy evaluation, immutable data structures, higher-order functions, and types to provide a powerful and scalable framework. While complex, these functional techniques help tackle challenges of scale and complexity that arise in real-world AI systems.
Using PostgreSQL with Bibliographic Data
An investigation of how PostgreSQL and its latest capabilities (JSONB data type, GIN indices, Full Text Search) can be used to store, index and perform queries on structured Bibliographic Data such as MARC21/MARCXML, breaking the dependence on proprietary and arcane or obsolete software products.
Talk presented at FOSDEM 2016 in Brussels on 31/01/2016. This is a very practical & hands-on presentation with example code which is certainly not optimal ;)
Clustering van IT-componenten
Clustering can provide high availability and scalability. Shared nothing architectures are best for achieving both high availability and scalability together. Oracle Real Application Cluster (RAC) offers advantages over alternative Oracle clustering configurations, but its scalability is limited. The cost-effectiveness of using RAC in a redundant array of inexpensive servers configuration is small due to its limited scalability. Alternatives may be more suitable depending on specific needs and requirements.
Optimizing your java applications for multi core hardware
Session Presented at 5th IndicThreads.com Conference On Java held on 10-11 December 2010 in Pune, India
WEB: http://J10.IndicThreads.com
------------
Rising power dissipation in microprocessor chips is leading to a trend towards increasing the number of cores on a chip (multi-core processors) rather than increasing clock frequency as the primary basis for increasing system performance. Consequently the number of threads in commodity hardware has also exploded. This leads to complexity in designing and configuring high performance Java applications that make effective use of new hardware. In this talk we provide a summary of the changes happening in the multi-core world and subsequently discuss about some of the JVM features which exploit the multi-core capabilities of the underlying hardware. We also explain techniques to analyze and optimize your application for highly concurrent systems. Key topics include an overview of Java Virtual Machine features & configuration, ways to correctly leverage java.util.concurrent package to achieve enhanced parallelism for applications in a multi-core environment, operating system issues, virtualization, Java code optimizations and useful profiling tools and techniques.
Takeaways for the Audience
Attendees will leave with a better understanding of the new multi-core world, understanding of Java Virtual Machine features which exploit mulit-core and the techniques they can apply to ensure their Java applications run well in mulit-core environment.
Contenu connexe
Tendances
A real-time architecture using Hadoop & Storm - Nathan Bijnens & Geert Van La...
The document discusses a real-time architecture using Hadoop and Storm. It proposes a layered architecture with a batch layer, speed layer, and serving layer. The batch layer uses Hadoop for batch processing and view generation. The speed layer uses Storm for stream processing and real-time views. The serving layer queries both the batch and real-time views to provide merged results. This architecture is known as the Lambda architecture and allows discarding and recomputing views from the immutable raw data as needed.
Integrating Deep Learning Libraries with Apache Spark
The combination of deep learning with Apache Spark has the potential to make a huge impact. Joseph Bradley and Xiangrui Meng share best practices for integrating popular deep learning libraries with Apache Spark. Rather than comparing deep learning systems or specific optimizations, Joseph and Xiangrui focus on issues that are common to many deep learning frameworks when running on a Spark cluster, such as optimizing cluster setup and data ingest (clusters can be configured to avoid task conflicts on GPUs and to allow using multiple GPUs per worker), configuring the cluster (setting up pipelines for efficient data ingest improves job throughput), and monitoring long-running jobs (interactive monitoring facilitates both the work of configuration and checking the stability of deep learning jobs). Joseph and Xiangrui then demonstrate the techniques using Google’s popular TensorFlow library.
Apache Spark: What's under the hood
This document provides an overview of Apache Spark's architectural components through the life of simple Spark jobs. It begins with a simple Spark application analyzing airline on-time arrival data, then covers Resilient Distributed Datasets (RDDs), the cluster architecture, job execution through Spark components like tasks and scheduling, and techniques for writing better Spark applications like optimizing partitioning and reducing shuffle size.
Deep Dive: Memory Management in Apache Spark
Memory management is at the heart of any data-intensive system. Spark, in particular, must arbitrate memory allocation between two main use cases: buffering intermediate data for processing (execution) and caching user data (storage). This talk will take a deep dive through the memory management designs adopted in Spark since its inception and discuss their performance and usability implications for the end user.
Introduction to Spark ML Pipelines Workshop
Introduction to Spark ML Pipelines Workshop slides - companion IJupyter notebooks in Python & Scala are available from my github at https://github.com/holdenk/spark-intro-ml-pipeline-workshop
Writing Continuous Applications with Structured Streaming Python APIs in Apac...
Description:
We are amidst the Big Data Zeitgeist era in which data comes at us fast, in myriad forms and formats at intermittent intervals or in a continuous stream, and we need to respond to streaming data immediately. This need has created a notion of writing a streaming application that’s continuous, reacts and interacts with data in real-time. We call this continuous application, which we will discuss.
Abstract:
We are amidst the Big Data Zeitgeist era in which data comes at us fast, in myriad forms and formats at intermittent intervals or in a continuous stream, and we need to respond to streaming data immediately. This need has created a notion of writing a streaming application that’s continuous, reacts and interacts with data in real-time. We call this continuous application.
In this talk we will explore the concepts and motivations behind the continuous application, how Structured Streaming Python APIs in Apache Spark 2.x enables writing continuous applications, examine the programming model behind Structured Streaming, and look at the APIs that support them.
Through a short demo and code examples, I will demonstrate how to write an end-to-end Structured Streaming application that reacts and interacts with both real-time and historical data to perform advanced analytics using Spark SQL, DataFrames and Datasets APIs.
You’ll walk away with an understanding of what’s a continuous application, appreciate the easy-to-use Structured Streaming APIs, and why Structured Streaming in Apache Spark 2.x is a step forward in developing new kinds of streaming applications.
How Machine Learning and AI Can Support the Fight Against COVID-19
In this session, we show how to leverage CORD dataset, containing more than 400000 scientific papers on COVID and related topics, and recent advances in natural language processing and other AI techniques to generate new insights in support of the ongoing fight against this infectious disease.
The idea explored in our talk is to apply modern NLP methods, such and named entity recognition (NER) and relation extraction to article’s abstracts (and, possibly, full text), to extract some meaningful insights from the text, and to enable semantically rich search over the paper corpus. We first investigate how to train NER model using Medical NER dataset from Kaggle, and specialized version of BERT (PubMedBERT) as a feature extractor, to allow automatic extraction of such entities as medical condition names, medicine names and pathogens. Entity extraction alone can provide us with some interesting findings, such as how approaches to COVID treatment evolved with time, in terms of mentioned medicines. We demonstrate how to use Azure Machine Learning for training the model.
To take this investigation one step further, we also investigate the usage of pre-trained medical models, available as Text Analytics for Health service on the Microsoft Azure cloud. In addition to many entity types, it can also extract relations (such as the dosage of medicine provisioned), entity negation, and entity mapping to some well-known medical ontologies. We investigate the best way to use Azure ML at scale to score large paper collection, and to store the results.
Inferno Scalable Deep Learning on Spark
This document summarizes a presentation on Inferno, a system for scalable deep learning on Apache Spark. Inferno allows deep learning models built with Blaze, La Trobe University's deep learning system, to be trained faster using a Spark cluster. It coordinates distributed training of Blaze models across worker nodes, with optimized communication of weights and hyperparameters. Evaluation shows Inferno can train ResNet models on ImageNet up to 4-5 times faster than a single GPU. The presentation provides an overview of deep learning and Spark, demonstrates how Blaze allows easy model building, and explains Inferno's architecture for distributed deep learning training on Spark.
Packed Objects: Fast Talking Java Meets Native Code - Steve Poole (IBM)
Presented at JAX London 2013
Worried about the future of Java? Want to see it keep moving forward? Don't be concerned. The transformation of Java is already underway. Driven by new technologies and new opportunities Java and the JVM are entering uncharted worlds and challenging old approaches. In this session learn about one such expedition in the form of an introductory talk to technology being developed by IBM. This experimental technology is exploring a new way to share data between the JVM and other runtimes.
A really really fast introduction to PySpark - lightning fast cluster computi...
Apache Spark is a fast and general engine for distributed computing & big data processing with APIs in Scala, Java, Python, and R. This tutorial will briefly introduce PySpark (the Python API for Spark) with some hands-on-exercises combined with a quick introduction to Spark's core concepts. We will cover the obligatory wordcount example which comes in with every big-data tutorial, as well as discuss Spark's unique methods for handling node failure and other relevant internals. Then we will briefly look at how to access some of Spark's libraries (like Spark SQL & Spark ML) from Python. While Spark is available in a variety of languages this workshop will be focused on using Spark and Python together.
Beyond Shuffling - Effective Tips and Tricks for Scaling Spark (Vancouver Sp...
This document provides a summary of a presentation on scaling Apache Spark. It discusses techniques for reusing RDDs through caching, persistence levels and checkpointing. It also covers best practices for working with key-value data to avoid problems from groupByKey, and using Spark SQL and accumulators. Finally, it previews bringing code generation to Spark ML to improve performance.
Elasticsearch And Apache Lucene For Apache Spark And MLlib
This document summarizes a presentation about using Elasticsearch and Lucene for text processing and machine learning pipelines in Apache Spark. Some key points:
- Elasticsearch provides text analysis capabilities through Lucene and can be used to clean, tokenize, and vectorize text for machine learning tasks.
- Elasticsearch integrates natively with Spark through Java/Scala APIs and allows indexing and querying data from Spark.
- A typical machine learning pipeline for text classification in Spark involves tokenization, feature extraction (e.g. hashing), and a classifier like logistic regression.
- The presentation proposes preparing text analysis specifications in Elasticsearch once and reusing them across multiple Spark pipelines to simplify the workflows and avoid data movement between systems
Apache Spark Fundamentals Meetup Talk
The document provides an overview of Apache Spark fundamentals including what Spark is, its ecosystem and terminology, how to create RDDs and use different operations like transformations and actions, RDD lineage and evolution from RDDs to DataFrames and DataSets. It also discusses concepts like job lifecycle, persistency, and running Spark on a YARN cluster. Code samples are shown to demonstrate different Spark features. The presenter has a computer engineering background and currently works on data analytics and transformations using Spark.
Spark and spark streaming internals
Spark and Spark Streaming internals allow for low latency, fault tolerance, and diverse workloads. Spark uses a Resilient Distributed Dataset (RDD) model where data is partitioned across a cluster. A directed acyclic graph (DAG) is used to schedule tasks across stages in an optimized way. Spark Streaming runs streaming computations as small deterministic batch jobs by chopping live streams into batches and processing them using RDD transformations and actions.
Beyond parallelize and collect - Spark Summit East 2016
As Spark jobs are used for more mission critical tasks, beyond exploration, it is important to have effective tools for testing. This talk expands on “Effective Testing For Spark Programs” (not required to have been seen) to discuss how to create large scale test jobs without depending on collect & parallelize which limit the sizes of datasets we can work with. Testing Spark Streaming jobs can be especially challenging, as the normal techniques for loading test data don’t work and additional work must be done to collect the results and stop streaming. We will explore the difficulties with testing Streaming Programs, options for setting up integration testing, beyond just local mode, with Spark, and also examine best practices for acceptance tests.
Problem Solving Recipes Learned from Supporting Spark: Spark Summit East talk...
Due to Spark, writing big data applications has never been easier…at least until they stop being easy! At Lightbend we’ve helped our customers out of a number of hidden Spark pitfalls. Some crop up often; the ever-persistent OutOfMemoryError, the confusing NoSuchMethodError, shuffle and partition management, etc. Others occur less frequently; an obscure configuration affecting SQL broadcasts, struggles with speculating, a failing stream recovery due to RDD joins, S3 file reading leading to hangs, etc. All are intriguing! In this session we will provide insights into their origins and show how you can avoid making the same mistakes. Whether you are a seasoned Spark developer or a novice, you should learn some new tips and tricks that could save you hours or even days of debugging.
Deep Learning: DL4J and DataVec
This document discusses using DL4J and DataVec to build deep learning workflows for modeling time series sensor data with recurrent neural networks. It provides an example of loading and transforming sensor data with DataVec, configuring an RNN with DL4J, and training the model both locally and distributed on Spark. The overall workflow involves extracting, transforming, and loading data with DataVec, vectorizing it, modeling with DL4J, evaluating performance, and deploying trained models for execution on Spark/Hadoop platforms.
Internals
we will see internal architecture of spark cluster i.e what is driver, worker,executer and cluster manager, how spark program will be run on cluster and what are jobs,stages and task.
Spark as the Gateway Drug to Typed Functional Programming: Spark Summit East ...
This document discusses how Spark can serve as a gateway to typed functional programming by introducing developers to Scala and functional programming concepts. It covers how Spark leverages concepts like lazy evaluation, immutable data structures, higher-order functions, and types to provide a powerful and scalable framework. While complex, these functional techniques help tackle challenges of scale and complexity that arise in real-world AI systems.
Using PostgreSQL with Bibliographic Data
An investigation of how PostgreSQL and its latest capabilities (JSONB data type, GIN indices, Full Text Search) can be used to store, index and perform queries on structured Bibliographic Data such as MARC21/MARCXML, breaking the dependence on proprietary and arcane or obsolete software products.
Talk presented at FOSDEM 2016 in Brussels on 31/01/2016. This is a very practical & hands-on presentation with example code which is certainly not optimal ;)
Tendances (20)
A real-time architecture using Hadoop & Storm - Nathan Bijnens & Geert Van La...
A real-time architecture using Hadoop & Storm - Nathan Bijnens & Geert Van La...
Integrating Deep Learning Libraries with Apache Spark
Integrating Deep Learning Libraries with Apache Spark
Writing Continuous Applications with Structured Streaming Python APIs in Apac...
Writing Continuous Applications with Structured Streaming Python APIs in Apac...
How Machine Learning and AI Can Support the Fight Against COVID-19
How Machine Learning and AI Can Support the Fight Against COVID-19
Packed Objects: Fast Talking Java Meets Native Code - Steve Poole (IBM)
Packed Objects: Fast Talking Java Meets Native Code - Steve Poole (IBM)
A really really fast introduction to PySpark - lightning fast cluster computi...
A really really fast introduction to PySpark - lightning fast cluster computi...
Beyond Shuffling - Effective Tips and Tricks for Scaling Spark (Vancouver Sp...
Beyond Shuffling - Effective Tips and Tricks for Scaling Spark (Vancouver Sp...
Elasticsearch And Apache Lucene For Apache Spark And MLlib
Elasticsearch And Apache Lucene For Apache Spark And MLlib
Beyond parallelize and collect - Spark Summit East 2016
Beyond parallelize and collect - Spark Summit East 2016
Problem Solving Recipes Learned from Supporting Spark: Spark Summit East talk...
Problem Solving Recipes Learned from Supporting Spark: Spark Summit East talk...
Spark as the Gateway Drug to Typed Functional Programming: Spark Summit East ...
Spark as the Gateway Drug to Typed Functional Programming: Spark Summit East ...
Similaire à Breaking The Clustering Limits @ AlphaCSP JavaEdge 2007
Clustering van IT-componenten
Clustering can provide high availability and scalability. Shared nothing architectures are best for achieving both high availability and scalability together. Oracle Real Application Cluster (RAC) offers advantages over alternative Oracle clustering configurations, but its scalability is limited. The cost-effectiveness of using RAC in a redundant array of inexpensive servers configuration is small due to its limited scalability. Alternatives may be more suitable depending on specific needs and requirements.
Optimizing your java applications for multi core hardware
Session Presented at 5th IndicThreads.com Conference On Java held on 10-11 December 2010 in Pune, India
WEB: http://J10.IndicThreads.com
------------
Rising power dissipation in microprocessor chips is leading to a trend towards increasing the number of cores on a chip (multi-core processors) rather than increasing clock frequency as the primary basis for increasing system performance. Consequently the number of threads in commodity hardware has also exploded. This leads to complexity in designing and configuring high performance Java applications that make effective use of new hardware. In this talk we provide a summary of the changes happening in the multi-core world and subsequently discuss about some of the JVM features which exploit the multi-core capabilities of the underlying hardware. We also explain techniques to analyze and optimize your application for highly concurrent systems. Key topics include an overview of Java Virtual Machine features & configuration, ways to correctly leverage java.util.concurrent package to achieve enhanced parallelism for applications in a multi-core environment, operating system issues, virtualization, Java code optimizations and useful profiling tools and techniques.
Takeaways for the Audience
Attendees will leave with a better understanding of the new multi-core world, understanding of Java Virtual Machine features which exploit mulit-core and the techniques they can apply to ensure their Java applications run well in mulit-core environment.
Best Practices for performance evaluation and diagnosis of Java Applications ...
Session Presented at 5th IndicThreads.com Conference On Java held on 10-11 December 2010 in Pune, India
WEB: http://J10.IndicThreads.com
------------
Enterprise applications typically comprise of multi layered stacks including the application modules, application servers, the Java Virtual Machine and the underlying Operating System. Consequently the performance of these applications are a factor of these different layers. In the eventuality of a performance problem, it is often difficult to determine the starting point for diagnosis. The Java Virtual Machine is the ‘engine’ for most of the applications. It is responsible broadly for efficient execution and memory management of applications. End users have difficulty attributing the effect of the JVM on the performance of the application. This is because usually JVM is viewed as a ‘black box’.
This talk provides an insight into the key subsystems of the JVM by looking under the hood of a high performance JVM. It ventures onto talk about approaches and techniques for analyzing performance issues. It concludes by introducing the audience to a tool called the “Health Center” which is useful for evaluating and comprehending the JVM behavior of a running application in an unobtrusive, lightweight manner.
Takeaways for the Audience A better understanding of key JVM components, approaches and techniques to diagnose performance issues and performance evaluation using the Health Center
J2EE Batch Processing
The document summarizes findings from a project testing batch processing performance using J2EE. It discusses considerations for batch frameworks, infrastructure, caching, logging, design challenges, and whether to use batch processing. It also outlines the design of the batch process used, including leveraging raw JDBC, Oracle caching, and tools for performance monitoring.
Java one 2010
- The document summarizes key announcements and projects from JavaOne 2010, including Project Coin, Project Lambda, and Project Jigsaw which focus on language enhancements for productivity, closures, and modularity.
- It also discusses case studies from various companies on architectures using technologies like Spring, Hibernate, caching, and NoSQL databases to handle large-scale applications.
- Trends highlighted include focus on asynchronous and event-driven architectures, partitioning, and monitoring to handle thousands of servers and billions of requests per day.
Introduction to Real Time Java
The document provides an overview of real-time Java and its key concepts:
- Real-time means determinism where deadlines must be met, as opposed to just fast throughput.
- The Real-Time Specification for Java (RTSJ) addresses limitations of Java SE for real-time applications like unpredictable delays from class loading and garbage collection.
- RTSJ implementations like IBM WebSphere Real Time and Sun RTS add features like high resolution timers, priority scheduling, and asynchronous event handling.
Java Core | Modern Java Concurrency | Martijn Verburg & Ben Evans
The document provides an overview of modern Java concurrency. It discusses how concurrency has become important for performance as CPUs evolved to include multiple cores. It summarizes the java.util.concurrent utilities and common concurrency constructs like locks, queues, thread pools. It advocates using higher-level concurrency abstractions and more immutable and thread-safe collections to make concurrent programming easier.
Lightweight Grids With Terracotta
The document discusses how Terracotta can be used to create lightweight grids for enterprise applications to improve availability and scalability. It provides an overview of challenges like single points of failure and bottlenecks. Terracotta allows clustering of Java applications across multiple JVMs in a transparent way and reduces the complexity of distributed computing. A simple example demonstrates how Terracotta can be used to distribute a queue across multiple nodes.
Clustered Architecture Patterns Delivering Scalability And Availability
The document discusses different architecture patterns for delivering scalability and availability in clustered systems. It covers load-balanced and partitioned scale-out patterns, and how to balance simplicity, scalability, and availability. JVM-level clustering is presented as an approach that can address these patterns by sharing memory across JVMs in a transparent way.
Spark to Production @Windward
This document provides an overview of Spark and how Windward uses it to analyze maritime data. It introduces Spark concepts like RDDs, persistence, and transformations. It discusses best practices for running Spark in development, testing, and production environments including configuration, data storage, automation, testing, logging, and monitoring. The goal is to explain how Windward leverages Spark's distributed processing capabilities to organize and analyze large volumes of maritime data.
How Java 19 Influences the Future of Your High-Scale Applications .pdf
Java 19 introduces several new features and improvements to help applications scale better. These include virtual threads which allow efficient asynchronous programming without blocking threads, pattern matching for more readable code, and improvements to JNI for safer native code integration. Java 19 also deprecates some legacy methods and makes timezone data updates to improve sustainability. Overall, the release focuses on enhancing scalability and maintainability for high performance applications.
Terracotta DSO
Terracotta is Java infrastructure software that allows applications to scale across multiple computers without custom coding. It provides transparent clustering at the JVM level through a shared memory space called Network Attached Memory (NAM). Applications using Terracotta are unaware that it is installed and function the same with or without it. This allows state to be shared across instances.
Shopzilla On Concurrency
Slides from my presentation about Shopzilla's concurrency strategies to the Pasadena Java User's Group on April 26, 2010. This is essentially the same material as covered by my colleague Rodney Barlow in an earlier presentation http://www.slideshare.net/rodneypbarlow/shopzilla-on-concurrency, with a few minor tweaks.
JCConf 2016 - Cloud Computing Applications - Hazelcast, Spark and Ignite
This session aims to establish applications running against distributed and scalable system, or as we know cloud computing system. We will introduce you not only briefing of Hazelcast but also deeper kernel of it, and how it works with Spark, the most famous Map-reduce library. Furthermore, we will introduce another in-memory cache called Apache Ignite and compare it with Hazelcast to see what's the difference between them. In the end, we will give a demonstration showing how Hazelcast and Spark work together well to form a cloud-base service which is distributed, flexible, reliable, available, scalable and stable. You can find demo code here: https://github.com/CyberJos/jcconf2016-hazelcast-spark
https://cyberjos.blog/java/seminar/jcconf-2016-cloud-computing-applications-hazelcast-spark-and-ignite/
Shopzilla On Concurrency
Abstract
Concurrency is everywhere. Prior to Java 5, concurrency was difficult
and error prone. Since Java 5, it's far more prevalent in our
application code, and through time it's been lurking in open-source
frameworks and containers. Concurrency is also a fundamental part of
Shopzilla's web-site and services ecosystem.
Introduction
Rod Barlow from Shopzilla will explore a brief history of concurrency, and the key
concurrency features and techniques provided by the Java API since
Java 5. Topics covered include Immutability, Atomic References, Blocking
Queues, Locks and Deadlocks. Also covered is Concurrency in
Frameworks, and Shopzilla's Website Concurrency Framework, including
Thread Pools, Executors and Futures.
Low latency in java 8 v5
How can we use Lambdas in Java 8 in a low latency context? What are some of the performance consideration in using Java 8.
The Adventure: BlackRay as a Storage Engine
- BlackRay is an in-memory relational database and search engine that supports SQL and fulltext search. It was originally developed in 2005 for a phone directory with over 80 million records.
- The presentation discusses BlackRay's architecture, data loading process, indexing performance, and support for transactions, clustering, and APIs. It also describes efforts to implement BlackRay as a MySQL storage engine.
- Going forward, the team aims to improve SQL support, add security features, and explore using BlackRay as a backend for other applications like LDAP directories.
node.js 실무 - node js in practice by Jesang Yoon
Sharing 4 years of experience about node.js - A google chrome V8 engine javascript based web server technology. This slide covers about wide range of knowledge about node.js learned from 4 years of production, experiment, test & failures
4년 동안 node.js 서버 프로그래밍을 경험한 내용을 간략하게 정리해 보았습니다. node.js 를 접하시는 분들에게 도움이 되었으면 합니다.
11g R2
- Oracle Database 11g Release 2 provides many advanced features to lower IT costs including in-memory processing, automated storage management, database compression, and real application testing capabilities.
- It allows for online application upgrades using edition-based redefinition which allows new code and data changes to be installed without disrupting the existing system.
- Oracle provides multiple upgrade paths from prior database versions to 11g to allow for predictable performance and a safe upgrade process.
Spring Boot 3 And Beyond
Dan Vega discussed new features and capabilities in Spring Boot 3 and beyond, including support for JDK 17, Jakarta EE 9, ahead-of-time compilation, observability with Micrometer, Docker Compose integration, and initial support for Project Loom's virtual threads in Spring Boot 3.2 to improve scalability. He provided an overview of each new feature and explained how they can help Spring applications.
Similaire à Breaking The Clustering Limits @ AlphaCSP JavaEdge 2007 (20)
Optimizing your java applications for multi core hardware
Optimizing your java applications for multi core hardware
Best Practices for performance evaluation and diagnosis of Java Applications ...
Best Practices for performance evaluation and diagnosis of Java Applications ...
Java Core | Modern Java Concurrency | Martijn Verburg & Ben Evans
Java Core | Modern Java Concurrency | Martijn Verburg & Ben Evans
Clustered Architecture Patterns Delivering Scalability And Availability
Clustered Architecture Patterns Delivering Scalability And Availability
How Java 19 Influences the Future of Your High-Scale Applications .pdf
How Java 19 Influences the Future of Your High-Scale Applications .pdf
JCConf 2016 - Cloud Computing Applications - Hazelcast, Spark and Ignite
JCConf 2016 - Cloud Computing Applications - Hazelcast, Spark and Ignite
Plus de Baruch Sadogursky
DevOps Patterns & Antipatterns for Continuous Software Updates @ NADOG April ...
The document discusses patterns and antipatterns for continuous software updates. It describes challenges with continuous updates including things that can go wrong with updates and solutions like local rollbacks, over-the-air updates, continuous rather than batch updates, automated deployments, frequent updates, state awareness, progressive delivery, observability, rollbacks, and feature flags. The goal is to transition to extremely tiny and frequent updates, called Liquid Software, to provide an illusion of software flowing continuously from development to the target.
DevOps Patterns & Antipatterns for Continuous Software Updates @ DevOps.com A...
So, you want to update the software for your user, be it the nodes in your K8s cluster, a browser on user’s desktop, an app in user’s smartphone or even a user’s car. What can possibly go wrong?
In this talk, we’ll analyze real-world software update fails and how multiple DevOps patterns, that fit a variety of scenarios, could have saved the developers. Manually making sure that everything works before sending an update and expecting the user to do acceptance tests before they update is most definitely not on the list of such patterns.
Join us for some awesome and scary continuous update horror stories and some obvious (and some not so obvious) proven ideas for improvement and best practices you can start following tomorrow.
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at Oracle Code NY...
In this talk, we’ll take you to a scaling journey, from 3 developers to a 100. We’ll talk about the challenges each milestone in this growth brings, both technological and methodological, and how to solve those challenges using the right mix of people, the right selection of tools and the correctly crafted process. The speakers excel in the different aspects of this triangle and went through this journey (more than once) themselves. And the fun and entertaining presentation as a Greek tragedy can’t hurt, can it?
Data driven devops as presented at QCon London 2018
Devops is usually viewed from a traditional perspective of a collaboration of Dev, Ops, and QA, driven by the change in Culture, People, and Process. But how do you know where you stand and where to move? As in almost any field, data and metrics give you the gauges and instruments. In this talk, we’ll talk about the key measurements for the DevOps transformation process and provide you with 3 metrics you can start measuring tomorrow.
A Research Study Into DevOps Bottlenecks as presented at Oracle Code LA 2018
We asked the Fortune 500 software delivery leaders what holds them back. This talk is the analysis of their insights on what bottlenecks they encountered in their DevOps journey.
Java Puzzlers NG S03 a DevNexus 2018
You know what to expect by now: funny and puzzling questions about Java 8 and Java 9, JFrog t-shirts are airborne, the usual combo of learning and fun ahead!
Where the Helm are your binaries? as presented at Canada Kubernetes Meetups
Do you always know what’s going on with your product artifacts since the moment they are built by the CI server from Git sources all the way to being deployed by Helm into Kuberenetes?
In this talk, we will show how to build a reliable and transparent pipeline from code to cluster using Git, Artifactory, Docker, Kubernetes, and Helm. We’ll show how you such a pipeline can help you answer the big questions: What to deploy, What is deployed, and what is this artifact that I am looking for. This kind of transparency is critical for today’s environments, and Kubernetes with Helm shouldn’t be an exception.
Data driven devops as presented at Codemash 2018
By Baruch Sadogursky
Devops is usually viewed from a traditional perspective of a collaboration of Dev, Ops and QA, driven by the change in Culture, People and Process. But how do you know where you stand and were to move? As in almost any field, data and metrics give you the gauges and instruments. In this talk we’ll talk about the key measurements for the DevOps transformation process and provide you with 3 metrics you can start measuring tomorrow.
A Research Study into DevOps Bottlenecks as presented at Codemash 2018
By Baruch Sadogursky
We asked the Fortune 500 software delivery leaders what holds them back. This talk is the analysis of their insights on what bottlenecks they encountered in their DevOps journey.
Best Practices for Managing Docker Versions as presented at JavaOne 2017
By Baruch Sadogursky
There are three hard things in computer science: cache invalidation, naming things, and off-by-one errors. This session tackles naming, especially Docker version naming. Labels, tags, checksums...how should you use them to keep track of Docker versions? What about dev versus prod images—how best to distinguish those? What about the “latest” tag? What about cleanup? Could we do more? Versioning often seems like a simple problem, but when you have a tool that gives you as much power and flexibility as Docker does, it often helps to develop guidelines. The presentation examines the tools available for managing Docker images and some simple patterns you can employ in various use cases for CI/CD to keep track of your containers.
Troubleshooting & Debugging Production Microservices in Kubernetes as present...
Debugging applications in production is like being the detective in a crime movie. Especially with microservices. Especially with containers. Especially in the cloud. Trying to see what’s going on in a production deployment at scale is impossible without proper tools! Google has spent over a decade deploying containerized Java applications at unprecedented scale and the infrastructure and tools developed by Google have made it uniquely possible to manage, troubleshoot, and debug, at scale.
Join this session to see how you can diagnose and troubleshoot production issues w/ out of the box Kubernetes tools, as well as getting insight from the ecosystem with Weave Scope, JFrog Artifactory & Stackdriver tools.
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at Devoxx 2017
As in a good Greek Tragedy, scaling devops to big teams has 3 stages and usually end badly. In this play (it’s more than a talk!) we’ll present you with Pentagon Inc, and their way to scaling devops from a team of 3 engineers to a team of 100 (spoiler – it’s painful!)
Amazon Alexa Skills vs Google Home Actions, the Big Java VUI Faceoff as prese...
In this session we will compare and contrast the experience of implementing voice user interface for the two market leader voice activated assistants. Both are extendable, both have Java APIs, but which is better? Two speakers, two laptops, two IDEs writing Java code to implement the same Alexa Skill and Google Home Action and you pick the winner!
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at DevOps Days Be...
As in a good Greek Tragedy, scaling devops to big teams has 3 stages and usually end badly. In this play (it’s more than a talk!) we’ll present you with Pentagon Inc, and their way to scaling devops from a team of 3 engineers to a team of 100 (spoiler – it’s painful!)
Java Puzzlers NG S02: Down the Rabbit Hole as it was presented at The Pittsbu...
Moar puzzlers! The more we work with Java 8, the more we go into the rabbit hole. Did they add all those streams, lambdas, monads, Optionals and CompletableFutures only to confuse us? It surely looks so! And Java 9 that heads our way brings even more of what we like the most, more puzzlers, of course! In this season we as usual have a great batch of the best Java WTF, great jokes to present them and great prizes for the winners!
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at The Pittsburgh...
As in a good Greek Tragedy, scaling devops to big teams has 3 stages and usually end badly. In this play (it’s more than a talk!) we’ll present you with Pentagon Inc, and their way to scaling devops from a team of 3 engineers to a team of 100 (spoiler – it’s painful!)
Let’s Wing It: A Study in DevRel Strategy
Developer relations strategy is often an afterthought. This session’s speaker asks whether that’s OK and gets the opinion of DevRel leaders from companies large and small.
Log Driven First Class Customer Support at Scale
In this talk, Baruch Sadogursky presents the challenges of a high demand SaaS product incident triage at scale, as well as discuss the sources of log items, including the platform, tenants and other types of log sources. He will show practical examples of collector and filters configuration and will take you through a number of real world examples of problems investigations using Artifactory and Sumo Logic.
[Webinar] The Frog And The Butler: CI Pipelines For Modern DevOps
No relationship in DevOps is more important than that between your CI/CD server and your Binary Repository. Jenkins has long been the go-to server for CI/CD, and JFrog Artifactory has long been one of the most popular integrations with it. This webinar focuses on the new features of the integration, leveraging the Jenkins Pipeline DSL for infrastructure-as-code of your favorite artifactory features whether it be generic, maven, gradle or Docker, and will show an end-to-end example of pipelines across multiple technologies and how powerful these new capabilities are.
Patterns and antipatterns in Docker image lifecycle as was presented at DC Do...
While Docker has enabled an unprecedented velocity of software production, it is all too easy to spin out of control. A promotion-based model is required to control and track the flow of Docker images as much as it is required for a traditional software development lifecycle. New tools often introduce new paradigms. We will examine the patterns and the antipatterns for Docker image management, and what impact the new tools have on the battle-proven paradigms of the software development lifecycle.
Plus de Baruch Sadogursky (20)
DevOps Patterns & Antipatterns for Continuous Software Updates @ NADOG April ...
DevOps Patterns & Antipatterns for Continuous Software Updates @ NADOG April ...
DevOps Patterns & Antipatterns for Continuous Software Updates @ DevOps.com A...
DevOps Patterns & Antipatterns for Continuous Software Updates @ DevOps.com A...
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at Oracle Code NY...
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at Oracle Code NY...
Data driven devops as presented at QCon London 2018
Data driven devops as presented at QCon London 2018
A Research Study Into DevOps Bottlenecks as presented at Oracle Code LA 2018
A Research Study Into DevOps Bottlenecks as presented at Oracle Code LA 2018
Where the Helm are your binaries? as presented at Canada Kubernetes Meetups
Where the Helm are your binaries? as presented at Canada Kubernetes Meetups
A Research Study into DevOps Bottlenecks as presented at Codemash 2018
A Research Study into DevOps Bottlenecks as presented at Codemash 2018
Best Practices for Managing Docker Versions as presented at JavaOne 2017
Best Practices for Managing Docker Versions as presented at JavaOne 2017
Troubleshooting & Debugging Production Microservices in Kubernetes as present...
Troubleshooting & Debugging Production Microservices in Kubernetes as present...
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at Devoxx 2017
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at Devoxx 2017
Amazon Alexa Skills vs Google Home Actions, the Big Java VUI Faceoff as prese...
Amazon Alexa Skills vs Google Home Actions, the Big Java VUI Faceoff as prese...
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at DevOps Days Be...
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at DevOps Days Be...
Java Puzzlers NG S02: Down the Rabbit Hole as it was presented at The Pittsbu...
Java Puzzlers NG S02: Down the Rabbit Hole as it was presented at The Pittsbu...
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at The Pittsburgh...
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at The Pittsburgh...
[Webinar] The Frog And The Butler: CI Pipelines For Modern DevOps
[Webinar] The Frog And The Butler: CI Pipelines For Modern DevOps
Patterns and antipatterns in Docker image lifecycle as was presented at DC Do...
Patterns and antipatterns in Docker image lifecycle as was presented at DC Do...
Dernier
Finale of the Year: Apply for Next One!
Presentation for the event called "Finale of the Year: Apply for Next One!" organized by GDSC PJATK
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Digital Marketing Trends in 2024 | Guide for Staying Ahead
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI, built on the robust SAP Business Technology Platform (SAP BTP), is the latest and most advanced version of our AI development, reaffirming our commitment to delivering top-tier AI solutions. Skybuffer AI harnesses all the innovative capabilities of the SAP BTP in the AI domain, from Conversational AI to cutting-edge Generative AI and Retrieval-Augmented Generation (RAG). It also helps SAP customers safeguard their investments into SAP Conversational AI and ensure a seamless, one-click transition to SAP Business AI.
With Skybuffer AI, various AI models can be integrated into a single communication channel such as Microsoft Teams. This integration empowers business users with insights drawn from SAP backend systems, enterprise documents, and the expansive knowledge of Generative AI. And the best part of it is that it is all managed through our intuitive no-code Action Server interface, requiring no extensive coding knowledge and making the advanced AI accessible to more users.
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
Fueling AI with Great Data with Airbyte Webinar
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
When it comes to unit testing in the .NET ecosystem, developers have a wide range of options available. Among the most popular choices are NUnit, XUnit, and MSTest. These unit testing frameworks provide essential tools and features to help ensure the quality and reliability of code. However, understanding the differences between these frameworks is crucial for selecting the most suitable one for your projects.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
A Comprehensive Guide to DeFi Development Services in 2024
DeFi represents a paradigm shift in the financial industry. Instead of relying on traditional, centralized institutions like banks, DeFi leverages blockchain technology to create a decentralized network of financial services. This means that financial transactions can occur directly between parties, without intermediaries, using smart contracts on platforms like Ethereum.
In 2024, we are witnessing an explosion of new DeFi projects and protocols, each pushing the boundaries of what’s possible in finance.
In summary, DeFi in 2024 is not just a trend; it’s a revolution that democratizes finance, enhances security and transparency, and fosters continuous innovation. As we proceed through this presentation, we'll explore the various components and services of DeFi in detail, shedding light on how they are transforming the financial landscape.
At Intelisync, we specialize in providing comprehensive DeFi development services tailored to meet the unique needs of our clients. From smart contract development to dApp creation and security audits, we ensure that your DeFi project is built with innovation, security, and scalability in mind. Trust Intelisync to guide you through the intricate landscape of decentralized finance and unlock the full potential of blockchain technology.
Ready to take your DeFi project to the next level? Partner with Intelisync for expert DeFi development services today!
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Azure API Management to expose backend services securely
How to use Azure API Management to expose backend service securely
Dernier (20)
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Azure API Management to expose backend services securely
Azure API Management to expose backend services securely
Breaking The Clustering Limits @ AlphaCSP JavaEdge 2007
- 2. Breaking the Clustering Limits Baruch Sadogursky Consultant, AlphaCSP
- 18. Grid Types
- 19. Data Grids
- 24. Master/Worker
- 25. Map/Reduce
- 30. Implementations
- 31. EHCache
- 33. EHCache Example
- 34. GlassFish Shoal
- 36. JXTA Usage in Shoal
- 42. JBoss POJO Cache
- 45. JBoss POJO Cache Usage
- 47. GigaSpaces
- 50. GigaSpaces XAP
- 51. OpenTerracotta
- 58. Terracotta Demo