The Missing Piece of On-Demand Clusters

•

0 j'aime•1,617 vues

The Missing Piece of On-Demand Clusters Presented by Calvin Jia, Alluxio Introduction to Alluxio Meetup at Princeton http://www.meetup.com/futureofdata-princeton/events/232927731/

Logiciels

The Missing Piece of On-Demand Clusters
October 2016
Calvin Jia

About Me
• Calvin Jia
• Software Engineer @ Alluxio, Inc.
• Alluxio PMC
• #1 Alluxio Contributor
• Twitter: @JiaCalvin
2

Outline
• Cloud Architectures
• Alluxio in Cloud Architectures
• Takeaways
3

Cloud Architectures – Overview
• Mostly service based, from providers
– Amazon Web Services
– Google Cloud Platform
• Separate compute and storage clusters
• Compute clusters are ephemeral
4

Cloud Architectures – Pros & Cons
Pros
• Low maintenance
• Pay as you go
• Elastic and scalable
• Cost effective storage
Cons
• Lower Performance
5

Outline
• Cloud Architectures
• Alluxio in Cloud Architectures
• Takeaways
6

Alluxio in Cloud Architectures – Overview
• Deployed in compute clusters
– Memory speed data access
– Transparent data access to any storage
• Simple to deploy
– Mount storage systems similar to local disks
7

Alluxio in Cloud Architectures – Example
8

Alluxio in Cloud Architectures – Benefits
9
• Remedies the performance drawback
• Acceleration due to memory-speed I/O
• Designed to improve the affinity of compute
and storage

Alluxio in Cloud Architectures – Data Path
10
FAST
104 - 105 MB/s
MODERATE 103 - 104 MB/s
SLOW 102 - 103 MB/s
Only when necessary
Limited
Often
SSD
HDD
Mem

Takeaways – Experiment Results
11
0
100
200
300
400
500
600
Initial
Read Subsequent
Read Read
from

Separate
Job
Runtime
(seconds)
Spark
-‐ No
Persist
Spark
-‐ Persist
Alluxio

Takeaways – Alluxio & Cloud Architectures
12
• Cloud architectures have significant upsides
• Alluxio alleviates the major downsides

Try out Alluxio 1.2.0
http://www.alluxio.org/releases
13

Read the blog athttp://www.alluxio.com/2016/08/accelerating-on-demand-data-analytics-with-alluxio
Read the whitepaper at www.alluxio.com.
Check out www.alluxio.org.
Thank you!
14

Recommandé

Alluxio: The missing piece of on-demand clusters at Alluxio Meetup 2016

Alluxio, Inc.

Spark Summit EU talk by Jiri Simsa

Spark Summit

Alluxio Presentation at AMPLab Summer Retreat 2016

Alluxio, Inc.

Enable Fast Big Data Analytics on Ceph with Alluxio at Ceph Days 2017

Alluxio, Inc.

Alluxio (Formerly Tachyon): Unify Data At Memory Speed at Global Big Data Con...

Alluxio, Inc.

Unify Data at Memory Speed by Haoyuan Li - VAULT Conference 2017

Alluxio, Inc.

Accessing Data Anywhere with Unified Namespace

Alluxio, Inc.

Alluxio: Unify Data at Memory Speed at Strata and Hadoop World San Jose 2017

Alluxio, Inc.

Recommandé

Alluxio: The missing piece of on-demand clusters at Alluxio Meetup 2016

Alluxio, Inc.

Spark Summit EU talk by Jiri Simsa

Spark Summit

Alluxio Presentation at AMPLab Summer Retreat 2016

Alluxio, Inc.

Enable Fast Big Data Analytics on Ceph with Alluxio at Ceph Days 2017

Alluxio, Inc.

Alluxio (Formerly Tachyon): Unify Data At Memory Speed at Global Big Data Con...

Alluxio, Inc.

Unify Data at Memory Speed by Haoyuan Li - VAULT Conference 2017

Alluxio, Inc.

Accessing Data Anywhere with Unified Namespace

Alluxio, Inc.

Alluxio: Unify Data at Memory Speed at Strata and Hadoop World San Jose 2017

Alluxio, Inc.

Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017

Alluxio, Inc.

Alluxio (formerly Tachyon): Open Source Memory Speed Virtual Distributed Storage

Alluxio, Inc.

Alluxio (formerly Tachyon): The Journey thus far and the Road Ahead

Alluxio, Inc.

Alluxio-FUSE as a data access layer for Dask

Alluxio, Inc.

ALLUXIO (formerly Tachyon): Unify Data at Memory Speed - Effective using Spar...

Alluxio, Inc.

Rise of Intermediate APIs - Beam and Alluxio at Alluxio Meetup 2016

Alluxio, Inc.

Introduction to Alluxio (formerly Tachyon) and how it brings up to 300x perfo...

Alluxio, Inc.

Alluxio Use Cases at Strata+Hadoop World Beijing 2016

Alluxio, Inc.

Alluxio: Unify Data at Memory Speed; 2016-11-18

Alluxio, Inc.

Open Source Memory Speed Virtual Distributed Storage

Alluxio, Inc.

Best Practices for Using Alluxio with Spark

Alluxio, Inc.

Alluxio Mesos Meetup - SMACK to SMAACK

Alluxio, Inc.

Best Practices for Using Alluxio with Spark

Alluxio, Inc.

Alluxio Keynote at Strata+Hadoop World Beijing 2016

Alluxio, Inc.

Securely Enhancing Data Access in Hybrid Cloud with Alluxio

Alluxio, Inc.

Deploying Alluxio in the Cloud for Machine Learning

Alluxio, Inc.

Alluxio Presentation at Strata San Jose 2016

Jiří Šimša

Building a high-performance data lake analytics engine at Alibaba Cloud with ...

Alluxio, Inc.

Getting Started with Alluxio + Spark + S3

Alluxio, Inc.

Presto on Alluxio Hands-On Lab

Alluxio, Inc.

Tachyon workshop 2015-07-19

Tachyon Nexus, Inc.

Accelerating Machine Learning Pipelines with Alluxio at Alluxio Meetup 2016

Alluxio, Inc.

Contenu connexe

Tendances

Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017

Alluxio, Inc.

Alluxio (formerly Tachyon): Open Source Memory Speed Virtual Distributed Storage

Alluxio, Inc.

Alluxio (formerly Tachyon): The Journey thus far and the Road Ahead

Alluxio, Inc.

Alluxio-FUSE as a data access layer for Dask

Alluxio, Inc.

ALLUXIO (formerly Tachyon): Unify Data at Memory Speed - Effective using Spar...

Alluxio, Inc.

Rise of Intermediate APIs - Beam and Alluxio at Alluxio Meetup 2016

Alluxio, Inc.

Introduction to Alluxio (formerly Tachyon) and how it brings up to 300x perfo...

Alluxio, Inc.

Alluxio Use Cases at Strata+Hadoop World Beijing 2016

Alluxio, Inc.

Alluxio: Unify Data at Memory Speed; 2016-11-18

Alluxio, Inc.

Open Source Memory Speed Virtual Distributed Storage

Alluxio, Inc.

Best Practices for Using Alluxio with Spark

Alluxio, Inc.

Alluxio Mesos Meetup - SMACK to SMAACK

Alluxio, Inc.

Best Practices for Using Alluxio with Spark

Alluxio, Inc.

Alluxio Keynote at Strata+Hadoop World Beijing 2016

Alluxio, Inc.

Securely Enhancing Data Access in Hybrid Cloud with Alluxio

Alluxio, Inc.

Deploying Alluxio in the Cloud for Machine Learning

Alluxio, Inc.

Alluxio Presentation at Strata San Jose 2016

Jiří Šimša

Building a high-performance data lake analytics engine at Alibaba Cloud with ...

Alluxio, Inc.

Getting Started with Alluxio + Spark + S3

Alluxio, Inc.

Presto on Alluxio Hands-On Lab

Alluxio, Inc.

Tendances (20)

Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017

Alluxio (formerly Tachyon): Open Source Memory Speed Virtual Distributed Storage

Alluxio (formerly Tachyon): The Journey thus far and the Road Ahead

Alluxio-FUSE as a data access layer for Dask

ALLUXIO (formerly Tachyon): Unify Data at Memory Speed - Effective using Spar...

Rise of Intermediate APIs - Beam and Alluxio at Alluxio Meetup 2016

Introduction to Alluxio (formerly Tachyon) and how it brings up to 300x perfo...

Alluxio Use Cases at Strata+Hadoop World Beijing 2016

Alluxio: Unify Data at Memory Speed; 2016-11-18

Open Source Memory Speed Virtual Distributed Storage

Best Practices for Using Alluxio with Spark

Alluxio Mesos Meetup - SMACK to SMAACK

Best Practices for Using Alluxio with Spark

Alluxio Keynote at Strata+Hadoop World Beijing 2016

Securely Enhancing Data Access in Hybrid Cloud with Alluxio

Deploying Alluxio in the Cloud for Machine Learning

Alluxio Presentation at Strata San Jose 2016

Building a high-performance data lake analytics engine at Alibaba Cloud with ...

Getting Started with Alluxio + Spark + S3

Presto on Alluxio Hands-On Lab

En vedette

Tachyon workshop 2015-07-19

Tachyon Nexus, Inc.

Accelerating Machine Learning Pipelines with Alluxio at Alluxio Meetup 2016

Alluxio, Inc.

Tachyon: An Open Source Memory-Centric Distributed Storage System

Tachyon Nexus, Inc.

This talk was first presented at the 2015 Strata+Hadoop World NYC (http://strataconf.com/big-data-conference-ny-2015/public/schedule/detail/43484) Deep learning algorithms have been widely used in many real-world applications, including computer vision, machine translation, and fraud detection. Unfortunately, deep learning only works best when the model is big and trained on large-scale datasets. Meanwhile, distributed computing platforms like Spark are designed to handle big data, and have been used extensively. By having deep learning available on Spark, businesses can fully take advantage of deep learning capabilities on their datasets using their existing Spark infrastructure. In this talk, we present a scalable implementation of predictive deep learning algorithms on Spark, including feedforward neural networks, convolutional neural networks (CNNs), and recurrent neural networks (RNNs). This, to our best knowledge, is the first successful implementation of CNNs and RNNs on Spark. To support big model training, we use Tachyon as common storage layers between the Spark workers. With its in-memory distributed execution model, Tachyon provides a scalable approach even when the model is too big to be handled on a single machine. Our solution also exploits graphical processing units (GPUs) for matrix computation whenever they are available on worker nodes, further improving execution time. The attendees will learn about deep learning models, the architecture of the system, and how to train and run deep learning models on Spark with Tachyon.

First-ever scalable, distributed deep learning architecture using Spark & Tac...

Arimo, Inc.

Presentation by TachyonNexus & Intel at Strata Singapore 2015

Tachyon Nexus, Inc.

Presentation by TachyonNexus & Baidu at Strata Singapore 2015

Tachyon Nexus, Inc.

Tachyon Presentation at AMPCamp 6 (November, 2015)

Tachyon Nexus, Inc.

Spark Summit EU talk by Jiri Simsa

Alluxio, Inc.

FCSarch 09 Gothic

jdankoff

La spectroscopie Raman appliquée au contrôle de qualité analytique des médicaments injectables : évaluation analytique et économique comparative versus CLHP et UV/visible-IRTF. La centralisation hospitalière, sous couvert pharmaceutique, du façonnage des solutions injectables des médicaments anticancéreux est devenue obligation légale en France. Nous avons établi de longue date que les Objets Thérapeutiques (OTs) ainsi fabriqués devraient tous bénéficier d’un Contrôle de Qualité Analytique (CQA) idéalement Libératoire (CQAL) pour les paramètres clés d’identité, de pureté et de concentration nominale en tel ou tel principe actif. Au cours de travaux récents, nous avons établi dans plusieurs situations, la supériorité technique de la Spectroscopie Raman (SR) non intrusive sur toute autre solution analytique et notamment sur les méthodes séparatives chromatographiques CLHP ainsi que sur l’analyse vibrationnelle par UV/visible-IRTF. Une évaluation à la fois qualitative, économique et environnementale croisée vient enrichir ces travaux. Les 3 modèles analytiques ont a été comparés en situation opérationnelle au moyen : a) d’une grille de critères qualitatifs, b) de tableaux d’amortissement des équipements, c) des coûts en biens consommables, d) du poids éventuel des équipements de support et des locaux, e) de l’Unité d’Œuvre (UO) dont le coût direct composite (€) varie en fonction de l’option technique retenue, de la charge catégorielle en ressources humaines (RHs) opérateurs (diverses combinaisons sont possibles) et, de l’éventuelle fraction de biens consommables. L’UO peut ainsi prendre 12 valeurs possibles comprises entre les bornes extrêmes de 1 à 5,5 €. Une grille d’évaluation qualitative et de performance positionne la technologie SR non intrusive comme constamment supérieure ou équivalente aux 2 autres techniques éprouvées. Les données rapportées confirment le caractère prometteur de la SR pour le CQA non intrusif et y compris dans des situations inaccessibles aux autres techniques. Nos résultats confirment la place centrale que pourra dans l’avenir occuper cette solution d’exploration contextuelle d’objets de géométrie quelconque e.g. diffuseurs portables. La SR apparaît comme un fort contributeur à la sécurisation du circuit (ou de la filière) du médicament injectable ainsi qu’à la protection des acteurs de soins et de leur environnement de travail.

SFPC 2014 - Communication orale - SR & CQAL - A. Amin et col.

pharmex

Combined Spinal Epidural Anesthesia

Bilal Baig

En vedette (11)

Tachyon workshop 2015-07-19

Accelerating Machine Learning Pipelines with Alluxio at Alluxio Meetup 2016

Tachyon: An Open Source Memory-Centric Distributed Storage System

First-ever scalable, distributed deep learning architecture using Spark & Tac...

Presentation by TachyonNexus & Intel at Strata Singapore 2015

Presentation by TachyonNexus & Baidu at Strata Singapore 2015

Tachyon Presentation at AMPCamp 6 (November, 2015)

Spark Summit EU talk by Jiri Simsa

FCSarch 09 Gothic

SFPC 2014 - Communication orale - SR & CQAL - A. Amin et col.

Combined Spinal Epidural Anesthesia

Similaire à The Missing Piece of On-Demand Clusters

Boomi Molecule Migration to the Cloud: Top 5 Strategies Revealed

Kellton Tech Solutions Ltd

Distributed Query Service Powered By Presto & Alluxio Across Clouds @Walmart...

Ashish Tadose

Cloud Computing Basics

DataNext Solutions

Enterprise Distributed Query Service powered by Presto & Alluxio across cloud...

Alluxio, Inc.

When small problems become big problems

Adrian Cole

Today everyone is talking about clouds, and a few are building them, but far fewer are operating successful clouds. In this session we'll examine a variety of paradigm shifts IT makes when moving from a traditional virtualization and management mindset to operating a successful cloud. For most organizations, without careful planning the hype of a cloud solution can quickly overcome its capabilities and pre-existing best practices can combine to create the worst possible cloud scenario -- a cloud which isn't economical to operate, and which is more cumbersome to manage than a traditional virtualization farm. Key topics covered include: - Successful transition of operational and management paradigm - How the VM density of clouds change Ops - What it means to monitor the network in a cloud environment, at hyper-dense virtualization levels - Preventing storage costs from outpacing delivery costs

Taming the cost of your first cloud - CCCEU 2014

Tim Mackey

Migrate to platform of your choice

Ashnikbiz

Cloud Computing

Yong Heui Cho

Enterprise Distributed Query Service powered by Presto & Alluxio across cloud...

Shubham Tagra

Building a Hybrid Cloud Solution

Cloudian

Apps Associates - Insmed INNOVATE16- EBS on the AWSCloudREV

Christopher Colucci

Transforming Enterprises through Next-generation Cloud Applications

Tata Consultancy Services

Architecting on The Cloud

Amazon Web Services

What's new CloudBerry Backup 5.7

Evgeny Rudinsky

Learn why Snowflake analytic data warehouse makes sense for BI including data loading flexibility and scalability, consumption-based storage and compute costs, Time Travel and data sharing features, support across a range of BI tools like Power BI and Tableau and ability to allocate compute costs. View this on-demand webinar: https://senturus.com/resources/10-reasons-snowflake-is-great-for-analytics/. Senturus offers a full spectrum of services in business intelligence and training on Cognos, Tableau and Power BI. Our resource library has hundreds of free live and recorded webinars, blog posts, demos and unbiased product reviews available on our website at: http://www.senturus.com/senturus-resources/.

10 Reasons Snowflake Is Great for Analytics

Senturus

Venture capitalist Matt Ocko’s 20-year track record of success in the startup world has given him unique insight into how AWS has changed the venture financing process. In this session, you’ll learn about industries susceptible to disruption by AWS-based startups, and where VCs are willing to take new risks on those startups, including the heavily-regulated medical, government, financial, and industrial sectors. Matt will talk about how new, supercomputing startups are now possible because of AWS technologies. Hear about how using AWS technologies can actually reduce risk – and reduce time to customer penetration – from a VC perspective, and how to go from ‘AWS to Series A’ in 5 easy pieces.

From AWS to Series A in 5 Easy Pieces

Amazon Web Services

Alluxio Tech Talk Mar 12, 2019 Speaker: Bin Fan, Alluxio Matt Fuller, Starburst As data analytic needs have increased with the explosion of data, the importance of the speed of analytics and the interactivity of queries has increased dramatically In this tech talk, we will introduce the Starburst Presto, Alluxio, and cloud object store stack for building a highly-concurrent and low-latency analytics platform. This stack provides a strong solution to run fast SQL across multiple storage systems including HDFS, S3, and others in public cloud, hybrid cloud, and multi-cloud environments. You’ll learn about: - The architecture of Presto, an open source distributed SQL engine, as well as innovations by Starburst like as it’s cost-based optimizer - How Presto can query data from cloud object storage like S3 at high performance and cost-effectively with Alluxio - How to achieve data locality and cross-job caching with Alluxio no matter where the data is persisted and reduce egress costs In addition, we’ll present some real world architectures & use cases from internet companies like JD.com and NetEase.com running the Presto and Alluxio stack at the scale of hundreds of nodes.

Interactive Analytics with the Starburst Presto + Alluxio stack for the Cloud

Alluxio, Inc.

Microsoft applications like SharePoint and Exchange are excellent candidates to move to the cloud, because they readily benefit from the scalability and low storage costs that Amazon Web Services offers. Review this webinar, led by the AWS Premier Consulting Partner Smartronix, to learn best practices in designing, building, and managing Microsoft workloads on AWS. Webinar topics include: • Cloud deployment options best suited to Microsoft SharePoint and Exchange environments • Governance, security, and compliance considerations when migrating Microsoft workloads • Proven methods for enabling enterprise-class Microsoft solutions on AWS.

AWS Partner Webcast - Rapid and Reliable Implementation of Microsoft Workload...

Amazon Web Services

Unified Big Data Analytics: Any Stack, Any Cloud

Alluxio, Inc.

Automating Infrastructure as a Service Deployments and monitoring – TEC213

Chris Kernaghan

Similaire à The Missing Piece of On-Demand Clusters (20)

Boomi Molecule Migration to the Cloud: Top 5 Strategies Revealed

Distributed Query Service Powered By Presto & Alluxio Across Clouds @Walmart...

Cloud Computing Basics

Enterprise Distributed Query Service powered by Presto & Alluxio across cloud...

When small problems become big problems

Taming the cost of your first cloud - CCCEU 2014

Migrate to platform of your choice

Cloud Computing

Enterprise Distributed Query Service powered by Presto & Alluxio across cloud...

Building a Hybrid Cloud Solution

Apps Associates - Insmed INNOVATE16- EBS on the AWSCloudREV

Transforming Enterprises through Next-generation Cloud Applications

Architecting on The Cloud

What's new CloudBerry Backup 5.7

10 Reasons Snowflake Is Great for Analytics

From AWS to Series A in 5 Easy Pieces

Interactive Analytics with the Starburst Presto + Alluxio stack for the Cloud

AWS Partner Webcast - Rapid and Reliable Implementation of Microsoft Workload...

Unified Big Data Analytics: Any Stack, Any Cloud

Automating Infrastructure as a Service Deployments and monitoring – TEC213

Plus de Alluxio, Inc.

As GenAI and AI continue to transform businesses, scaling these workloads requires optimized underlying infrastructure. A multi-cloud architecture allows organizations to leverage different cloud services to meet diverse workload demands while maximizing efficiency, reducing costs, and avoiding vendor lock-in. However, achieving a multi-cloud vision can be challenging. In this webinar, Tarik will share how an agonistic data layer, like Alluxio, allows you to embrace the separation of storage from compute and simplify the adoption of multi-cloud for AI. - Learn why leveraging multiple cloud providers is critical for balancing performance, scalability, and cost of your AI platform - Discover how an agnostic data layer like Alluxio provides seamless data access in multi-cloud that bridges storage and compute without data replication - Gain insights into real-world examples and best practices for deploying AI across on-prem, hybrid, and multi-cloud environments

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Alluxio, Inc.

Optimizing Data Access for Analytics And AI with Alluxio

Alluxio, Inc.

Speed Up Presto at Uber with Alluxio Caching

Alluxio, Inc.

Alluxio x Tobiko - ETL Happy Hour April 16, 2024 For more Alluxio events: https://alluxio.io/events/ Speaker: Toby Mao (CTO @ Tobiko Data) Writing efficient and correct incremental pipelines is challenging. Data practitioners who take on this challenge are viewed as performing an "advanced" function, which discourages broader teams from adopting incremental loads. In this lightning talk, CTO of Tobiko Data, Toby Mao, will demystify incremental loading data at scale.

Correctly Loading Incremental Data at Scale

Alluxio, Inc.

Big Data Bellevue Meetup March 21, 2024 For more Alluxio events: https://alluxio.io/events/ Speakers: Bin Fan (VP of Open Source, Alluxio) In this presentation, Bin Fan (VP of Open Source @ Alluxio) will address a critical challenge of optimizing data loading for distributed Python applications within AI/ML workloads in the cloud, focusing on popular frameworks like Ray and Hugging Face. Integration of Alluxio’s distributed caching for Python applications is accomplished using the fsspec interface, thus greatly improving data access speeds. This is particularly useful in machine learning workflows, where repeated data reloading across slow, unstable or congested networks can severely affect GPU efficiency and escalate operational costs. Attendees can look forward to practical, hands-on demonstrations showcasing the tangible benefits of Alluxio’s caching mechanism across various real-world scenarios. These demos will highlight the enhancements in data efficiency and overall performance of data-intensive Python applications. This presentation is tailored for developers and data scientists eager to optimize their AI/ML workloads. Discover strategies to accelerate your data processing tasks, making them not only faster but also more cost-efficient.

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio, Inc.

Alluxio Monthly Webinar Feb. 27, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer, Alluxio) As GenAI and AI continue to transform businesses, scaling these workloads requires optimized underlying infrastructure. A multi-cloud architecture allows organizations to leverage different cloud services to meet diverse workload demands while maximizing efficiency, reducing costs, and avoiding vendor lock-in. However, achieving a multi-cloud vision can be challenging. In this webinar, Tarik will share how an agonistic data layer, like Alluxio, allows you to embrace the separation of storage from compute and simplify the adoption of multi-cloud for AI. - Learn why leveraging multiple cloud providers is critical for balancing performance, scalability, and cost of your AI platform - Discover how an agnostic data layer like Alluxio provides seamless data access in multi-cloud that bridges storage and compute without data replication - Gain insights into real-world examples and best practices for deploying AI across on-prem, hybrid, and multi-cloud environments

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio, Inc.

Alluxio Monthly Webinar Jan. 30, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Kevin Petrie (VP of Research, Eckerson Group) - Omid Razavi (SVP of Customer Success, Alluxio) 2024 is gearing up to be an impactful year for AI and analytics. Join us on January 30, as Kevin Petrie (VP of Research at Eckerson Group) and Omid Razavi (SVP of Customer Success at Alluxio) share key trends that data and AI leaders should know. This event will efficiently guide you with market data and expert insights to drive successful business outcomes. - Assess current and future trends in data and AI with industry experts - Discover valuable insights and practical recommendations - Learn best practices to make your enterprise data more accessible for both analytics and AI applications

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Juncheng Yang(Ph.D Candidate, @CMU) As a cache eviction algorithm, FIFO has a lot of attractive properties, such as simplicity, speed, scalability, and flash-friendliness. The most prominent criticism of FIFO is its low efficiency (high miss ratio). In this talk, I will describe a simple, scalable FIFO-based algorithm with three static queues (S3-FIFO). Evaluated on 6594 cache traces from 14 datasets, we show that S3- FIFO has lower miss ratios than state-of-the-art algorithms across traces. Moreover, S3-FIFO’s efficiency is robust — it has the lowest mean miss ratio on 10 of the 14 datasets. FIFO queues enable S3-FIFO to achieve good scalability with 6× higher throughput compared to optimized LRU at 16 threads. Our insight is that most objects in skewed workloads will only be accessed once in a short window, so it is critical to evict them early (also called quick demotion). The key of S3-FIFO is a small FIFO queue that filters out most objects from entering the main cache, which provides a guaranteed demotion speed and high demotion precision.

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jingwen Ouyang (Product Manager, @Alluxio) In this session, Jingwen presents an overview of using Alluxio Edge caching to accelerate Trino or Presto queries. She offers practical best practices for using distributed caching with compute engines. In addition, this session also features insights from real-world examples.

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Siyuan Sheng (Senior Software Engineer, @Alluxio) - Chunxu Tang (Research Scientist, @Alluxio) In this session, cloud optimization specialists Chunxu and Siyuan break down the challenges and present a fresh architecture designed to optimize I/O across the data pipeline, ensuring GPUs function at peak performance. The integrated solution of PyTorch/Ray + Alluxio + S3 offers a promising way forward, and the speakers delve deep into its practical applications. Attendees will not only gain theoretical insights but will also be treated to hands-on instructions and demonstrations of deploying this cutting-edge architecture in Kubernetes, specifically tailored for Tensorflow/PyTorch/Ray workloads in the public cloud.

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Alluxio, Inc.

Data Infra Meetup | ByteDance's Native Parquet Reader

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jing Zhao (Principal Engineer, @Uber) Uber builds one of the biggest data lakes in the industry, which stores exabytes of data. In this talk, we will introduce the evolution of our data storage architecture, and delve into multiple key initiatives during the past several years. Specifically, we will introduce: - Our on-prem HDFS cluster scalability challenges and how we solved them - Our efficiency optimizations that significantly reduced the storage overhead and unit cost without compromising reliability and performance - The challenges we are facing during the ongoing Cloud migration and our solutions

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio, Inc.

Alluxio Monthly Webinar Nov. 15, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer) - Beinan Wang (Senior Staff Engineer & Architect) Many companies are working with development architectures for AI platforms but have concerns about efficiency at scale as data volumes increase. They use centralized cloud data lakes, like S3, to store training data for AI platforms. However, GPU shortages add more complications. Storage and compute can be separate, or even remote, making data loading slow and expensive: 1) Optimizing a developmental setup can include manual copies, which are slow and error-prone 2) Directly transferring data across regions or from cloud to on-premises can incur expensive egress fees This webinar covers solutions to improve data loading for model training. You will learn: - The data loading challenges with distributed infrastructure - Typical solutions, including NFS/NAS on object storage, and why they are not the best options - Common architectures that can improve data loading and cost efficiency - Using Alluxio to accelerate model training and reduce costs

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Adit Madan (Director of Product Management, @Alluxio) In this session, Adit Madan, Director of Product Management at Alluxio, presents an overview of using distributed caching to accelerate model training and serving. He explores the requirements of data access patterns in the ML pipeline and offers practical best practices for using distributed caching in the cloud. This session features insights from real-world examples, such as AliPay, Zhihu, and more.

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (Cheif Architect, VP of Open Source, @Alluxio) As the AI landscape rapidly evolves, the advancements in generative AI technologies, such as ChatGPT, are driving a need for a robust AI infra stack. This opening keynote will explore the key trends of the AI infra stack in the generative AI era.

AI Infra Day | The AI Infra in the Generative AI Era

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Machine Learning Engineer, @Alluxio) - Shawn Sun (Software Engineer, @Alluxio) This hands-on session will discuss best practices for using PyTorch and Alluxio during model training on AWS. Chunxu and Lu will provide a step-by-step demonstration of how to use Alluxio on EKS as a distributed cache to accelerate computer vision model training jobs that read datasets from S3. This architecture significantly improves the utilization of GPUs from 30% to 90%+, archives ~5x faster training, and lower cloud storage costs.

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jordan Plawner (Global Director of Artificial intelligence Product Management and Strategy, @Intel) ChatGPT and other massive models represents an amazing step forward in AI, yet they do not solve real-world business problems. We will survey how the AI ecosystem has worked non-stop over this last year to take these all-purpose multi-task models and optimize them to they can be used by organizations to address domain specific problems. We will explain these new AI-for-the-real world techniques and methods such as fine tuning and how can be applied to deliver results which are highly performant with state-of-the-art accuracy while also being economical to build and deploy everywhere to enhance products and services.

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

Alluxio, Inc.

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Sally (Mihyoung) Lee (Senior Staff Engineer, TLM, @Uber) Machine learning models power Uber’s everyday business. However, developing and deploying a model is not a one-time event but a continuous process that requires careful planning, execution, and monitoring. This session will highlight Uber’s practice on the machine learning lifecycle to ensure high model quality.

AI Infra Day | Model Lifecycle Management Quality Assurance at Uber Scale

Alluxio, Inc.

Alluxio Monthly Webinar Oct. 3, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Greg Palmer (Lead Solutions Engineer) Model training requires extensive computational and GPU resources. When training models on AWS, loading data from S3 often becomes a major bottleneck, wasting valuable GPU cycles. Optimizing data loading can greatly reduce GPU idle time and increase GPU utilization. In this webinar, Greg Palmer will discuss best practices for efficient data loading during model training on AWS. He will demonstrate how to use Alluxio on EKS as a distributed cache to accelerate PyTorch training jobs that read datasets from S3. This architecture significantly improves the utilization of GPUs from 30% to 90%+, archives ~5x faster training, and lower cloud storage costs. What you will learn: - The challenges of feeding data-hungry GPUs in the cloud - How to accelerate model training by optimizing data loading on AWS - The reference architecture for running PyTorch jobs with Alluxio cache on EKS while reading data from S3, with benchmark results of training ResNet50 and BERT - How to use TensorBoard to identify bottlenecks in GPU utilization

Alluxio Monthly Webinar | Efficient Data Loading for Model Training on AWS

Alluxio, Inc.

Plus de Alluxio, Inc. (20)

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Optimizing Data Access for Analytics And AI with Alluxio

Speed Up Presto at Uber with Alluxio Caching

Correctly Loading Incremental Data at Scale

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Data Infra Meetup | ByteDance's Native Parquet Reader

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

AI Infra Day | The AI Infra in the Generative AI Era

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

AI Infra Day | Model Lifecycle Management Quality Assurance at Uber Scale

Alluxio Monthly Webinar | Efficient Data Loading for Model Training on AWS

Dernier

Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pretoria ● Abortion Pills For Sale In Pretoria ● Pretoria 🏥🚑!! Abortion Clinic Near Me Cost, Price, Women's Clinic Near Me, Abortion Clinic Near, Abortion Doctors Near me, Abortion Services Near Me, Abortion Pills Over The Counter, Abortion Pill Doctors' Offices, Abortion Clinics, Abortion Places Near Me, Cheap Abortion Places Near Me, Medical Abortion & Surgical Abortion, approved cyctotec pills and womb cleaning pills too plus all the instructions needed This Discrete women’s Termination Clinic offers same day services that are safe and pain free, we use approved pills and we clean the womb so that no side effects are present. Our main goal is that of preventing unintended pregnancies and unwanted births every day to enable more women to have children by choice, not chance. We offer Terminations by Pill and The Morning After Pill.” Our Private VIP Abortion Service offers the ultimate in privacy, efficiency and discretion. we do safe and same day termination and we do also womb cleaning as well its done from 1 week up to 28 weeks. We do delivery of our services world wide SAFE ABORTION CLINICS/PILLS ON SALE WE DO DELIVERY OF PILLS ALSO Abortion clinic at very low costs, 100% Guaranteed and it’s safe, pain free and a same day service. It Is A 45 Minutes Procedure, we use tested abortion pills and we do womb cleaning as well. Alternatively the medical abortion pill and womb cleansing !!!

Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

Announcing Codolex 2.0 from GDK Software

Jim McKeeth

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal.

%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...

masabamasaba

Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in Tembisa ● Abortion Pills For Sale in Tembisa ● Tembisa 🏥🚑!! Abortion Clinic Near Me Cost, Price, Women's Clinic Near Me, Abortion Clinic Near, Abortion Doctors Near me, Abortion Services Near Me, Abortion Pills Over The Counter, Abortion Pill Doctors' Offices, Abortion Clinics, Abortion Places Near Me, Cheap Abortion Places Near Me, Medical Abortion & Surgical Abortion, approved cyctotec pills and womb cleaning pills too plus all the instructions needed This Discrete women’s Termination Clinic offers same day services that are safe and pain free, we use approved pills and we clean the womb so that no side effects are present. Our main goal is that of preventing unintended pregnancies and unwanted births every day to enable more women to have children by choice, not chance. We offer Terminations by Pill and The Morning After Pill.” Our Private VIP Abortion Service offers the ultimate in privacy, efficiency and discretion. we do safe and same day termination and we do also womb cleaning as well its done from 1 week up to 28 weeks. We do delivery of our services world wide SAFE ABORTION CLINICS/PILLS ON SALE WE DO DELIVERY OF PILLS ALSO Abortion clinic at very low costs, 100% Guaranteed and it’s safe, pain free and a same day service. It Is A 45 Minutes Procedure, we use tested abortion pills and we do womb cleaning as well. Alternatively the medical abortion pill and womb cleansing !!!

Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

WSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity

WSO2

Direct Style Effect Systems -The Print[A] Example- A Comprehension Aid

Philip Schwarz

WSO2Con204 - Hard Rock Presentation - Keynote

WSO2

%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain

masabamasaba

%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...

masabamasaba

WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...

WSO2

%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg

masabamasaba

WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...

WSO2

WSO2CON 2024 - Does Open Source Still Matter?

WSO2

+971565801893 Mtp-Kit (500MG) Prices » Dubai [(+971565801893**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Leen Whatsapp +971565801893 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971565801893''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971565801893' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Clinic in Abu Dhabi, United Arab Emirates.+971565801893

+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...

Health

WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...

WSO2

%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein

masabamasaba

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

masabamasaba

%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...

masabamasaba

%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview

masabamasaba

%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...

masabamasaba

Dernier (20)

Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...

Announcing Codolex 2.0 from GDK Software

%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...

Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...

WSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity

Direct Style Effect Systems -The Print[A] Example- A Comprehension Aid

WSO2Con204 - Hard Rock Presentation - Keynote

%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain

%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...

WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...

%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg

WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...

WSO2CON 2024 - Does Open Source Still Matter?

+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...

WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...

%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...

%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview

%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...

The Missing Piece of On-Demand Clusters

1. The Missing Piece of On-Demand Clusters October 2016 Calvin Jia

2. About Me • Calvin Jia • Software Engineer @ Alluxio, Inc. • Alluxio PMC • #1 Alluxio Contributor • Twitter: @JiaCalvin 2

3. Outline • Cloud Architectures • Alluxio in Cloud Architectures • Takeaways 3

4. Cloud Architectures – Overview • Mostly service based, from providers – Amazon Web Services – Google Cloud Platform • Separate compute and storage clusters • Compute clusters are ephemeral 4

5. Cloud Architectures – Pros & Cons Pros • Low maintenance • Pay as you go • Elastic and scalable • Cost effective storage Cons • Lower Performance 5

6. Outline • Cloud Architectures • Alluxio in Cloud Architectures • Takeaways 6

7. Alluxio in Cloud Architectures – Overview • Deployed in compute clusters – Memory speed data access – Transparent data access to any storage • Simple to deploy – Mount storage systems similar to local disks 7

8. Alluxio in Cloud Architectures – Example 8

9. Alluxio in Cloud Architectures – Benefits 9 • Remedies the performance drawback • Acceleration due to memory-speed I/O • Designed to improve the affinity of compute and storage

10. Alluxio in Cloud Architectures – Data Path 10 FAST 104 - 105 MB/s MODERATE 103 - 104 MB/s SLOW 102 - 103 MB/s Only when necessary Limited Often SSD HDD Mem

11. Takeaways – Experiment Results 11 0 100 200 300 400 500 600 Initial Read Subsequent Read Read from Separate Job Runtime (seconds) Spark -‐ No Persist Spark -‐ Persist Alluxio

12. Takeaways – Alluxio & Cloud Architectures 12 • Cloud architectures have significant upsides • Alluxio alleviates the major downsides

13. Try out Alluxio 1.2.0 http://www.alluxio.org/releases 13

14. Read the blog athttp://www.alluxio.com/2016/08/accelerating-on-demand-data-analytics-with-alluxio Read the whitepaper at www.alluxio.com. Check out www.alluxio.org. Thank you! 14