Submit Search
Upload
Durability Simulator Design for OpenStack Swift
•
1 like
•
522 views
Kota Tsuyuzaki
Follow
Slides for OpenStack 2014 Paris Design Summit.
Read less
Read more
Software
Report
Share
Report
Share
1 of 20
Download now
Download to read offline
Recommended
Global EC Cluster Updates (OpenStack Mitaka Swift Design Summit)
Global EC Cluster Updates (OpenStack Mitaka Swift Design Summit)
Kota Tsuyuzaki
More Efficient Object Replication in OpenStack Summit Juno
More Efficient Object Replication in OpenStack Summit Juno
Kota Tsuyuzaki
NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus
Hirofumi Ichihara
Contiki OS Research Projects Guidance
Contiki OS Research Projects Guidance
Network Simulation Tools
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
Jakub Pavlik
NTTs Journey with Openstack-final
NTTs Journey with Openstack-final
shintaro mizuno
CFI 2015 - Flow-centric Visibility Tools for OF@TEIN
CFI 2015 - Flow-centric Visibility Tools for OF@TEIN
GIST (Gwangju Institute of Science and Technology)
Building the IoT - Coding Serbia 2015
Building the IoT - Coding Serbia 2015
Benjamin Cabé
Recommended
Global EC Cluster Updates (OpenStack Mitaka Swift Design Summit)
Global EC Cluster Updates (OpenStack Mitaka Swift Design Summit)
Kota Tsuyuzaki
More Efficient Object Replication in OpenStack Summit Juno
More Efficient Object Replication in OpenStack Summit Juno
Kota Tsuyuzaki
NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus
Hirofumi Ichihara
Contiki OS Research Projects Guidance
Contiki OS Research Projects Guidance
Network Simulation Tools
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
Jakub Pavlik
NTTs Journey with Openstack-final
NTTs Journey with Openstack-final
shintaro mizuno
CFI 2015 - Flow-centric Visibility Tools for OF@TEIN
CFI 2015 - Flow-centric Visibility Tools for OF@TEIN
GIST (Gwangju Institute of Science and Technology)
Building the IoT - Coding Serbia 2015
Building the IoT - Coding Serbia 2015
Benjamin Cabé
OpenContrail Implementations
OpenContrail Implementations
Jakub Pavlik
OpenPOWER ADG key note
OpenPOWER ADG key note
Ganesan Narayanasamy
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Takashi Torii
OpenStack Ottawa Q3 Meetup September 26th 2017
OpenStack Ottawa Q3 Meetup September 26th 2017
Stacy Véronneau
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
VMware Tanzu
CERN User Story
CERN User Story
Tim Bell
Collect, summarize and notify of OpenStack's log
Collect, summarize and notify of OpenStack's log
NTT Communications Technology Development
Containers on Baremetal and Preemptible VMs at CERN and SKA
Containers on Baremetal and Preemptible VMs at CERN and SKA
Belmiro Moreira
Future semantic segmentation with convolutional LSTM
Future semantic segmentation with convolutional LSTM
Kyuri Kim
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
Masaaki Nakagawa
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
TEST Huddle
BIRTE-13-Kawashima
BIRTE-13-Kawashima
Hideyuki Kawashima
Understanding performance aspects of etcd and Raft
Understanding performance aspects of etcd and Raft
Hitoshi Mitake
Modeling the IoT with TitanDB and Cassandra
Modeling the IoT with TitanDB and Cassandra
twilmes
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
Yuji Kubota
Tsinghua University: Two Exemplary Applications in China
Tsinghua University: Two Exemplary Applications in China
DataStax Academy
Icbai 2018 ver_1
Icbai 2018 ver_1
BlackhatGAURAV
Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)
Vincenzo Gulisano
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET Journal
The Pill for Your Migration Hell
The Pill for Your Migration Hell
Databricks
Transport SDN & OpenDaylight Use Cases in Korea
Transport SDN & OpenDaylight Use Cases in Korea
Justin Park
Online learning for low-latency streaming
Online learning for low-latency streaming
Theo Karagkioules
More Related Content
What's hot
OpenContrail Implementations
OpenContrail Implementations
Jakub Pavlik
OpenPOWER ADG key note
OpenPOWER ADG key note
Ganesan Narayanasamy
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Takashi Torii
OpenStack Ottawa Q3 Meetup September 26th 2017
OpenStack Ottawa Q3 Meetup September 26th 2017
Stacy Véronneau
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
VMware Tanzu
CERN User Story
CERN User Story
Tim Bell
Collect, summarize and notify of OpenStack's log
Collect, summarize and notify of OpenStack's log
NTT Communications Technology Development
Containers on Baremetal and Preemptible VMs at CERN and SKA
Containers on Baremetal and Preemptible VMs at CERN and SKA
Belmiro Moreira
What's hot
(8)
OpenContrail Implementations
OpenContrail Implementations
OpenPOWER ADG key note
OpenPOWER ADG key note
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
OpenStack Ottawa Q3 Meetup September 26th 2017
OpenStack Ottawa Q3 Meetup September 26th 2017
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
CERN User Story
CERN User Story
Collect, summarize and notify of OpenStack's log
Collect, summarize and notify of OpenStack's log
Containers on Baremetal and Preemptible VMs at CERN and SKA
Containers on Baremetal and Preemptible VMs at CERN and SKA
Similar to Durability Simulator Design for OpenStack Swift
Future semantic segmentation with convolutional LSTM
Future semantic segmentation with convolutional LSTM
Kyuri Kim
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
Masaaki Nakagawa
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
TEST Huddle
BIRTE-13-Kawashima
BIRTE-13-Kawashima
Hideyuki Kawashima
Understanding performance aspects of etcd and Raft
Understanding performance aspects of etcd and Raft
Hitoshi Mitake
Modeling the IoT with TitanDB and Cassandra
Modeling the IoT with TitanDB and Cassandra
twilmes
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
Yuji Kubota
Tsinghua University: Two Exemplary Applications in China
Tsinghua University: Two Exemplary Applications in China
DataStax Academy
Icbai 2018 ver_1
Icbai 2018 ver_1
BlackhatGAURAV
Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)
Vincenzo Gulisano
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET Journal
The Pill for Your Migration Hell
The Pill for Your Migration Hell
Databricks
Transport SDN & OpenDaylight Use Cases in Korea
Transport SDN & OpenDaylight Use Cases in Korea
Justin Park
Online learning for low-latency streaming
Online learning for low-latency streaming
Theo Karagkioules
Globecom 2015: Adaptive Raptor Carousel for 802.11
Globecom 2015: Adaptive Raptor Carousel for 802.11
Andrew Nix
The Role of Inter-Controller Traffic in SDN Controllers Placement
The Role of Inter-Controller Traffic in SDN Controllers Placement
Paolo Giaccone
OpenTelemetry For Architects
OpenTelemetry For Architects
Kevin Brockhoff
Digital_system_design_A (1).ppt
Digital_system_design_A (1).ppt
BUCHUPALLIVIMALAREDD2
Basic Tutorial for Robotic Arm
Basic Tutorial for Robotic Arm
Yu Wei Chen
Speeding up information extraction programs: a holistic optimizer and a learn...
Speeding up information extraction programs: a holistic optimizer and a learn...
INRIA-OAK
Similar to Durability Simulator Design for OpenStack Swift
(20)
Future semantic segmentation with convolutional LSTM
Future semantic segmentation with convolutional LSTM
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
BIRTE-13-Kawashima
BIRTE-13-Kawashima
Understanding performance aspects of etcd and Raft
Understanding performance aspects of etcd and Raft
Modeling the IoT with TitanDB and Cassandra
Modeling the IoT with TitanDB and Cassandra
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
Tsinghua University: Two Exemplary Applications in China
Tsinghua University: Two Exemplary Applications in China
Icbai 2018 ver_1
Icbai 2018 ver_1
Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
The Pill for Your Migration Hell
The Pill for Your Migration Hell
Transport SDN & OpenDaylight Use Cases in Korea
Transport SDN & OpenDaylight Use Cases in Korea
Online learning for low-latency streaming
Online learning for low-latency streaming
Globecom 2015: Adaptive Raptor Carousel for 802.11
Globecom 2015: Adaptive Raptor Carousel for 802.11
The Role of Inter-Controller Traffic in SDN Controllers Placement
The Role of Inter-Controller Traffic in SDN Controllers Placement
OpenTelemetry For Architects
OpenTelemetry For Architects
Digital_system_design_A (1).ppt
Digital_system_design_A (1).ppt
Basic Tutorial for Robotic Arm
Basic Tutorial for Robotic Arm
Speeding up information extraction programs: a holistic optimizer and a learn...
Speeding up information extraction programs: a holistic optimizer and a learn...
More from Kota Tsuyuzaki
Storlets Project Update for Train
Storlets Project Update for Train
Kota Tsuyuzaki
Project Updates Storlets Denver 2019
Project Updates Storlets Denver 2019
Kota Tsuyuzaki
NVIDIA/deepopsを触ってみた話
NVIDIA/deepopsを触ってみた話
Kota Tsuyuzaki
OpenStack Swift Introduction 2019
OpenStack Swift Introduction 2019
Kota Tsuyuzaki
Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage
Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage
Kota Tsuyuzaki
OpenStack Summit Storlets Project Update Queens
OpenStack Summit Storlets Project Update Queens
Kota Tsuyuzaki
OpenStack Swiftの最新機能とStorlets
OpenStack Swiftの最新機能とStorlets
Kota Tsuyuzaki
Using Storlets/Docker For Large Scale Image Processing
Using Storlets/Docker For Large Scale Image Processing
Kota Tsuyuzaki
OpenStack Swift紹介
OpenStack Swift紹介
Kota Tsuyuzaki
OpenStack Summit Vancouver Swift 報告
OpenStack Summit Vancouver Swift 報告
Kota Tsuyuzaki
Container Listing Update (Liberty Swift Design Summit)
Container Listing Update (Liberty Swift Design Summit)
Kota Tsuyuzaki
日本OpenStackユーザ会 Atlantaサミット報告会 Swift関連報告
日本OpenStackユーザ会 Atlantaサミット報告会 Swift関連報告
Kota Tsuyuzaki
More from Kota Tsuyuzaki
(12)
Storlets Project Update for Train
Storlets Project Update for Train
Project Updates Storlets Denver 2019
Project Updates Storlets Denver 2019
NVIDIA/deepopsを触ってみた話
NVIDIA/deepopsを触ってみた話
OpenStack Swift Introduction 2019
OpenStack Swift Introduction 2019
Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage
Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage
OpenStack Summit Storlets Project Update Queens
OpenStack Summit Storlets Project Update Queens
OpenStack Swiftの最新機能とStorlets
OpenStack Swiftの最新機能とStorlets
Using Storlets/Docker For Large Scale Image Processing
Using Storlets/Docker For Large Scale Image Processing
OpenStack Swift紹介
OpenStack Swift紹介
OpenStack Summit Vancouver Swift 報告
OpenStack Summit Vancouver Swift 報告
Container Listing Update (Liberty Swift Design Summit)
Container Listing Update (Liberty Swift Design Summit)
日本OpenStackユーザ会 Atlantaサミット報告会 Swift関連報告
日本OpenStackユーザ会 Atlantaサミット報告会 Swift関連報告
Recently uploaded
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
confluent
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
9953056974 Low Rate Call Girls In Saket, Delhi NCR
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprise
preethippts
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. Salesforce
BrainSell Technologies
EY_Graph Database Powered Sustainability
EY_Graph Database Powered Sustainability
Neo4j
Advantages of Odoo ERP 17 for Your Business
Advantages of Odoo ERP 17 for Your Business
Envertis Software Solutions
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
smiwainfosol
Best Web Development Agency- Idiosys USA.pdf
Best Web Development Agency- Idiosys USA.pdf
IdiosysTechnologies1
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
stazi3110
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
Christoph Pohl
Implementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with Azure
Dinusha Kumarasiri
Automate your Kamailio Test Calls - Kamailio World 2024
Automate your Kamailio Test Calls - Kamailio World 2024
Andreas Granig
Cyber security and its impact on E commerce
Cyber security and its impact on E commerce
manigoyal112
2.pdf Ejercicios de programación competitiva
2.pdf Ejercicios de programación competitiva
Diego Iván Oliveros Acosta
How to Track Employee Performance A Comprehensive Guide.pdf
How to Track Employee Performance A Comprehensive Guide.pdf
Livetecs LLC
Xen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdf
Stefano Stabellini
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
jennyeacort
Cloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEE
VICTOR MAESTRE RAMIREZ
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
Alina Yurenko
Software Project Health Check: Best Practices and Techniques for Your Product...
Software Project Health Check: Best Practices and Techniques for Your Product...
Velvetech LLC
Recently uploaded
(20)
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprise
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. Salesforce
EY_Graph Database Powered Sustainability
EY_Graph Database Powered Sustainability
Advantages of Odoo ERP 17 for Your Business
Advantages of Odoo ERP 17 for Your Business
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Best Web Development Agency- Idiosys USA.pdf
Best Web Development Agency- Idiosys USA.pdf
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
Implementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with Azure
Automate your Kamailio Test Calls - Kamailio World 2024
Automate your Kamailio Test Calls - Kamailio World 2024
Cyber security and its impact on E commerce
Cyber security and its impact on E commerce
2.pdf Ejercicios de programación competitiva
2.pdf Ejercicios de programación competitiva
How to Track Employee Performance A Comprehensive Guide.pdf
How to Track Employee Performance A Comprehensive Guide.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdf
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Cloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEE
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
Software Project Health Check: Best Practices and Techniques for Your Product...
Software Project Health Check: Best Practices and Techniques for Your Product...
Durability Simulator Design for OpenStack Swift
1.
Copyright©2014 NTT Corp.
All Rights Reserved. Durability Simulator Design for OpenStack Swift (Interactive Durability Calculation Tools) Kota Tsuyuzaki [IRC: kota_] tsuyuzaki.kota@lab.ntt.co.jp NTT Software Innovation Center Copyright(c)2009-2014 NTT CORPORATION. All Rights Reserved.
2.
2 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Goal & Benefits •How to calculate? •Demo Outline Etherpad: https://etherpad.openstack.org/p/kilo-swift-durability-simulator
3.
3 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Issue User I wanna build a durable object storage system by using OpenStack Swift. I wanna know also the durability to confirm it will be enough for our SLA.
4.
4 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Issue User Provider A Provider B Provider C Hey, guys. Could you tell me the Swift system architecture and its storage durability you support. OpenStack Providers
5.
5 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Issue User Provider A Provider B Provider C A: 7-9s durability with 3 copies B: 9-9s durability with 3 copies C: 11-9s durability with 3 copies WHAT’S HAPPEN!? WHICH IS CORRECT? OpenStack Providers
6.
6 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Goal •Building durability calculation tools supported (or recommended) by Swift community •Enabling to get the calculation result easily from both specs of system component HWs and swift configures. (e.g. # of disks, size of each disk, # of partitions) •Benefits •Swift Administrators (almost beginners) can find their own system durability easily •Enable to standardize the calculation definition among Swift providers •Swift Users can choose the policy for their use case (Replica? EC? Which # of parities are best for you?) Goal & Benefits
7.
7 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential How to calculate the durability?
8.
8 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential For Replica Case
9.
9 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Calculation Using Markov Model (Markov Process) •2 Replica -> k = 1, m = 1 •i.e. Data Lost with 2 Fragments •3 Replica -> k = 1, m = 2 •i.e. Data Lost with 3 Fragments •Reference: •[1]: "Reliability Mechanisms for Very Large Storage Systems" •http://www.ssrc.ucsc.edu/Papers/xin-mss03.pdf How to Calculate EC Durability? [1]
10.
10 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Redundancy Set[1]: •Definition •A block group composed of data blocks or object and their associated replicas or parity blocks. A single redundancy set will typically contain 1MB to 1TB, though we expect that redundancy sets will be at least 1GB to minimize bookkeeping overhead and reduce the likelihood that two redundancy sets will be stored on the same set of object storage system. •Assuming a Reduandancy Set as a Partition Consideration for Swift’s Partition Ring MD5*(URL) = index partitions idx Copy 1 Copy 2 Copy 3 0 1 5 7 … … … … 8 3 2 6 Partition table from part to device id. From [1]
11.
11 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Definition: •Absorbing State: The end state in the state transition model. •P: Transition Probability Matrix Markov Process (1) Absorbing State Temporary State P=푄푈 푂퐼 ퟏ−ퟐ흁ퟐ흁ퟎ 풗ퟏ−(흁+풗)흁 ퟎퟎퟏ Q: Transition Probability Matrix among Temporary State U: Probability Matrix from Temporary State into Absorbing State O: Zero Matrix、I: Identity Matrix State0 State1 State2
12.
12 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Time (t) Limitation of State Transition Matrix (P) shows average # of state transition (M) from initial state to absorbing state •MTTDL (Time to be absorbing state) calculated from sum of each rows in MN Markov Process (2) 퐥퐢퐦 풕→∞ 푷풕=ퟎ푴푼 ퟎ푰 M = (I-Q)-1 MTTDLrs = M ퟏ ⋮ ퟏ P= ퟏ−ퟐ흁ퟐ흁ퟎ 풗ퟏ−(흁+풗)흁 ퟎퟎퟏ ퟏ ퟐ흁 흁+풗 흁 ퟐ 풗 흁 ퟐ State Transition Matrix for 2 replica M MTTDLrs ퟏ ퟐ흁ퟐ ퟑ흁+풗 ퟐ흁+풗 Durability = 1 – N/ MTTDLrs Probability for Data Lost Durability 1 - 2푵흁ퟐ ퟏ ퟑ흁+풗 ퟏ ퟐ흁+풗
13.
13 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential For EC Case
14.
14 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Object Size(bytes): n •# of Sliced Raw Objects: k •# of Parities: m •Total # of Fragments: k + m •Fragment Size(bytes): n / k (+ checksum) •Total Stored Size (bytes): Fragment Size * (k + m) Erasure Code Definition object Data fragment Data fragment parity fragment parity fragment … … k m encode decode Terminology Reference: http://specs.openstack.org/openstack/ swift-specs/specs/swift/erasure_coding.html
15.
15 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Basic Idea •Expansion of Durability Calculation for Replica Model •Calculation Using Markov Model (Markov Process) •Replica Model based on Markov Process: •2 Replica -> k = 1, m = 1 •i.e. Data Lost with 2 Fragments •3 Replica -> k = 1, m = 2 •i.e. Data Lost with 3 Fragments How to Calculate EC Durability? [1] ※ Markov Process works to calculate the durability with matrix calculation. [3]
16.
16 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Algorithms •State: Status (exists or lost) for All fragments •Each state is transferred by constant probability •μ = Disk Failure Rate, v = Fragments Repair Rate •Each Rate related to # of Fragments •E.g. RAID related to # of Devices •Extract States to m + 1 (i.e. data lost) Durability Calculation Algorithms 0 1 m-1 m … m+1 state transitions for “m” parities EC D = # of Devices (RAID5) N = k + m (N fragments located in the system) -Nμ v Nμ -(N-1)μ-v (N-m)μ mv (N-(m-1))μ -(N-(m-1))μ-mv
17.
17 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Demo
18.
18 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Demo
19.
19 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Demo
20.
20 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Kota Tsuyuzaki [IRC: kota_] tsuyuzaki.kota@lab.ntt.co.jp NTT Software Innovation Center Questions? Etherpad: https://etherpad.openstack.org/p/kilo-swift-durability-simulator
Download now