This slide deck describes the proposal to add automated tiering* as a native feature in OpenStack Swift. It also provides design & implementation details of a tiering prototype built in OpenStack Swift. *Automated tiering is a feature provided by storage systems that include multiple media types like SSD, HDD, and Tape. It enables seamless movement of active data to high-performance storage media and inactive data to low-cost, high capacity storage media. Effective utilization of these tiers results in an overall total cost of ownership (TCO) reduction for the customers.

1. OpenStack Swift Tiering
Proposal
© 2015 NetApp, Inc. All rights reserved.1
Shiva Chaitanya
Saeed Ghanbari
Barry Benight
Deepak Kenchammana

2. Problem Statement
§ Data hosted by capacity storage solutions such as Swift go through a
long information lifecycle.
§ As data ages, it becomes more and more unlikely to be accessed by the
user.
§ Slow but cheap, specialized archival-class products lower in $/GB than
Swift are better suited to store cold data in order to meet business TCO
needs.
§ Examples of such archival-class products include tape-based systems,
public cloud offerings such as Amazon Glacier.
© 2015 NetApp, Inc. All rights reserved.2

3. Application vs Swift – who should implement
tiering ?
§ Applications can potentially include the appropriate segregation in their
logic and periodically move data between storage tiers in Swift.
§ Such use cases are well served by existing Storage Policy feature which
provides the necessary API support for compartmentalizing data.
§ This proposal addresses use cases where Swift operators manage
multiple tiers of storage devices transparent to the application.
§ Operator’s key objective for cold data is to achieve best TCO across
archival devices spanning multiple hardware generations and media
types (SMR drives, tape, optical).
§ This proposal
§ Caters to applications that will find it harder to adapt to changes in workload &
underlying infrastructure over time. Selection of candidate cold data for tiering and
movement mechanisms are best deferred to Swift in such instances.
§ Aims to complement the Storage Policy feature by including data tiering as a native
feature (ala Ceph) in Swift.
© 2015 NetApp, Inc. All rights reserved.3

4. Proposal
§ Native tiering functionality in Swift.
§ Swift can leverage the gradual change in data properties to periodically
move cold data to lower cost storage tiers.
§ We propose to add a new background process in Swift to perform data
movement between storage tiers.
§ Data movement will be seamless to the client – no change in object
access API or namespace.
§ Potential applications of tiering feature:
§ Move cold data from a swift backend to an external tape device
§ Move cold data from a higher-cost swift backend (e.g SSD-based) to a lower-cost swift
backend (e.g HDD-based)
§ Move cold data from a lower-density swift backend (e.g replication policy) to a higher-
density swift backend (e.g erasure-coding policy)
© 2015 NetApp, Inc. All rights reserved.4

5. © 2015 NetApp, Inc. All rights reserved.5
Stub ObjA
E
ObjA
meta-data
Swift DAS backend
Overview of Tiering Prototype
Object Server
Tiering-obj
BackendAPI
E Swift Tape …
WSGI middleware
Obj
App
Add-on/pluggable
middleware
Swift DAS backend
Container Server
Container-migrator
BackendAPI
E Swift Tape …
Background process
Add-on/pluggable
background process
Patches to
existing code
Container DB
‘E’ stands for a generic external device that
can be accessed through BackendAPI

6. Code changes
§ Code changes primarily in the form of (i) a new background process
container-migrator and (ii) object server middleware
§ Container-migrator:
§ Runs on each container server
§ Scans each tiering-enabled container’s DB to identify objects that need to
be de-staged to external backend
§ Currently, we implement a simple age-threshold policy to determine
candidates for movement
§ Candidate objects are moved to the lower tier backend by invoking a
standard multi-backend compatible API
§ Moved objects from Swift are replaced by “Stub objects”
§ By default, there are three copies of container DB for each container. We
ensure that only one container-migrator process operates on a container
DB at any given time.
© 2015 NetApp, Inc. All rights reserved.6

7. Code changes (cont’d)
§ Object server middleware:
§ Interprets stub object when a client happen to access a de-staged object and
reconstructs the original object
§ Fetches lower tier backend directly via a standard multi-backend compatible
API
§ Handles requests from background processes. E.g GETs from container-sync,
DELETEs from account-reaper
§ Change to object server function:
§ Pass on (size, etag) of full object to container update operation – instead of
the stub object’s.
© 2015 NetApp, Inc. All rights reserved.7

8. Stub object
§ Stub is a zero-sized sparse reference object that points to an object
stored in a different backend
§ Object metadata are attached to the stub
§ All the ‘pointer’ information are attached to the stub as system
metadata
§ E.g
§ X-Object-Sysmeta-Eb-Dest-Path
§ X-Object-Sysmeta-Eb-Dest-Size
§ X-Object-Sysmeta-Eb-Dest-Etag
© 2015 NetApp, Inc. All rights reserved.8

9. Stub object (cont’d)
§ X-Object-Sysmeta-Eb-Dest-Path header helps locate the main object
on a GET request
§ X-Object-Sysmeta-Eb-Dest-Size/Etag headers help service HEAD
operations without accessing main object
§ A stub object is treated as just another regular swift object by
background processes. E.g,
§ Subject to 3-way replication by object-replicator to ensure reliability
§ Processed by object-auditor to ensure integrity
© 2015 NetApp, Inc. All rights reserved.9

10. Backend API
§ Currently using a simple four-function abstract class that can be
extended for custom implementations for Tape, Glacier, Swift etc.
§ __init__(args)
§ ‘args’ is an opaque list of params that can encapsulate arbitrary backend-
specific configuration info to enable setup, authentication, etc.
§ PUT(content_iter, path)
§ GET(path)
§ DELETE(path)
§ We have API implementation for a Swift container backend. This can
be used to tier objects between two Swift backends with different
policies (e.g SSD <-> HDD, Replication <-> Erasure-coding)
© 2015 NetApp, Inc. All rights reserved.10

11. © 2015 NetApp, Inc. All rights reserved.11
Stub ObjA
E
ObjA
meta-data
Swift DAS backend
Tiering process flow (object de-staging)
Object Server
Tiering-obj
BackendAPI
E Swift Tape …
WSGI middleware
Obj
App
Add-on/pluggable
middleware
Swift DAS backend
Container Server
Container-migrator
BackendAPI
E Swift Tape …
Background process
Add-on/pluggable
background process
Patches to
existing code
Container DB
1
2
3
4
Scan container DB to find next
candidate object for de-staging
Retrieve object from object server
Replace object with stub in the object
server
Write object to the lower tier
1
2
3
4‘E’ stands for a generic external device that
can be accessed through BackendAPI

12. © 2015 NetApp, Inc. All rights reserved.12
Stub ObjA
E
ObjA
meta-data
Swift DAS backend
Object Server
Tiering-obj
BackendAPI
E Swift Tape …
WSGI middleware
Obj
App
Add-on/pluggable
middleware
Add-on/pluggable
background process
Patches to
existing code
3
4
Incoming client GET request
Proxy server relays request to an
object server
Retrieve object and check if it’s a stub
If stub, fetch object from lower tier
backend
1
2
3
4‘E’ stands for a generic external device that
can be accessed through BackendAPI
Proxy Server 21
Client GET flow for a de-staged object

13. © 2015 NetApp, Inc. All rights reserved.13
Stub ObjA
E
ObjA
meta-data
Swift DAS backend
Client DELETE flow for a de-staged object
Object Server
Tiering-obj
BackendAPI
E Swift Tape …
WSGI middleware
Obj
App
Add-on/pluggable
middleware
Add-on/pluggable
background process
Patches to
existing code
3
4
Incoming client DELETE request
Proxy server relays request to an object server
Check via an internal ‘HEAD’ if object is a stub
If stub, delete object in lower tier backend
Remove the stub object
1
2
3
4
‘E’ stands for a generic external device that
can be accessed through BackendAPI
Proxy Server 21
5
5

14. © 2015 NetApp, Inc. All rights reserved.14
Container server
Stub ObjA
E
ObjA
meta-data
Swift DAS backend
Internal requests – e.g from container-sync
Container-sync
Object Server
Tiering-obj
BackendAPI
E Swift Tape …
WSGI middleware
Obj
App
Add-on/pluggable
middleware
Patches to
existing code
2
3
GET request from
container-sync to object
server
Fetch object from DAS
backend
If stub, fetch the object
contents from lower tier
backend
1
2
3
1
‘E’ stands for a generic external device that
can be accessed through BackendAPI

15. Object batching
§ Minimize number of requests to the lower tier. This requirement
becomes important when communicating across WAN to remote DCs.
§ Maximize size of objects de-staged to lower tiers such as tape that are
more suited for large I/O.
§ To cater to above two requirements, the tiering process can coalesce
multiple candidate objects into a compound object before de-staging
§ Reference counts need to be maintained for compound objects to do
effective garbage collection when objects are deleted.
§ Our prototype implements the above batching mechanism and stores
reference counts for compound objects in a separate table in the
container DB.
© 2015 NetApp, Inc. All rights reserved.15

16. Other optimizations
§ Client DELETE requests for de-staged objects can be completed
asynchronously by background process to avoid increased latencies for
foreground API requests.
§ On a client GET request for a de-staged object, we can “promote” it
back into Swift (higher tier) backend. Subsequently as the object ages,
tiering process will eventually move it back.
© 2015 NetApp, Inc. All rights reserved.16