S106285 spectrum-archive-taming-big data-istanbul-v1902a

2019 IBM Systems
Technical University
February 6-8
Istanbul, Turkey
IBM Spectrum Archive:
Taming big data with LTFS standard
Tony Pearson
Master Inventor, Senior IT Architect
IBM Systems Lab Services

Abstract
2 © Copyright IBM Corporation 2019
IBM Spectrum Archive provides a scalable, cost effective
solution for the ever-expanding storage requirements
for big data archives feeding analytics solutions.
Spectrum Archive utilizes IBM LTFS technology and IBM
Spectrum Scale to extend the storage infrastructure to
lower cost and improve manageability of big data
storage.
Now, the IBM Spectrum Storage family can provide a
single, integrated solution for flash, disk, and LTFS tape.

Agenda
3
— Part 1: IT Challenges
— Part 2: What is LTFS Standard?
— Part 3: Why IBM Spectrum Archive
— Part 4: Introduction into IBM Spectrum
Archive Enterprise Edition (EE) and its
features
© Copyright IBM Corporation 2019

Data Growth
0
20
40
60
80
100
120
2009 2010 2011 2012 2013 2014 2015 2016 2017
Unstructured Data
Structured Data
*Exabytes
*1 exabyte = 1,000 petabytes = 1 million terabytes = 1 billion gigabytes Source: IDC
Unstructured
data growth
of
60 - 80%
per year

With Growth Comes Complexity of Management
332%
growth in mobile data traffic
between 2015 and 20181
90%of total mobile data traffic will
be cloud apps by 20192
10x
growth of the amount of data
on the planet by 20203
80%
of all data is unstructured (web,
social, video, audio, pictures,
scans, email)4
1. Extrapolated from Gartner press release, http://www.gartner.com/newsroom/id/3098617; 2. Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2014-2019
3. IDC annual Digital Universe study, http://www.computerweekly.com/news/2240217788/Data-set-to-grow-10-fold-by-2020-as-internet-of-things-takes-off; 4. IBM data

Data is being generated everywhere

Data content is growing too fast to manage
yet IT budgets are shrinking

Traditional Storage Models are being disrupted by the explosion of data

Business SLAs Challenging Traditional Storage Approaches

Solution: Cost Optimization by Policy-based Multi-Tier Storage
TS4500
Tape Library
Last
Accessed
> 30days
Last
Accessed
> 60days
When Silver pool is >60% full
Drain it down to 20%
Spectrum ScaleSpectrum Scale
Accessed
today and
file size is
<1G
Send it back
to Silver pool
when
accessed
System pool
(SSD)
System pool
(SSD)
Gold pool
(SAS)
Gold pool
(SAS)
Silver pool
(NL SAS)
Silver pool
(NL SAS)
Spectrum ArchiveSpectrum Archive
Off-site storage by trucking

Tape Evolution – Denser, Faster, and Cheaper
EARLY DAYS
IBM 726 Tape System,
announced on May 21, 1952
IBM TS1155 Tape System,
announced on May 8, 2017
TODAY
15TB in palm of your hand
(45TB with 3:1 compression)
Sustained I/O Rate @ 360MB/s
FUTURE
Equivalent to 330TB
in single tape
Research Prototype
Press release on Aug 2, 2017
2million digits
in single 8-inch reel

Tape Storage Advantages
—It’s reliable
• Tape Drives and Cartridges are very robust
• Tape Storage exists for more than 60 years
—It’s cost-effective
• Lowest storage cost for the foreseeable
future
—It’s environment-friendly
• Tape is the most energy efficient method of
storing data
• Cartridges in a shelf consume no energy
—It’s scalable
• Easy to add additional storage (i.e. add
cartridges)
• Tape provides “infinite capacity” on demand
—It’s removable, transportable and
shareable
• Off side storage for disaster recovery
• Cartridges are easy to ship (XX PetaBytes /
Day)
—It’s integrated with data
management solutions
• Tape Library
• 24x7 operation for Backup, ILM, HSM
• Linear Tape File System (LTFS)

Two Tape Technology Options
IBM TS1155 LTO-8
Enterprise Tape Multi-vendor solution
Native Capacity
(latest product)
JD media: 15TB
(45TB with 1:3 compression)
L8 media: 12TB
(30TB with 1:2.5 compression)
M8 media: 9TB (22.5TB)
Sustained Transfer Rate 360MB/s 360MB/s (Full Height Drive)
300MB/s (Half Height Drive)

Tape Evolution – Long Term Roadmap
http://www.lto.org/technology/what-is-lto-technology/
feature since LTO-5
LTO-8: 12TB/9TB native capacity
Roadmap To LTO-12
Announced in Oct/2018
New

Storage Technologies Areal Density Trends

Data Archive Requirements
— Open Format
• Non-proprietary data format
• Upgradable and adaptable to future technology
— Self-Describing Containers of data
• No external database or reference required to recover / read /
transfer / sell / move data on containers
— Cross-Platform Interchange
• Data accessible via Linux, MacOS, Windows

Tape Evolution – New Use Case with Liner Tape File System
— File System Designed for Tape
• Not traditional use of Tape by TAR command nor Tape Backup Software
— Allows the user to access to the data on tape using the file
manager (Windows Explorer), just like USB memory stick and
CD/DVD
• Application program can access to the files directly without
tape backup/restore program or database
LTFS-format
cartridge
Directory Structure
File List ViewCD/DVD disc HDD
USB Memory
15TB
128GB

LTFS Advantages
— File System designed for Long-Term Retention and Media Portability
— Award-winning technology, invented and maintained by IBM
o File System implementation available as open source since 2010
— Open Standard
o ISO/IEC 20919:2016
o Data structure on tape
> Two Partitions – Index Partition and Data Partition
o Industry Collaboration - SNIA Technical Working Group
Version 2.4 approved in 2017
o Logo Program (Compatibility Testing) by LTO Consortium
— Self-Describing
o Information Exchange by Tape – No Vendor Lock-in
o No separate DB server for managing tape contents
— Available as IBM Spectrum Archive
o 3 Editions; Single Drive Edition (SDE), Library Edition (LE), and
Enterprise Edition (EE)

Loading the tape cartridge
• Tape Drive icon appears on Windows explorer, and it changes to
the Cartridge icon at the mount
• Use standard copy command or File Browser to
move/search/open the files
IBM Spectrum Archive Single Drive Edition (SDE)
External LTO Tape Drive
Server with
IBM Spectrum Archive SDE
Linux, MacOS, Windows:
Binary modules are downloadable from IBM
Fix Central
Source Code (under BSD license):
https://github.com/LinearTapeFileSyetem/ltfs

•Each tape cartridge will appear as
subdirectory under the mount point
(Linux) or drive letter (Windows)
• Barcode-based Identification
•Use Filesystem alone or use as the
toolkit for 3rd Party Archive Solution
• Managed through POSIX API and CLI
Internal View of Tape Library Hardware
IBM Spectrum Archive Library Edition (LE)
IBM or non-IBM Tape Library
Windows Explorer View
Server with
IBM Spectrum Archive LE
Linux, Windows:
Orderable from AAS (with SWMA)
Free Version (without Warranty/Support)
available from IBM Fix Central, too

IBM Spectrum Archive Enterprise Edition (EE)
• Automated Data Management
• Seamlessly incorporates tape storage under the single
namespace
• Keep data in active archive at much lower costs
• Policy-based data placement
• Persistent view of the data - data still listed in
directories
• Once data is accessed it is moved to disk – recall
on demand
• Tape as the external pool of Spectrum Scale
• Tapes in LTFS format
• Exporting tapes to other site and accessible with
other LTFS software
• CLI, REST API, Grafana-based dashboard screen
Flash
Gold Pool
Disk
Silver Pool
Tier 1
Tier 2
Single name space
Spectrum Scale
CIO Finance Engineering
Tape
LTFS
Tier 3
Spectrum Archive EE
Up to 500 PB
(with TS1155 Tape Drives)
2 Tape Libraries
Multiple Protocol Support
Client Applications
Linux:
Orderable from AAS or PPA
Trial Version available from IBM Web site

IBM Spectrum Archive: Editions and Deployment
Options
23
Management and
Integration of IBM Tape
Hardware Features
Data Movement
Job Management
Tape Pool Management
Hardware Agnostic
File API
IBM Asset Management Solutions
- Archive & Essence Manager (AREMA)
- Or, 3rd Party Software
Licensed / Free
Free
Licensed Software
Spectrum Archive Library Edition (LE)
•Integrates the support of tape automation
•Scalable storage space by 1U TS2900 to TS4500
•Supports Linux and Windows
Spectrum Archive
Enterprise Edition (EE)
•Integration of GPFS and LTFS
•Multi-node scale-out capability
Spectrum Archive Single Drive Edition (SDE)
•Free download
•Support IBM LTO and Enterprise Tape TS11xx
•Supports Linux, Mac, and Windows
SwiftHLM
Swift API

Integrates tapes as low cost tier of your operational storage

TS4500
Tape Library
Last
Accessed
> 30days
Last
Accessed
> 60days
When Silver pool is >60% full
Drain it down to 20%
Spectrum ScaleSpectrum Scale
Accessed
today and
file size is
<1G
Send it back
to Silver pool
when
accessed
System pool
(SSD)
System pool
(SSD)
Gold pool
(SAS)
Gold pool
(SAS)
Silver pool
(NL SAS)
Silver pool
(NL SAS)
Spectrum ArchiveSpectrum Archive
Policy-based Control of Data Lifecycle and Cost
— Powerful policy engine
Fast metadata ”scanning” and data
movement
Automated data migration based on
threshold
— Users not affected by data migration
Single namespace
Persistent view of the data

Functional Overview
Spectrum Scale Node 1 Spectrum Scale Node
2
Global name space
User file systemUser file system
Spectrum Archive
Node 1
Pool 1
Migration with
optional copy to
other tape
Recall with
option for bulk
recall
Tape management: reclamation (free space) and reconcilation (synchronize)
Export with
option to keep
stub in GPFS
Import
(only creates
stubs in GPFS)
Rebuild file
system
Library
Pool 2
Spectrum Archive
Node 2

EE 1.2.6: In-Pool Technology Upgrade using Intermixed Tape Pool
— Allow writing to specific media type
• “Appendable” tapes will be used for migration or as the target of reclaim
— Others will be treated as “Non-appendable” tape (still eligible for recall and source of reclaim)
— Media restriction can be set by “ltfsee pool set” command per pool
• No restriction by default
• Specify one media type For example, “JD” or “L7”
• 3592 media will be also enforced to use a particular format density (“E07”, “E08”, or “55F”)
— “Free space” in CLI and REST API indicates the appendable space
• New information (“Non-appendable space” and “Appendable” flag will be displayed
Appendable
(Intermixed Technology)
Appendable
(New Technology)
Non-Appendable
(Old Technology)
Media Restriction = L8
Legacy Pool Mixed Pool Split Mixed Pool
L8
L8L8
L8
pool1
pool2
pool3
pool4 pool5

EE 1.2.6: Pool-to-Pool Data Migration
— New “ltfsee datamigrate” - Allows to move the content of tape to different pool
• Source and Target Pools needs to be in the same library and same node group
• This cannot be used to increase the replicas
— Usage example
1. Redirect the new migration to new pool
1. Change the name of existing pool to something else (pool1 pool2)
2. Create a new pool with old name
2. Move the contents of old pool by using “ltfsee datamigrate” command
Stub file will be updated by datamigrate command
3. After all contents are transferred to new pool, withdraw the old pool
Old Pool New Pool
“datamigrate” command
Stub on GPFS
L5 L7
LTO5 drives LTO7 drives

File State Change
File State
Resident Premigrated Migrated
• Data is on disk • Data is on both disk and
tape
• Data is on tape only
• Initial state • Rapid access to the data
available on disk while
the redundant data is
available on tape
• Most economical
State will be
changed
transparently by
user file access, or
explicit command

Example of Data Lifecycle with 4 Tiers
Year 1 Year 2 Year 3-10
Disk
Silver Pool
Tier 2
Tape
LTFS
Tier 3
Flash
Gold Pool
Tier 1
Premigrated
(disk and tape)
Migrated
(tape only)
Resident
(data on disk)
Migrated &
Offline Export
Migration from
Flash to Disk
Migration from Disk to 2 Tapes
Data
Expiration
3 copies 2 copies
Single
online
copy
Read recall
(pre-migrated)

Building Block Options for Scale Out
1. Servers
• CPU - x86 or POWER Little Endian
• Software – Spectrum Scale and
Spectrum Archive
2. Disk Storage
• Internal or External
3. Tape Drive and Tape Media
• Drive Type (LTO or Enterprise Tape,
Generation)
• WORM or Non-WORM
• Interconnect (HBA, Switches, Zoning)
4. Tape Library
• Up to 2 tape libraries for redundancy
• TS4500 and TS3500 (houses up to
20000 tapes)
• 19 inch rack mount TS3310 and TS4300
Spectrum Scale
Server
Spectrum Archive
Server
App Server
Tape Library
NFS
CIFS
Object

Library
Pool
2
Pool 1
EE Node
1
F F
D D
T1T2T3 T4 T5
Ethernet
D D
EE Node
2
F F
GPFS Native
Client
GPFS Native
Client
GPFS Native
Client
GPFS Native
Client
GPFS Native
ClientCIFS/SMB Client
GPFS Native
Client
GPFS Native
ClientNFS Client
GPFS Native
Client
GPFS Native
ClientFTP Client
Free Tapes
SAN
EE Node
3
F F
D D
EE Node
4
F F
GPFS
D D
Spectrum Archive Cluster
Scale-out Configuration

Multi-Library Configuration (Library Redundancy and Capacity Scale-out)
Library 1
Pool 2Pool 1
EE Node
1
F F
D D
T1T2T3 T4 T5
Ethernet
Library 2
Pool BPool A
D D
Ta Tb Tc Td Te
D D
EE Node
2
F F
EE Node
3
F F
EE Node
4
F F
D D
GPFS Native
Client
GPFS Native
Client
GPFS Native
Client
GPFS Native
Client
GPFS Native
ClientCIFS/SMB Client
GPFS Native
Client
GPFS Native
ClientNFS Client
GPFS Native
Client
GPFS Native
ClientFTP Client
Free TapesFree Tapes
SAN
Spectrum Archive Cluster
Node Group 1 Node Group 2
GPFS

Management Interface
Policy-based Cost Optimization
— Rules execution by capacity threshold or by file criteria
(File Heat, Size, Name, …)
• Collocation of data by using tape pool
• File replication on multiple tapes (Up to 3 copies)
Spectrum Archive CLI for Tape Management
- Human-readable or JSON output
REST API
Monitoring
Server
Search
Engine
Data
Visualizer
Dashboard
Spectrum Archive CLI
Spectrum Scale
Policy Command
Spectrum Archive
Node
Data
Collector
RULE EXTERNAL POOL 'ltfsee'
EXEC '/opt/ibm/ltfsee/bin/ltfsee'
OPTS ‘-p Pool1@Library1 PoolA@Library2'
/* The following statement is the migration rule */
RULE 'ee_sysmig' MIGRATE FROM POOL 'system'
THRESHOLD(80,60)
WEIGHT(ACCESS_TIME)
TO POOL 'ltfsee'
WHERE (KB_ALLOCATED > 10000)

REST API

Dashboard - Performance Counter and Drive Configuration

Dashboard - Storage Usage and Pool Configuration

Object Interface: SwiftHLM High-Level Overview

Object Interface: 4K/8K Video Editing Workflow Demo at
NAB2018
4K/ 8K Camera
(Android Phone)
File transfer through
Phone App
TS4300
Tape Library
Archive
IBM Spectrum Scale
IBM Spectrum
Scale Object
+ SwiftHLM
IBMSpectrum Archive EE
(LTFS)
Migration
MAM
SwiftHLM
Browser access
LTO 8 - 12 TB
LTO 7 Type M - 9 TB
Editing Studio
Data on Tape
transparently recall-
able at any time
Mobile App.

Free Trial of IBM Spectrum Archive EE in Virtual Machine Image
Fast, simple, and flexible POC enablement
• Download and deploy in minutes
• Fully functional version for demonstrations, hands-on training,
functional testing, and data management planning
• Try it on your laptop, desktop, or server – no need for tape
hardware
• Requires VirtualBox
Get ready for managing ever growing big data with an economical, scalable archive solution
https://www-03.ibm.com/systems/storage/tape/ltfs/trial.html

41
Q: How do you
optimize storage TCO
for ever-growing enterprise
data?
0
20
40
60
80
100
120
2009 2010 2011 2012 2013 2014 2015 2016 2017
Unstructured Data
Structured Data

A: Archive your data
on tapes with
convenience of quick
access
42
LTFS
format

Advantages of IBM Spectrum Archive

TRANSFORM WITH NEXT-GEN APPLICATION
AI and Big Data infrastructure to support High
Performance Data Analytics and HPC leveraging the
power of the Open Source community, supporting the very
small to the largest clusters in the world
IBM Solutions
Private Clouds for traditional Application infrastructure that
are as agile, flexible and cost effective as Public Clouds,
that seamlessly extend to leverage Public Cloud Infrastructure
as business demands
MODERNIZE TRADITIONAL WORKLOADS
Patterns
APPLICATION REFACTORING / INTEGRATION
Integrated Docker and Kubernetes
Private Clouds for App modernization and agile
development with full portability to leverage Public
Cloud Infrastructure
Spectrum Virtualize
Spectrum Accelerate
Spectrum Archive
Spectrum Virtualize
Spectrum Protect
Spectrum CDM
Spectrum Scale
Spectrum ScaleSpectrum
Computing
Spectrum
Computing
Cloud Object Storage
Cloud Object Storage
IBM Z
Multi-Cloud Workload & Infrastructure Key Patterns

Resources
Product Manual (Knowledge Center) Redbooks
http://www.redbooks.ibm.com/redpieces/abstracts/sg248333.html
http://www.redbooks.ibm.com/abstracts/redp5384.html
http://www.redbooks.ibm.com/abstracts/redp5430.html
Available NowDraft Available Available Now
https://www.ibm.com/support/knowledgecenter/ST9MBR

Thank you!
Please complete the Session
Evaluation!
Speaker: Tony Pearson
Session: s106285
IBM Spectrum Archive:
Taming big data with LTFS standard

About the Speaker
47
Tony Pearson is a Master Inventor and Senior IT Architect for the IBM Storage product line. Tony joined
IBM Corporation in 1986 in Tucson, Arizona, USA, and has lived there ever since. In his current role,
Tony presents briefings on storage topics covering the entire IBM Storage product line, IBM Spectrum
Storage software products, and topics related to Cloud Computing, Analytics and Cognitive Solutions.
He interacts with clients, speaks at conferences and events, and leads client workshops to help clients
with strategic planning for IBM’s integrated set of storage management software, hardware, and
virtualization solutions.
Tony writes the “Inside System Storage” blog, which is read by thousands of clients, IBM sales reps and
IBM Business Partners every week. This blog was rated one of the top 10 blogs for the IT storage
industry by “Networking World” magazine, and #1 most read IBM blog on IBM’s developerWorks. The
blog has been published in series of books, Inside System Storage: Volume I through V.
Over the past years, Tony has worked in development, marketing and consulting for various storage
hardware and software products. Tony has a Bachelor of Science degree in Software Engineering, and a
Master of Science degree in Electrical Engineering, both from the University of Arizona. Tony is an
inventor or co-inventor of 19 patents in the field of electronic data storage.
9000 S. Rita Road
Bldg 9032 Floor 1
Tucson, AZ 85744
+1 520-799-4309 (Office)
tpearson@us.ibm.com
Tony Pearson
Master Inventor
Senior IT Architect
IBM Storage

Special Thanks for the following contributors to this presentation
Christopher Vollmar – Storage Architect
• cvollmar@ca.ibm.com
Carl Reasoner – Washington System Centre
• creason@us.ibm.com

Additional Resources from Tony Pearson
Email:
tpearson@us.ibm.com
Twitter:
twitter.com/az990tony
Blog:
ibm.co/Pearson
Books:
www.lulu.com/spotlight/990_tony
IBM Expert Network on Slideshare:
www.slideshare.net/az990tony
Facebook:
www.facebook.com/tony.pearson.16121
LinkedIn:
https://www.linkedin.com/in/az990tony

This presentation uses the IBM Plex™ font
IBM Plex™ is our new typeface. It’s global, it’s versatile and it’s
distinctly IBM.
IBM Plex
Sans
The IBM company is freeing itself from the cold, modernist cliché
and replacing Helvetica with a new corporate typeface. Also
replaces Arial, Calibri, Lucida Grande, Trebuchet, etc.
IBM Plex
Mono
A little something for developers. Replaces
Courier New, Letter Gothic, Lucida Console, etc.
IBM Plex
Serif
A hybrid of the third kind (combining the best of Plex, Bodoni,
and Janson into a contemporary serif). Replaces Cambria,
Garamond, Lucida Bright, Times New Roman, etc.
IBM Plex is freely available as TrueType and OpenType at: https://github.com/IBM/plex/releases

Notices and disclaimers
© 2019 International Business Machines Corporation. No part of this
document may be reproduced or transmitted in any form without
written permission from IBM.
U.S. Government Users Restricted Rights — use, duplication or
disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to
products that have not yet been announced by IBM) has been reviewed
for accuracy as of the date of initial publication and could include
unintentional technical or typographical errors. IBM shall have no
responsibility to update this information. This document is distributed
“as is” without any warranty, either express or implied. In no event,
shall IBM be liable for any damage arising from the use of this
information, including but not limited to, loss of data, business
interruption, loss of profit or loss of opportunity. IBM products and
services are warranted per the terms and conditions of the agreements
under which they are provided.
IBM products are manufactured from new parts or new and used parts.
In some cases, a product may not be new and may have been previously
installed. Regardless, our warranty terms apply.”
Any statements regarding IBM's future direction, intent or product
plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a
controlled, isolated environments. Customer examples are presented as
illustrations of how those
customers have used IBM products and the results they may have
achieved. Actual performance, cost, savings or other results in other
operating environments may vary.
References in this document to IBM products, programs, or services does
not imply that IBM intends to make such products, programs or services
available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared
by independent session speakers, and do not necessarily reflect the
views of IBM. All materials and discussions are provided for informational
purposes only, and are neither intended to, nor shall constitute legal or
other guidance or advice to any individual participant or their specific
situation.
It is the customer’s responsibility to insure its own compliance with legal
requirements and to obtain advice of competent legal counsel as to
the identification and interpretation of any relevant laws and regulatory
requirements that may affect the customer’s business and any actions
the customer may need to take to comply with such laws. IBM does not
provide legal advice or represent or warrant that its services or products
will ensure that the customer follows any law.

Notices and disclaimers, continued
Information concerning non-IBM products was obtained from the
suppliers of those products, their published announcements or other
publicly available sources. IBM has not tested those products about this
publication and cannot confirm the accuracy of performance,
compatibility or any other claims related to non-IBM products. Questions
on the capabilities of non-IBM products should be addressed to the
suppliers of those products. IBM does not warrant the quality of any
third-party products, or the ability of any such third-party products to
interoperate with IBM’s products. IBM expressly disclaims all
warranties, expressed or implied, including but not limited to, the
implied warranties of merchantability and fitness for a purpose.
The provision of the information contained herein is not intended to, and
does not, grant any right or license under any IBM patents, copyrights,
trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com and [names of other referenced IBM
products and services used in the presentation] are trademarks of
International Business Machines Corporation, registered in many
jurisdictions worldwide. Other product and service names might
be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark
information" at: www.ibm.com/legal/copytrade.shtml.
.

S106285 spectrum-archive-taming-big data-istanbul-v1902a

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à S106285 spectrum-archive-taming-big data-istanbul-v1902a

Similaire à S106285 spectrum-archive-taming-big data-istanbul-v1902a (20)

Plus de Tony Pearson

Plus de Tony Pearson (20)

Dernier

Dernier (20)

S106285 spectrum-archive-taming-big data-istanbul-v1902a