Backup solution based on named pipes to storreduce scheduled by Cronicle.

1. Backup multi-cloud solution
based on named Pipes.
Leandro Totino Pereira
System Engineer
Cronicle | Storreduce | Named pipes | S3

2. Agenda
• What is cronicle ?
• What is storreduce ?
• Backup Architecture
• Solution architecture

3. Cronicle
It is a multi-server task scheduler and runner, with a web based
front-end UI. It handles both scheduled, repeating and on-
demand jobs, targeting any number of slave servers, with real-
time stats and live log viewer. It's basically a fancy Cron
replacement written in Node.js. You can give it simple shell
commands, or write Plugins in virtually any language.

4. Cronicle advantages
• Couchbase database
• High availbility included
• Basics metrics
• Stats
• API
• Easy configuration

5. Storreduce
StorReduce is a specialized cloud object storage deduplication
solution, designed to meet the unique requirements of
companies using cloud storage for large volumes of data.
StorReduce sits between your applications and S3,
transparently deduplicating data inline. StorReduce's internal
testing reports speeds of up to 900 MB/s, with lower storage
and bandwidth transmission costs, speeding up the transfer of
data to the cloud, and between clouds, by as much as 30 times.

6. Storreduce Advantanges
• Up to 99% deduplication: Reduces data transmission and cloud storage volumes between 80 and
99%. This level of deduplication can help you reduce costs.
• Fast sustained throughput: StorReduce's internal testing reports write speeds up to 900
megabytes per second, adding under 50ms of latency with sustained throughput 24/7. StoreReduce
does not buffer data.
• Scalable: StoreReduce's testing has shown the ability to store up to 40 petabytes (40,000,000
gigabytes) of data per Google Compute Engine instance on Cloud Platform. Deploying multiple
servers can help you reach nearly any capacity you need.
• Cloud-native: Deduplicated data is immediately accessible to cloud services through StorReduce’s
S3 REST API.
• Multi-cloud: Data can be stored in Google Cloud Storage, including Multi-Regional, Regional,
Nearline, or Coldline, and also including Amazon S3 or S3IA, Microsoft Azure Blob Storage, and
other S3-compatible private cloud object stores such as Cloudian and IBM Cleversafe.
• Resilient: Index information required to re-hydrate the data is stored in multiple locations, both in
Google Cloud Storage and on each StorReduce server. Extensive hashing and checking ensures
data integrity during deduplication and rehydration.
• Software-only solution: No hardware is required. StorReduce is free from hardware costs and
lock-in to particular hardware.
Source: https://cloud.google.com/solutions/partners/storreduce-cloud-deduplication

7. Storreduce key features
• Read replicas: Additional StorReduce servers can be deployed in the cloud or on-premises as read
replicas. Uploaded data is only stored once but is immediately available in multiple locations,
enabling migrated backup workloads to be re-purposed in the cloud for development, test, QA,
disaster recovery and to be used by cloud-based services.
• High availability: Several StorReduce servers can be combined into a high-availability cluster to
provide automatic failover in the event of a server failure.
• Data replication: StorReduce can replicate data between regions, between cloud vendors, or
between public and private clouds, providing increased data resilience. Only the unique data is
transferred.
• Backup software integration and data management: Works with existing data management or
backup software that is compatible with Google Cloud Storage, including Veritas NetBackup and
CommVault.
• Data encryption: Supports client-side data encryption using KMS for key management. Data can
be encrypted on-premises before being sent to the cloud, or cloud encryption-at-rest services can
be used. Data is always encrypted in-transit.
• Secure user account and key management: Users or servers can be given individual user
accounts within StorReduce, allowing data access to be restricted. Multiple access keys can be
created and managed as needed for each user account.
• Write speed throttling: A maximum write speed can be set to prevent StorReduce using too much
bandwidth when sharing an Internet connection with other infrastructure.
Source: https://cloud.google.com/solutions/partners/storreduce-cloud-deduplication

8. The 30000-Foot View

9. The 30,000-Foot view workflow

10. SQLite architecture (server-agent
database)
After retention time setup the script backup create a
new database and mv the old one to archive directory

11. IPC – Named Pipes

12. Server and agents on Github
• https://github.com/tatototino/backup-mem
• backup.py – script which create diferencial list into sqlite and backup the
files
• client.py – Connect to specified server and send a signal to create backup
• server.py – Open port to receive the signal from client to create backup
• mysql.py – Create backup mysql databases

13. Servers – group configuration

14. Plugin configuration

15. Job configuring

16. Dashboard

17. History and Stats

18. Using Cronicle API
API Examples:
get job status:
curl -H 'X-API-Key: e11fc8948079ec429f7a5a25c97ee21c'
http://xxxxxxxx:3012/api/app/get_job_status?id=jjl6owa28sm
get event:
curl -H 'X-API-Key: e11fc8948079ec429f7a5a25c97ee21c'
http://xxxxxxxxx:3012/api/app/get_job_status?id=jjl6owa28sm

19. Manually execution workflow

20. Backup Diff - workflow
1- Backup full – Create new
Sqlite database “full-date”
2 - List all the files from full
backup in sqlite
3 - Create/modify some files
4 – Showing differentials files since last full backup

21. Internal object Storage vs AWS replica
We can create file/delete using s3 browser
connected to Internal Storreduce (on-premisse)
We can´t create file/delete using s3 browser connected
to AWS Replica Storreduce (replica node)

22. Storreduce - Deduplication ratio

23. Thank you!
Questions?
More Information:
Linkedin:
https://www.linkedin.com/in/leandro-totino-pereira