API S3 & Object Storage
•Download as PPTX, PDF•
0 likes•288 views
Découvrez comment l'API compatible S3 permet une migration transparente vers un fournisseur de cloud européen et facilite votre stratégie de cloud hybride. Face à une augmentation exponentielle du volume de données non structurées (vidéos, photos, pages web, fichiers audio, etc.), le stockage d'objets a été massivement utilisé. Aujourd’hui, il détient une part importante des informations sur Internet. La plupart des outils de développement possèdent donc un plugin compatible avec les services de stockage d'objets, le de facto standard de marché restant cependant l'API S3. OVHcloud a ajouté une API compatible S3 à son service de stockage d'objets. Cela permet aux développeurs d'utiliser les outils qu'ils connaissent déjà, afin de tirer parti des solutions OVHcloud sans modifier le code de leur application. Ce webinaire permettra aux responsables d’infrastructures, devops et administrateurs systèmes de comprendre comment l'API compatible S3 d’OVHcloud permet la migration de données vers un fournisseur de cloud européen et facilite leur stratégie de cloud hybride. Après avoir pris connaissance des concepts clefs du stockage d'objets et de ses cas d'usage, les participants verront à travers une démonstration comment une application utilisant l'API S3 peut désormais exploiter les solutions OVHcloud. Et ce, sans rien changer d'autre qu'un point de terminaison et des informations d'identification.
Recommended
Recommended
More Related Content
More from OVHcloud
More from OVHcloud (20)
Recently uploaded
Recently uploaded (20)
API S3 & Object Storage
- 1. Public Cloud – Object Storage S3 compatible API Jean-Daniel Bonnetot / Andry Ramiandrasoa 2020
- 2. 2
- 3. 3
- 4. 4 S3 compatible API opens up opportunities Migrate to divide by 3 your storage bill Reduce risk and avoid vendor lock-in Keep your code, seamless change
- 5. Do you have that kind of use case ? - Logs archiving - Backups - Large business data repository - Asynchrone data transfert system (wetransfert like) - Document Management System (pdf, doc,…) - Media Library (images, video, sounds…) - Storing static web content (html, css, jpg, …) - Any static data - …
- 6. Object Storage adresses replication, scaling and indexing issues ?
- 7. File System changeable poor / no hard thousands tree structure Type of data Meta data / Routines Scaling Number of items Address Object Storage static rich / yes progressive hundreds of billions OID
- 9. 9 Public or private Static web content Large objects Version control Container synchronisation CORS Temporary URL Cheksum Object expiration dates Redirections via DNS ACL management S3 credential generation
- 10. How to interact with your Object Storage container S3 Application Programming Interface (S3 API)
- 11. Demo 5 min 1. Setup awscli 2. Upload a file 3. Add credentials in my PHP app
- 12. OVHcloud Case Study Object Cloud Storage SDK Connector SDKs take the complexity out of coding by providing language-specific APIs
- 13. OVHcloud Case Study Object Cloud Storage Off-the-shelf Application - Wordpress
- 14. OVHcloud Case Study Object Cloud Storage Shelved Application - Wordpress
- 15. OVHcloud Case Study Object Cloud Storage Massive Storage of any unstructed data types Large dataset Big data injection per day « infinite » scale-out capabilities Wep/app/mobile access hundreds of concurrent user connections
- 16. OVHCLOUD OBJECT STORAGE IS MULTI- LOCAL Only few milisecond from your users Choice between several datacenter In compliance with local legislation
- 17. OVHCLOUD OBJECT STORAGE Example with 4TB Upload : 0€ (included) Monthly retention cost : 40€
- 18. OBJECT STORAGE IS ACCESSIBLE Competitoooor #1 Competitor #2 0 500 1000 1500 Storage Obj storage traffic Example of a high traffic mobile application with: 10TB of Object Storage 10TB of Object Storage traffic Focusing on Storage cost
- 19. OBJECT STORAGE IS ACCESSIBLE Competitoooor #1 Competitor #2 Example of a e-learning application with streamed content: 200GB of Object Storage 30TB of Object Storage traffic Focusing on Storage cost 0 500 1000 1500 2000 2500 3000 Storage Obj storage traffic
- 20. Questions?
- 21. Thank you.
Editor's Notes
- As the volume of unstructured data such as videos, photos, web pages, audio files etc… is exploding, object storage has been massively used and is holding a major share of the data on the internet. Most tools in the cloud market has a plugin to provide combability with object storage services, but the de-facto standard remains the S3 API. OVHcloud added S3 compatible API on top of his Object Storage service which enables developers to use the tools they already know to take advantage of OVHcloud service without modifying their application’s code. This webinar session will explain and help you understand how data migration to a data sovereignty focused cloud provider and hybrid cloud strategy is made easier with OVHcloud S3 compatible API. After learning about object storage key concepts and use cases, attendees will see through a demo how an application using S3 API can now leverage OVHcloud service without changing anything but an endpoint and credentials.
- OVHcloud already proposes an object storage service. We started working on OpenStack Swift in 2012 and quickly reached a large scale. 8 years of object storage, in production, billion of objects, 250 PB of storage space, this service is really resiliant.
- We are glad to add on top of our service, the S3 API, the standard on the market. It’s the same service, the same resiliancy, the same availability with the API you used to manage with the tools you already have.
- At high level, this compability make it easier to migrate from S3 to OVHcloud. We highlight here the associated benefits. Protect yourself from vendor lock-in Maintain your data sovereignty with our S3-compatible API. This way, you can host your data with a provider that respects reversibility. Control your spending Get the very best prices on the market for storage and traffic. You manage your own billing, so you won’t get any nasty surprises at the end of the month. Adapt your application effortlessly Update your configuration (the endpoint and credentials) while keeping the same application code, and the same tools.
- To indroduce the topic, let me just remind you where Object Storage can be used. Basically, Object Storage can be used in so many use cases, everywhere when you have a nework connectivity and static data. I’ll say it again, Object Storage could be used everywhere as soon as you have a network and static files like documents, images, etc. AWS S3 is the main/core/principal component of cloud services that AWS provides because every services has a link to S3 to store data. So I’m ready to bet each of you has at least one of those use case to manage, that’s why object storage is so central in many application but paradoxaly, it’s still not well known so we’ll have a 5 min reminder on what is object storage and what issue does it solve.
- Ok, so now let’s illustrate, with an old fashion approach, the issues solved by Object Storage systems. I’m a sysadmin and I manage a platform which needs a lot of storage. For example, an image or video application. Regarding the storage part of my architecture, I have 3 issues : 1st: I have to replicate my data. The storage devices can fail and I have to anticipate an monitor that. 2nd: The plus sign show that I’ll have situation where I have not enough space left and I need to add more storage. That implies CAPEX which won’t be cost-effective before a long periode of time. 3rd: The question mark illustrate that I have to maintain an index describing which data is where as I have multiple storage device. That’s exactly what Object Storage systems solve by removing any effort on the user side on replication, scaling and indexing.
- Here I wanted to share you a summary about differences between a file system (the storage system you use to manipulate when you navigate on your document on your laptop) and Object Storage That can help to identify when Object Storage can be used and when it’s not the right answer. In the middle, you have the questions you should address by priority order. But let me focus on 2 specific points. If the data I have to manage are changeable (like a database for example) that’s not possible to use OS. The scalability, if I know that I have to manage more and more data in the futur, classical file system will be painful to maintain. And from a customer point of view, it’s the perfect « pay as you grow » illustration. I’ll pay only the storage space I use, not the storage space I have to anticipate.
- Data This is pure data, for a file that’s the content inside the file. Network address That’s usually an HTTP address Most of the time it requires an authentication to access the data Metadata Many kind of information classical ones like: kind of content, size, last upload, acces right more intellignent ones like: signature MD5SUM, expiration date
- Here is some key functionalities we are providing with our object storage service Public or private By default, containers and their objects are only accessible after authentication. In order to make objects publicly available, you can open up access to them on the web without authentication. Version control By managing versions of activatable objects in a container (versioning), you can log changes to your objects and restore previous versions on demand. Container synchronisation Synchronise your containers, so your data is the same everywhere. This function can be useful for maintaining data in a consistent state across several regions, for example. Temporary URL You can use a temporary URL to provide read or write access to an object, for a given period of time. Once the time period has elapsed, the URL will no longer be valid. Object expiration dates Objects can have an expiration date as an attribute. In such cases, the object will simply be deleted once that date has passed. ACL management You can configure fine access permissions for objects and containers, by both user and access method. Static web content By configuring a container in static mode, OVH's Object Storage will act as a standard web server by first serving "index.html", or presenting an HTML index of the objects' URLs if no "index.html" objects exist by default. Large objects With Object Storage, you can store objects of up to several TB with no size limit. CORS Using this mechanism managed via code in web browsers, you can provide access to objects from other domain names, separating a website's code from its static objects. Cheksum The object metadata automatically incorporates an MD5 signature, in order to ensure the integrity of all transferred objects. Redirections via DNS You can redirect to a container in OVH Object Storage by configuring a special entry in your domain name’s DNS zone. S3 credential generation To configure a tool on the S3 API, you can generate credentials that are specific to this use. Simply change the endpoints in the application by using these credentials to benefit from this compatibility.
- Before jumping into the demo, we wanted to explain from a more conceptual view point how to interact with your Object Storage using S3 API Most common way in which data is stored, managed, and retrieved by object stores. S3 API is a frontend API on top of the OpenStack Swift one. To use the S3 API at OVHcloud, you need to get S3 credentials form Keystone (1) which is the authentication module in OpenStack. This will provide you with an Access Key ID and a Secret Key that you can use in your S3 tool (2). Once you have these credentials you will be able to communicate with OVHcloud using the “language” of S3 and use our Object Storage solutions. S3 API will verify your credentials (4) and translate your calls into Swift API (5) in order to execute your requests (6).
- First we’ll install the awscli Then using the openstack cli, we’ll generate aws credentials in order to setup our awscli using the proper endpoints. Then we’ll add those credentials and endpoints in our PHP app. This PHP snipet use the official AWS library. It just connects to the s3 API, get an object and display it or return an error. Finally e’ll upload a file (jacques.jpg) in our bucket (demo) and check the result in our browser This is a really quick demo to show you there is nothing to change except the credential and the endpoint to move from AWS S3 to OVHcloud
- Instead of listing some specific use cases, I wanted to start by saying that as we already mentionned it, tones of use cases can be addressed with object storage service. As a consequence, many libraries exists in many laguages. In the demo I used the official AWS S3 SDK but you have many others like JavaScript, Python, .net, Ruby, Java, Go, nodeJS, C++ etc… Those libraries helps you to use the S3 API in your code without managing the complexity directly in your application. Doing that, your application can access an ulimited storage space and smart storage system.
- Here is a concret example that shows how to integrate S3 usage on wordpress to store the media on OVHcloud Object storage. I used a plugin named Media Cloud that provides a S3 compatible driver in wordpress that can take a specific API endpoint. This is an important point of attention you need to take care when you’ll chose a tool. Check if that tool can configure a specific endpoint. If it’s not the case and the AWS endpoints are hardcoded in the software, you won’t be able to manage any other provider. Once you configured the plugin properly, every images, video or other documents will be pushed in Object Storage and linked to your blog articles.
- If we zoom in on the right part, we can see the network address for each image is something related to the OVHcloud Object Storage service. Here, some object storage features are used : Public acces, CORS and of course the S3 API. Based on that, you are free to store tones of objects, terabytes of video, etc… there is no limite on the storage part. Another side effect is that you’ll reduce the load on your primary server because many request and a lot of traffic will be redirected to the object storage service. I wrote a blog post with more details on that use case. It’s available on OVHcloud blog, feel free to have a look if you are interested.
- And finally, I wanted to illustrate the usage of Object Storage with a real customer use case, one big OVH customer: the European Spacial Agency ESA has a project called Copernicus. They use our Object Storage service to store tones of data coming from the satelites we have above our heads. Each satelite takes photos from the space and it’s send to our Object Storage and those data are available for organisations that will deliver essential public services. Each month, ESA send us multiple Petabytes and their applications are use by hundred of users. That means each month we add more than a tousand disks into our cluster just for them. You probably don’t have such a big use case but it gives you an idea of the capacities this service has and the resiliancy we provide on the data.
- In that part, we will explain what advantages we have with our Object Storage offer. First, our object storage offer is available in all our Public Cloud location You can move your data really close to your user And benefit from local legislation depending on your constraint. Think about GDPR or Patriot Act for example. Those locations are: France Canada Germany Unighted Kingdom Poland Singapore Sydney USA
- Our offer is also designed to have a simple billing model. If you are familiar with AWS bills, you probably know what I mean. You have to figure out: how frequently you will access your data (tier) to determine the price of storage transfert price are varying depending on location and which internal service you forward to charge on API call, depending on the type of request will impact price (PUT is more expensive than GET request for instance) Management of your object such as inventory, taggint to your object will impact the bill Our model is by-design simple so you can understand easily how it impact your bill. There is a price for the stoarage, 0.01 euros A price for the outgoing bandwidth 0.01 euros too And that’s it. Upload is free -- Back up data for international pricing OVHcloud: https://www.ovhcloud.com/en/public-cloud/prices/ Incoming traffic Included Outgoing traffic $0.011 /GB Storage replicated x3 $0.0112 /month/GBa AWS S3 standard : https://aws.amazon.com/s3/pricing/ First 50 TB / Month$0.0245 per GB S3 Standard : PUT, COPY, POST, LIST requests (per 1,000 requests) $0.0054 Outgoing to Internet: Next 9.999 TB / Month $0.09 per GB Greater than 150 TB / Month $0.05 per GB Outgoing to inter region (within AWS Cloud) Europe (Ireland) $0.02 per GB Cost per replication: GCP: https://cloud.google.com/storage/pricing Standard Storage (per GB per Month) : $0.026 Egress to WW: 0-1 TB: $0.12 10+ TB: $0.08 Egress to Asia Destinations : 0-1 TB: $0.12 10+ TB: $0.08 Egress to China: 0-1 TB: $0.23 10+ TB: $0.20 Egress to Australia: 0-1 TB: $0.19 10+ TB: $0.15
- So to illustrate that and make it more concreate, here is a use case, a real one for a mobile application with many servers managed by a kubernetes service. We’ll focus only on the object storage part considering the costs around storage. We have 10TB of data stored and every month, wa have approximatly 10TB of data transferred. We made the maths and here is how much it cost on some competitore you will recognize The blue part is the price for the storage, the green one is the traffic. As you can see, the total cost at OVH is equivalent to only the storage on competitors and the traffic will cost you a loooooot.
- An other exemple/ use case, smaller, this is an e-learning platforme with some videos watched by a lot of users each month (5k users). We have 200GB of storage but much more traffic, 30TB. Considéring only the storage cost again, you can see this time we don’t see the data storage part, it’s too small on the graph for all provider But again when you look at the traffic part, the difference is huge ! Now you have a concrete illustration of the benefit to migrate to OVHcloud for your data.