SlideShare une entreprise Scribd logo
1  sur  21
Building Linked Data
Platform with AWS...
...without actually building one
Eugene Morozov
Twitter: @eugenemorozov
Blog: http://emorozov.com
LinkedIn: https://www.linkedin.com/in/emorozov
If you really want to hear about it...
● Dev background
● Financial services
● Some background in building APIs - integration, reporting, etc.
Project experience
● Creating a reporting API that can be used by clients
● Getting away from the assumption that we must actually build something
● Solution to reuse AWS S3 with a thin veneer of custom authentication
Keeping things simple
● One doesn’t need complex infrastructure to host linked data
Exposing data sets with SemanticWeb standards
● Enter Linked Data Platform (https://www.w3.org/TR/ldp)
● Linked Data Platform (LDP) defines a set of rules for HTTP operations on
web resources, some based on RDF, to provide an architecture for read-write
Linked Data on the web.
LDP 101
● Resources
● Containers (Basic, Direct, Indirect)
● Content Types
● Headers
○ Mandates use of Link to specify
resources and containers
○ Use of Accept-Post, Accept-Patch
● REST verbs
○ All of the standard verbs, with
prescribed behaviour for containers
LDP 101
● Use headers to advertise what operations are available on the containers
○ Mandates use of Link to specify LDP resources and containers
○ Use of Accept-Post, Accept-Patch
● Suggests use of well-known terms DC, RDFS
Container Example
Do we really need to deploy stuff ourselves?
● Enter AWS (https://aws.amazon.com)
AWS 101
● Landscape
● https://aws.amazon.com/architecture/
● https://aws.amazon.com/products/
AWS 101 - S3
● Fairly rich API for storage via S3
● But that’s not quite enough to conform to LDP because of how headers are
treated
AWS 101 - API Gateway
● APIGateway is a way to define APIs in Swagger (with some custom
extensions) and deploy it
● Can be used to proxy other AWS services
AWS 101 - Route53
● DNS
AWS 101 - CloudFront CDN
● Use of CDN to proxy requests through to API -> S3
● More control over routing
● Caching
● Lower cost per API request if caching on top of the API
AWS 101 - Cognito
● Rich APIs for access control
● Don’t need to expose IAM roles to the outside world
AWS 101 - CloudFormation
● CloudFormation for provisioning
● Declarative deployment
● Can complete LDP-in-a-box
AWS 101 - LDP in a box
Demo
● Python, AWS
● Picking first two things off the list
○ Flatten content and push it into S3
○ Define subset of the DLP in Swagger
Future work
● Would be great to add Route53, Cognito and CDN
● Would be good to have fully conformant LDP definition in Swagger (doable)
● Perhaps going for a templated container in APIGateway, so we don’t need to
explicitly flatten the containers
● Add billing into LDP API, perhaps via S3 caller pays feature or log analysis
● In-a-box approach with CloudFormation
Some references and inspiration
● MarkLogic case study https://d0.awsstatic.com/whitepapers/marklogic-on-
aws.pdf
● RDF Data Management in Amazon Cloud
http://dl.acm.org/citation.cfm?id=2320790
● Automatic mapping of web APIs http://datalegend.net/assets/paper7.pdf
GitHub
● https://github.com/cadmiumkitty/aws-as-ldp

Contenu connexe

Tendances

Tendances (20)

From AWS to GCP, TABLEAPP Architecture Story
From AWS to GCP, TABLEAPP Architecture StoryFrom AWS to GCP, TABLEAPP Architecture Story
From AWS to GCP, TABLEAPP Architecture Story
 
Config management for_kubernetes: GitOps + Helm (CfgMgmtCamp 2020)
Config management for_kubernetes: GitOps + Helm (CfgMgmtCamp 2020)Config management for_kubernetes: GitOps + Helm (CfgMgmtCamp 2020)
Config management for_kubernetes: GitOps + Helm (CfgMgmtCamp 2020)
 
gRPC Overview
gRPC OverviewgRPC Overview
gRPC Overview
 
Introduction to DevOps and the Practical Use Cases at Credit OK
Introduction to DevOps and the Practical Use Cases at Credit OKIntroduction to DevOps and the Practical Use Cases at Credit OK
Introduction to DevOps and the Practical Use Cases at Credit OK
 
Kubeflow Pipelines (with Tekton)
Kubeflow Pipelines (with Tekton)Kubeflow Pipelines (with Tekton)
Kubeflow Pipelines (with Tekton)
 
CNCF opa
CNCF opaCNCF opa
CNCF opa
 
KFServing and Feast
KFServing and FeastKFServing and Feast
KFServing and Feast
 
CodiLime Tech Talk - Michał Pawluk: Our production deployment in AWS (HashiCo...
CodiLime Tech Talk - Michał Pawluk: Our production deployment in AWS (HashiCo...CodiLime Tech Talk - Michał Pawluk: Our production deployment in AWS (HashiCo...
CodiLime Tech Talk - Michał Pawluk: Our production deployment in AWS (HashiCo...
 
Docker for mac & local developer environment optimization
Docker for mac & local developer environment optimizationDocker for mac & local developer environment optimization
Docker for mac & local developer environment optimization
 
Configuration Management for the Cloud Native world with GitOps and Helm - To...
Configuration Management for the Cloud Native world with GitOps and Helm - To...Configuration Management for the Cloud Native world with GitOps and Helm - To...
Configuration Management for the Cloud Native world with GitOps and Helm - To...
 
[GID Live] Open-Source Cloud-Native Programming Language
[GID Live] Open-Source Cloud-Native Programming Language[GID Live] Open-Source Cloud-Native Programming Language
[GID Live] Open-Source Cloud-Native Programming Language
 
AWS Meetup Paris - Short URL project by Pernod Ricard
AWS Meetup Paris - Short URL project by Pernod RicardAWS Meetup Paris - Short URL project by Pernod Ricard
AWS Meetup Paris - Short URL project by Pernod Ricard
 
Logging in The World of DevOps
Logging in The World of DevOps Logging in The World of DevOps
Logging in The World of DevOps
 
[WSO2Con USA 2018] Deploying Applications in K8S and Docker
[WSO2Con USA 2018] Deploying Applications in K8S and Docker[WSO2Con USA 2018] Deploying Applications in K8S and Docker
[WSO2Con USA 2018] Deploying Applications in K8S and Docker
 
Journey to the cloud
Journey to the cloudJourney to the cloud
Journey to the cloud
 
KFServing and Kubeflow Pipelines
KFServing and Kubeflow PipelinesKFServing and Kubeflow Pipelines
KFServing and Kubeflow Pipelines
 
Prometheus loves Grafana
Prometheus loves GrafanaPrometheus loves Grafana
Prometheus loves Grafana
 
gRPC
gRPCgRPC
gRPC
 
Running Flink in Production: The good, The bad and The in Between - Lakshmi ...
Running Flink in Production:  The good, The bad and The in Between - Lakshmi ...Running Flink in Production:  The good, The bad and The in Between - Lakshmi ...
Running Flink in Production: The good, The bad and The in Between - Lakshmi ...
 
The Next Generation Software Stack: Meteor
The Next Generation Software Stack: MeteorThe Next Generation Software Stack: Meteor
The Next Generation Software Stack: Meteor
 

Similaire à Building Linked Data Platform with AWS

AWS re:Invent 2020 Serverless Recap
AWS re:Invent 2020 Serverless RecapAWS re:Invent 2020 Serverless Recap
AWS re:Invent 2020 Serverless Recap
Daniel Zivkovic
 

Similaire à Building Linked Data Platform with AWS (20)

Running R on AWS Lambda by Ana-Maria Niculescu
Running R on AWS Lambda by Ana-Maria NiculescuRunning R on AWS Lambda by Ana-Maria Niculescu
Running R on AWS Lambda by Ana-Maria Niculescu
 
Building self service framework
Building self service frameworkBuilding self service framework
Building self service framework
 
Architecting applications on amazon web services with node.js
Architecting applications on amazon web services with node.jsArchitecting applications on amazon web services with node.js
Architecting applications on amazon web services with node.js
 
Back-end (Flask_AWS)
Back-end (Flask_AWS)Back-end (Flask_AWS)
Back-end (Flask_AWS)
 
Write less (code) and build more with serverless
Write less (code) and build more with serverlessWrite less (code) and build more with serverless
Write less (code) and build more with serverless
 
Ghost Environment
Ghost EnvironmentGhost Environment
Ghost Environment
 
Aws serverless architecture
Aws serverless architectureAws serverless architecture
Aws serverless architecture
 
Deep dive into cloud security - Jaimin Gohel & Virendra Rathore
Deep dive into cloud security - Jaimin Gohel & Virendra RathoreDeep dive into cloud security - Jaimin Gohel & Virendra Rathore
Deep dive into cloud security - Jaimin Gohel & Virendra Rathore
 
PyConIT 2018 Writing and deploying serverless python applications
PyConIT 2018 Writing and deploying serverless python applicationsPyConIT 2018 Writing and deploying serverless python applications
PyConIT 2018 Writing and deploying serverless python applications
 
introduction to micro services
introduction to micro servicesintroduction to micro services
introduction to micro services
 
Skillenza Build with Serverless Challenge - Advanced Serverless Concepts
Skillenza Build with Serverless Challenge -  Advanced Serverless ConceptsSkillenza Build with Serverless Challenge -  Advanced Serverless Concepts
Skillenza Build with Serverless Challenge - Advanced Serverless Concepts
 
Serverless Data Platform
Serverless Data PlatformServerless Data Platform
Serverless Data Platform
 
AWS vs Azure vs Google (GCP) - Slides
AWS vs Azure vs Google (GCP) - SlidesAWS vs Azure vs Google (GCP) - Slides
AWS vs Azure vs Google (GCP) - Slides
 
Demistifying serverless on aws
Demistifying serverless on awsDemistifying serverless on aws
Demistifying serverless on aws
 
Low Cost AWS Services For Application Development in the Cloud
Low Cost AWS Services For Application Development in the CloudLow Cost AWS Services For Application Development in the Cloud
Low Cost AWS Services For Application Development in the Cloud
 
AWS re:Invent 2020 Serverless Recap
AWS re:Invent 2020 Serverless RecapAWS re:Invent 2020 Serverless Recap
AWS re:Invent 2020 Serverless Recap
 
Amazon web services (aws) main developer services
Amazon web services (aws)   main developer servicesAmazon web services (aws)   main developer services
Amazon web services (aws) main developer services
 
PyConIE 2017 Writing and deploying serverless python applications
PyConIE 2017 Writing and deploying serverless python applicationsPyConIE 2017 Writing and deploying serverless python applications
PyConIE 2017 Writing and deploying serverless python applications
 
Funky serverless features at aws
Funky serverless features at awsFunky serverless features at aws
Funky serverless features at aws
 
An introduction to cloud systems architecture
An introduction to cloud systems architectureAn introduction to cloud systems architecture
An introduction to cloud systems architecture
 

Plus de EugeneMorozov

Plus de EugeneMorozov (8)

Data Provenance and PROV Ontology
Data Provenance and PROV OntologyData Provenance and PROV Ontology
Data Provenance and PROV Ontology
 
Discoverability of Regulatory Rulebooks
Discoverability of Regulatory RulebooksDiscoverability of Regulatory Rulebooks
Discoverability of Regulatory Rulebooks
 
FIBO and SFTR - reflecting on FIBO workshop at DAS 2018 and applying FIBO to ...
FIBO and SFTR - reflecting on FIBO workshop at DAS 2018 and applying FIBO to ...FIBO and SFTR - reflecting on FIBO workshop at DAS 2018 and applying FIBO to ...
FIBO and SFTR - reflecting on FIBO workshop at DAS 2018 and applying FIBO to ...
 
Building Ontologies from Concept Maps
Building Ontologies from Concept MapsBuilding Ontologies from Concept Maps
Building Ontologies from Concept Maps
 
Streaming Linked Data to Web UI
Streaming Linked Data to Web UIStreaming Linked Data to Web UI
Streaming Linked Data to Web UI
 
Semantic Web and Micro Services
Semantic Web and Micro ServicesSemantic Web and Micro Services
Semantic Web and Micro Services
 
Documenting Enterprise Architectures Using Ontologies
Documenting Enterprise Architectures Using OntologiesDocumenting Enterprise Architectures Using Ontologies
Documenting Enterprise Architectures Using Ontologies
 
Working with RDF in Jupyter Notebooks: some lessons in getting rid of Excel f...
Working with RDF in Jupyter Notebooks: some lessons in getting rid of Excel f...Working with RDF in Jupyter Notebooks: some lessons in getting rid of Excel f...
Working with RDF in Jupyter Notebooks: some lessons in getting rid of Excel f...
 

Dernier

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
 

Dernier (20)

MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
Cyberprint. Dark Pink Apt Group [EN].pdf
Cyberprint. Dark Pink Apt Group [EN].pdfCyberprint. Dark Pink Apt Group [EN].pdf
Cyberprint. Dark Pink Apt Group [EN].pdf
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUKSpring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
 

Building Linked Data Platform with AWS

  • 1. Building Linked Data Platform with AWS... ...without actually building one Eugene Morozov Twitter: @eugenemorozov Blog: http://emorozov.com LinkedIn: https://www.linkedin.com/in/emorozov
  • 2. If you really want to hear about it... ● Dev background ● Financial services ● Some background in building APIs - integration, reporting, etc.
  • 3. Project experience ● Creating a reporting API that can be used by clients ● Getting away from the assumption that we must actually build something ● Solution to reuse AWS S3 with a thin veneer of custom authentication
  • 4. Keeping things simple ● One doesn’t need complex infrastructure to host linked data
  • 5. Exposing data sets with SemanticWeb standards ● Enter Linked Data Platform (https://www.w3.org/TR/ldp) ● Linked Data Platform (LDP) defines a set of rules for HTTP operations on web resources, some based on RDF, to provide an architecture for read-write Linked Data on the web.
  • 6. LDP 101 ● Resources ● Containers (Basic, Direct, Indirect) ● Content Types ● Headers ○ Mandates use of Link to specify resources and containers ○ Use of Accept-Post, Accept-Patch ● REST verbs ○ All of the standard verbs, with prescribed behaviour for containers
  • 7. LDP 101 ● Use headers to advertise what operations are available on the containers ○ Mandates use of Link to specify LDP resources and containers ○ Use of Accept-Post, Accept-Patch ● Suggests use of well-known terms DC, RDFS
  • 9. Do we really need to deploy stuff ourselves? ● Enter AWS (https://aws.amazon.com)
  • 10. AWS 101 ● Landscape ● https://aws.amazon.com/architecture/ ● https://aws.amazon.com/products/
  • 11. AWS 101 - S3 ● Fairly rich API for storage via S3 ● But that’s not quite enough to conform to LDP because of how headers are treated
  • 12. AWS 101 - API Gateway ● APIGateway is a way to define APIs in Swagger (with some custom extensions) and deploy it ● Can be used to proxy other AWS services
  • 13. AWS 101 - Route53 ● DNS
  • 14. AWS 101 - CloudFront CDN ● Use of CDN to proxy requests through to API -> S3 ● More control over routing ● Caching ● Lower cost per API request if caching on top of the API
  • 15. AWS 101 - Cognito ● Rich APIs for access control ● Don’t need to expose IAM roles to the outside world
  • 16. AWS 101 - CloudFormation ● CloudFormation for provisioning ● Declarative deployment ● Can complete LDP-in-a-box
  • 17. AWS 101 - LDP in a box
  • 18. Demo ● Python, AWS ● Picking first two things off the list ○ Flatten content and push it into S3 ○ Define subset of the DLP in Swagger
  • 19. Future work ● Would be great to add Route53, Cognito and CDN ● Would be good to have fully conformant LDP definition in Swagger (doable) ● Perhaps going for a templated container in APIGateway, so we don’t need to explicitly flatten the containers ● Add billing into LDP API, perhaps via S3 caller pays feature or log analysis ● In-a-box approach with CloudFormation
  • 20. Some references and inspiration ● MarkLogic case study https://d0.awsstatic.com/whitepapers/marklogic-on- aws.pdf ● RDF Data Management in Amazon Cloud http://dl.acm.org/citation.cfm?id=2320790 ● Automatic mapping of web APIs http://datalegend.net/assets/paper7.pdf

Notes de l'éditeur

  1. Hello, welcome to the third Semantic Web London Meetup My name is Eugene Morozov, I co-manage engineering practice at Lab49 We are a strategy, design and technology consulting firm and we specialise in capital markets Fine print - whatever is on the screen is my own work, not a work done for our clients
  2. Other TODO: Look into https://github.com/awslabs/ecs-refarch-cloudformation Serverless architectures (AWS APIGateway + Lambdas) https://github.com/awslabs/aws-big-data-blog/tree/master/aws-blog-titan-graph-database