SC1 - Hangout 2: The Open PHACTS pilot

BIG DATA EUROPE
H2020 CSA (2015-17)
SOCIETAL CHALLENGE “HEALTH”
Integrating Big Data, Software & Communities for Addressing
Europe’s Societal Challenges06.07.2016

BigDataEurope
6-Jul-16
Today:
•  Short overview of Big Data Europe Ronald Siebes
•  What is Open PHACTS Stian Soiland-Reyes, Bryn Williams-Jones
•  The Big Data Europe infrastructure Erika Pauwels, Aad Versteden
•  Pilot 1: The Open PHACTS docker Stian Soiland-Reyes
•  Q&A
Stian Soiland-Reyes
BioExcel and
University of Manchester
Ronald Siebes
VU Amsterdam
Erika Pauwels
Tenforce
Aad Versteden
Tenforce
Bryn Williams-Jones
Open PHACTS Foundation

6-Jul-16www.big-data-europe.eu

Q&A
6-Jul-16www.big-data-europe.eu

Open PHACTS
Architecture and
Docker install
Stian Soiland-Reyes, University of Manchester
http://orcid.org/0000-0001-9842-9718
@soilandreyes
This work is licensed under a
.Creative Commons Attribution 4.0 International License
Big Data Europe Webinar, 2016-07-06
This work has been done as part of the BioExcel CoE ( ),
a project funded by the EC H2020 program, contract number
www.bioexcel.eu
EINFRA-5-2015 675728
https://slides.com/soilandreyes/2016-07-06-openphacts
1

http://www.openphacts.org/
Bringing together pharmacological data resources
in an integrated, interoperable infrastructure
Data sources integrated and linked together
so that you can easily see the relationships
between compounds, targets, pathways,
diseases and tissues.
, , , ,
, , , ,
, ,
ChEBI ChEMBL ChemSpider ConceptWiki
DisGeNET DrugBank FAERS Gene Ontology
neXtProt SureChEMBL, UniProt WikiPathways
2 . 1

Data integration
https://www.openphacts.org/2/sci/data.html
2 . 2

https://dev.openphacts.org/docs/2.1
Re-exposed as
public API
2 . 3

{
"format": "linked-data-api",
"version": "1.5",
"result": {
"_about": "https://beta.openphacts.org/1.5/compound?uri=http%3A%2F%2Fwww.conceptwiki.org%2Fconce
"definition": "https://beta.openphacts.org/api-config",
"extendedMetadataVersion": "https://beta.openphacts.org/1.5/compound?uri=http%3A%2F%2Fwww.concep
"linkPredicate": "http://www.w3.org/2004/02/skos/core#exactMatch",
"activeLens": "Default",
"primaryTopic": {
"_about": "http://www.conceptwiki.org/concept/38932552-111f-4a4e-a46a-4ed1d7bdf9d5",
"inDataset": "http://www.conceptwiki.org",
"exactMatch": [
{
"_about": "http://bio2rdf.org/drugbank:DB00398",
"description_en": "Sorafenib (rINN), marketed as Nexavar by Bayer, is a drug approved for th
"description": "Sorafenib (rINN), marketed as Nexavar by Bayer, is a drug approved for the t
"drugType_en": [
"investigational",
"approved"
],
"drugType": [
"investigational",
"approved"
],
"genericName_en": "Sorafenib",
"genericName": "Sorafenib",
"metabolism_en": "Sorafenib is metabolized primarily in the liver, undergoing oxidative meta
"metabolism": "Sorafenib is metabolized primarily in the liver, undergoing oxidative metabol
"proteinBinding_en": "99.5% bound to plasma proteins.",
"proteinBinding": "99.5% bound to plasma proteins.",
"toxicity_en": "The highest dose of sorafenib studied clinically is 800 mg twice daily. The
"toxicity": "The highest dose of sorafenib studied clinically is 800 mg twice daily. The adv
"inDataset": "http://www.openphacts.org/bio2rdf/drugbank", 2 . 4

<?xml version="1.0" encoding="utf-8"?>
<result format="linked-data-api" version="1.5" href="https://beta.openphacts.org/1.5/compound?uri=
<primaryTopic href="http://www.conceptwiki.org/concept/38932552-111f-4a4e-a46a-4ed1d7bdf9d5">
<prefLabel xml:lang="en">Sorafenib</prefLabel>
<exactMatch>
<item href="http://rdf.ebi.ac.uk/resource/chembl/molecule/CHEMBL1336">
<type href="http://rdf.ebi.ac.uk/terms/chembl#SmallMolecule"/>
<inDataset href="http://www.ebi.ac.uk/chembl"/>
<mw_freebase datatype="double">464.82</mw_freebase>
</item>
<item href="http://ops.rsc.org/OPS379634">
<smiles>CNC(=O)C1=NC=CC(=C1)OC2=CC=C(C=C2)NC(=O)NC3=CC(=C(C=C3)Cl)C(F)(F)F</smiles>
<rtb datatype="double">5.0</rtb>
<ro5_violations datatype="double">1.0</ro5_violations>
<psa datatype="double">92.35</psa>
<molweight datatype="double">464.825</molweight>
<molformula>C21H16ClF3N4O3</molformula>
<logp datatype="double">5.158</logp>
<inchikey>MLDQJTXFUGDVEO-UHFFFAOYSA-N</inchikey>
<inchi>InChI=1S/C21H16ClF3N4O3/c1-26-19(30)18-11-15(8-9-27-18)32-14-5-2-12(3-6-14)28-20(31
<hbd datatype="double">3.0</hbd>
<hba datatype="double">7.0</hba>
<inDataset href="http://ops.rsc.org"/>
</item>
<item href="http://aers.data2semantics.org/resource/drug/NEXAVAR">
<prefLabel>NEXAVAR</prefLabel>
<reportedAdverseEvent>
<item href="http://aers.data2semantics.org/resource/diagnosis/HEAD_INJURY">
<prefLabel>HEAD INJURY</prefLabel>
<inDataset href="http://aers.data2semantics.org/"/>
</item>
<item href="http://aers.data2semantics.org/resource/diagnosis/SUPRAVENTRICULAR_TACHYCARD
<prefLabel>SUPRAVENTRICULAR TACHYCARDIA</prefLabel>
<inDataset href="http://aers.data2semantics.org/"/> 2 . 5

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix void: <http://rdfs.org/ns/void#> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix ns0: <http://www.openphacts.org/api#> .
@prefix ns1: <http://bio2rdf.org/> .
@prefix ns2: <http://rdf.ebi.ac.uk/terms/chembl#> .
@prefix chembl1336: <http://rdf.ebi.ac.uk/resource/chembl/molecule/CHEMBL1336#> .
@prefix linked-data: <http://purl.org/linked-data/api/vocab#> .
@prefix msg0: <http://www.openphacts.org/api/> .
<http://www.conceptwiki.org/concept/38932552-111f-4a4e-a46a-4ed1d7bdf9d5>
skos:exactMatch <http://aers.data2semantics.org/resource/drug/NEXAVAR> ;
skos:exactMatch <http://aers.data2semantics.org/resource/drug/SORAFENIB> ;
skos:exactMatch <http://www.conceptwiki.org/concept/38932552-111f-4a4e-a46a-4ed1d7b
skos:exactMatch <http://bio2rdf.org/drugbank:DB00398> ;
skos:exactMatch <http://rdf.ebi.ac.uk/resource/chembl/molecule/CHEMBL1336> ;
skos:exactMatch <http://ops.rsc.org/OPS379634> ;
skos:prefLabel "Sorafenib"@en ;
void:inDataset <http://www.conceptwiki.org> ;
foaf:isPrimaryTopicOf <https://beta.openphacts.org/1.5/compound?uri=http%3A%2F%2Fwww.conc
<https://beta.openphacts.org/1.5/compound?uri=http%3A%2F%2Fwww.conceptwiki.org%2Fconcept%2F3893255
foaf:primaryTopic <http://www.conceptwiki.org/concept/38932552-111f-4a4e-a46a-4ed1d7
linked-data:definition <https://beta.openphacts.org/api-config> ;
msg0:activeLens "Default" ;
void:linkPredicate skos:exactMatch ;
linked-data:extendedMetadataVersion <https://beta.openphacts.org/1.5/compound?uri=http%3A
<http://ops.rsc.org/OPS379634>
void:inDataset <http://ops.rsc.org> ;
ns0:smiles "CNC(=O)C1=NC=CC(=C1)OC2=CC=C(C=C2)NC(=O)NC3=CC(=C(C=C3)Cl)C(F)(F)F" ;
ns0:inchi "InChI=1S/C21H16ClF3N4O3/c1-26-19(30)18-11-15(8-9-27-18)32-14-5-2-12(3
ns0:inchikey "MLDQJTXFUGDVEO-UHFFFAOYSA-N" ; 2 . 6

Chemical Structure Search
RDF/SPARQL
(Virtuoso)
Identity Mapping Service
Identity Resolution Service
(ConceptWiki)
Chembl, Uniprot, ...
Data loading
4 . 3

SC1 Health Webinar
Technical overview6 July 2016

Platform goals
◎ Low total cost of ownership
◎ Simple to get started with Big Data
◎ Cater for widely varying use cases
◎ Embrace emerging Big Data technologies
◎ Simple integration with custom components

Big Data is
◎ Volume
o Quantity of data
◎ Velocity
o Speed at which data is provided
◎ Variety
o Different formats/models in which data is provided
◎ Veracity
o Accuracy/truthfulness of the data
Why did we need all this?

Semantic Big Data
ongoing research!
◎ Semantic Data Lake
o from data swamp to data lake
o query contents in the data lake
◎ SANSA stack
o Big Data analytics on semantic graph

Support layer
◎ Swarm UI
o Launch, install and manage pipelines
◎ Pipeline daemon & monitor
o Determine order in which steps are executed
o eg: Upload files before running computations
◎ Integrator UI
o Present dashboards in a unified interface

Platform installation
◎ Manual installation guide
◎ Using Docker Machine
o On local machine (VirtualBox)
o In the cloud (AWS, DigitalOcean, Azure)
o Bare metal

◎ High level picture
o docker-compose.yml describes pipeline topology
◎ Common components
o extend template image with your code
◎ New components
o build a Docker image for your component
o this is your own little Virtual Machine for your component
◎ Sharing
o publish topology as git repository
o publish new components on docker hub
Platform development

More monitoring
This topic is ongoing, many interesting options
◎ Visualise logs with Kibana?
◎ Combine logs for large overview?
◎ Monitor node load?
◎ Provide autoscheduling?

Concluding remarks
◎ Used in practice
◎ Easy to get started
◎ Improving as we speak

You can talk to us!
◎ Aad Versteden
aad.versteden@tenforce.com
◎ Erika Pauwels
erika.pauwels@tenforce.com

Linux Container technology
..light-weight "virtual" virtual machine
A container is started from a image
Images downloaded from Docker Hub
Dockerﬁle: Layer-based recipe
Philosophy: One service, one image → microservices
Cloud's best friend: scalable, reproducible, customizable
https://www.docker.com/
5 . 1

https://hub.docker.com/r/openphacts/
5 . 2

ops-ims
ops-mysql
ops-virtuoso
ops-apiops-memcached
ops-virtuosodata
ops-mysqldata
ops-virtuosostaging
ops-mysqlstaging
https://data.openphacts.org/
ops-explorer
:3001
:3002
:3004:3003
https://hub.docker.com/
ops-docker
https://github.com/openphacts/ops-docker/
5 . 3

Docker Compose
https://www.docker.com/products/docker-compose
Which images to download
Which data volumes to use
Which network ports are exposed
How are containers linked
How to start/stop the containers
$ docker-compose up -d
5 . 4

docker-compose.yml
# Open PHACTS platform
# Docker Compose configuration
explorer:
image: openphacts/explorer2
ports:
- "3001:3000"
links:
- api
environment:
- API_URL=http://localhost:3002
#restart: always
api:
image: openphacts/ops-linkeddataapi
ports:
- "3002:80"
links:
- ims
- memcached
- virtuoso:sparql
# SPARQL server
virtuoso:
build: virtuoso-ops
ports:
- "3003:8890"
volumes_from:
- virtuosodata
virtuosodata:
image: busybox
volumes:
- /virtuoso 5 . 5

Docker and data?
Docker Hub maximum image size: 10 GB
Open PHACTS data (compressed): ~30 GB
Open PHACTS data (installed): ~200 GB
Solution: Added staging Docker containers
Download from
Verify consistency
Import into Virtuso and mySQL
6 . 2

6 . 3

data.openphacts.org
RDF datasets
RDF linksets
VoID metadata/provenance
mySQL-imported linksets
Virtuoso-imported datasets
→ Maven repository
release data as software
→Research Objects
propagate metadata
6 . 4

https://github.com/openphacts/ops-docker
Hardware requirements:
150 GB of disk space (ideal: 250 GB)
16 GB of RAM (ideal: 128 GB)
4 CPU core (ideal: 8 cores)
Prerequisites:
Recent x64 Linux (Ubuntu 14.04 LTS, Centos 7)
Fast Internet connection
Docker
Docker Compose
What do I need?
7 . 2

Follow the GitHub tutorial exactly, customize later
Install latest Docker and Docker Compose
Just testing on Windows or OS X?
.. modify Docker's Linux VM to have enough disk and memory
Firewall? Diﬀerent settings depend on your ﬁrewall details.
Don't worry - Docker is containerized!
..you won't break your machine
Don't jump ahead..
7 . 3

Get the software
curl -L https://github.com/openphacts/ops-docker/archive/master.tar.gz | tar xzv
cd ops-docker-master
sudo docker-compose pull
7 . 4

Get the data
$ sudo docker-compose up --no-recreate -d mysqlstaging virtuosostaging
$ sudo docker-compose logs mysqlstaging virtuosostaging
ops-mysqlstaging | mySQL staging finished
ops-mysqlstaging exited with code 0
ops-virtuosostaging | 09:13:35 --> Backup file # 675 [0x3F02-0x74-0x8A]
ops-virtuosostaging | 09:13:36 --> Backup file # 676 [0x3F02-0x74-0x8A]
ops-virtuosostaging | 09:13:37 End of restoring from backup, 6751701 pages
ops-virtuosostaging | 09:13:37 Server exiting
ops-virtuosostaging | Loading completed
ops-virtuosostaging exited with code 0
7 . 5

Start the services
$ sudo docker-compose up --no-recreate -d
$ sudo docker-compose logs --tail=5
7 . 6

http://localhost:3001/ Explorer
8 . 2

http://localhost:3002/ API
8 . 3

http://localhost:3003/ SPARQL
8 . 4

http://localhost:3004/QueryExpander Identity Mapping
8 . 5

Custom data staging
Diﬀerent Open PHACTS 2.1 licensing options:
Non-Commercial users: Everything
Commercial users: No DrugBank, partial SureChembl
Open PHACTS members: Full SureChembl
9 . 2

Microservices pr dataset
Most queries have separate fragments per dataset
..which could be executed on separate microservices
Better cloud scalability
Easier to test upgrades of individual datasets
But still need "API" layer to do Identity Mapping
and selecting datasets to query
9 . 3

BioExcel Workflow blocks
BioExcel approach: Spin up virtual machine when an
Open PHACTS workflow is started
Workflow bound dynamically to VM instance(s)
Scalability (exclusive access)
Reproducibility (independent/fixed OPS install)
Tool descriptions - exposed in bio.tools
9 . 4

Customization
Make it easier to add third-party data:
datasets, linksets, queries, API calls
..so pharma industry can mix in their in-house data
.. so academics can upgrade and expand datasets
More tooling,
more documentation,
or more training?
9 . 5

Feedback
https://github.com/openphacts/ops-docker/issues
http://support.openphacts.org/
http://ask.bioexcel.eu/
10

SC1 - Hangout 2: The Open PHACTS pilot

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à SC1 - Hangout 2: The Open PHACTS pilot

Similaire à SC1 - Hangout 2: The Open PHACTS pilot (20)

Plus de BigData_Europe

Plus de BigData_Europe (20)

Dernier

Dernier (20)

SC1 - Hangout 2: The Open PHACTS pilot