A guided SQL tour of bioinformatics databases

A Guided SQL Tour of
Bioinformatics Databases
Yannick Pouliot, PhD
Bioresearch Informationist
lanebioresearch@stanford.edu
Lane Medical Library & Knowledge Management Center
2/28/2007

http://lane.stanford.edu

Content





Very abbreviated review of the relational principle
Some of the technology required to connect to a
remote database
Walk-through of the database schema for Ensembl




Walk-through of the database schema for
BioWarehouse




Hands-on querying

Hands-on querying

Resources


Details on connecting to a remote database

Lane Medical Library &
Knowledge Management Center

2

So Why Are We Here?


3

Bioinformatics Databases: Who
Supports Direct Querying?


4

Relational Database Terms


Database: Collection of tables and relationship

between tables


Table
 Collection of records that share a common
fundamental characteristic




E.g., patients and locations can each be stored in their own
table

Record
 Basic unit of information in a relational database


E.g., 1 record per perso

A record is composed of columns (“fields”)
Query
 Set of instructions to a database “engine” to retrieve,
sort and format returning data.






“find me all patients in my database”


5

Main Relational Database “Engines”




Filemaker
MS Access
MS SQL Server

 MySQL
 Oracle



Postgress
Sybase


6

Structure of Relational DB Tables

Data values
live in rows


7

Understanding the Relational Principle: A
Simple Database
“join”








Every patient gets ONE record in the Patients table
Every visit gets ONE record in the Visits table
Rows in different tables can be related one to another
using a shared key (identifier)
There can be multiple visits records for a given patient
There can be multiple tissue records for a given patient

return

8

The Relational Principle at Work


Related records can be found using a shared
key


Example: Patients.ID = Visits.PatientID
Table name Primary Key


9

SQL Querying…With What?
Query browsers used here:

MySQL Query Browser

WinSQL

Other query browsers exist but are more sophisticated



Often more expensive or more complex
Example: PL/SQL Developer, from Allround Automations


10

Example: Network Querying of Ensembl
Database Using MySQL Query Browser


What happens when you use query a remote
database?





DEMO

Of note:
May take some time




Big database, lots of data to return from far away…
Easy to write queries with voluminous output
May have to kill the query…

Setting up ODBC: not discussed here, but cheat sheet instructions are in
handout. Location will also be mailed

11

The Database Schema: Your
Roadmap For Querying


The schema describes all tables and all fields




Used to determine how to inter-relate tables to
retrieve the desired data

Very important:



Must understand schema for accurate querying
Wrong understanding = wrong results


12

Introducing The SQL Select Statement


Good news: This is the only SQL
statement you need to understand for
querying
SELECT LastName, FirstName
FROM Patients


13

Basic Syntax of Select Statement
SELECT field_name
FROM table
[WHERE condition]

[ ] = elective

Example:

Select LastName,FirstName
From Patients
Where Alive = ‘Y’;
Note: case sensitive for all but Oracle
 Query statement are written into a tool such as MS Query or
MySQL Query Browser

Handout: p2

14

SELECT – (Some) Details


15

Moving On:
Real
Biodatabase

Schemas


16

Schemas We’ll Look At…


Remember: Schemas…



describe all tables and all fields
used to determine how to inter-relate tables to
retrieve the desired data

Our schemas today:
 Ensembl
 BioWarehouse

17

Ensembl








Produced by Sanger Institute
Collection of genome databases for many different
organisms
Free, open source
Web querying: http://www.ensembl.org/
FAQ: What is Ensembl?
All PubMed references pertaining to Ensembl and written
by the Ensembl group


18

Exploring the Ensembl Schema


Ensembl CORE schema documentation




First place to go to answer: “what does this table
store?”
Problem: no graphical representation of overall
schema
Relationships harder to appreciate
 Use Catalog function and go from there…



19

“Fundamental” Tables
Fundamental tables
assembly
assembly_exception
attrib_type
coord_system
dna
dnac
exon
exon_stable_id
exon_transcript
gene
gene_stable_id
karyotype
meta
meta_coord
prediction_exon
prediction_transcript
seq_region
seq_region_attrib
supporting_feature
transcript
transcript_attrib
transcript_stable_id
translation
translation_attrib
translation_stable_id


Features and analyses
alt_allele
analysis
analysis_description
density_feature
density_type
dna_align_feature
map
marker
marker_feature
marker_map_location
marker_synonym
misc_attrib
misc_feature
misc_feature_misc_set
misc_set
prediction_transcript
protein_align_feature
protein_feature
qtl
qtl_feature
qtl_synonym
regulatory_factor
regulatory_factor_coding
regulatory_feature
regulatory_feature_object
regulatory_search_region
repeat_consensus
repeat_feature
simple_feature

ID Mapping (Map identifiers between releases)
gene_archive
mapping_session
peptide_archive
stable_id_event

Exernal references (IDs to objects in other dbs)
external_db
external_synonym
go_xref
identity_xref
object_xref
xref

Miscellaneous
interpro

20

Understanding
The Ensembl
Schema Using
The Catalog


21

Querying Ensembl
 Ensembl

runs on the MySQL
database engine
We’ll use WinSQL


MySQL Query Browser can also
be used, as well as lots of other
querying tools


22

Before Proceeding: A Word of Caution


Easy to write queries that…



Retrieve nonsense
Never complete






Scotty to Captain Kirk: “Where going in circles, and at warp 6
we’re going mighty fast…”

Understanding schema is only way to prevent this

Tips:




Use “count” to determine the number of rows in table
BEFORE returning large datasets
Remember: the more tables are joined, the slower the
query


Go to join
23

Demo Queries… To Get You
Started




Query 1: return number of genes stored in
Ensembl Human
Query 2: return number of transcripts
produced by genes stored in Ensembl
Human
 Demonstrates JOINing of tables


24

Exercises
Together:
1. the number of genes stored in Ensembl Human
 2. the number of transcripts produced by genes stored in
Ensembl Human
(10 min)


On your own:
3. the types of analyses that Ensembl provides
 4. the number of types of markers
 5. the number of markers per chromosome for all chromosomes
 6. Extra points: the minimum and maximum marker distances for
markers on chromosome 19
(20 min)



25

SELCT Statement: A Refresher
“Modifiers” of
select list:
 DISTINCT
FROM table_list
 COUNT
[WHERE conditions]
 SUM
 MIN
[START WITH] [CONNECT BY]
 MAX
[GROUP BY group_by_list]
Also:
 ORDER BY
[HAVING search_conditions]
 LIKE (used in
[ORDER BY order_list [ASC | DESC] ]
WHERE clause)
SELECT [DISTINCT] select_list


26

Example Of A Biologically-Useful
Query: All Markers on Chromosome 1


27

Now We’re Talking: Returning
Results into Your Favorite
Tool


SQL query results returned to…


MS Excel


… using Data/Import External Data/New
Database Query




Details: Excel Advanced Report Development
, Zapawa 2005

Spotfire


In Lane
catalog

28

Next:









BioWarehouse

Produced by SRI International
Integration of genome, biochem rxns, pathways, etc databases from
many different organisms
Free, open source
Accessing PublicHouse
FAQ
Schema
All PubMed references pertaining to BioWarehouse and written by
the BioWarehouse group

29

Conceptual Views of the
BioWarehouse Database


30


31

Querying BioWarehouse



We’ll query using MySQL Query Browser
Caveats:


Lots of datasets supported by BioWarehouse…


.. but some critical ones are missing from publichouse
due to licensing requirements, e.g.,





Also: Need to request account to query




MetaCyc
UniProt

Anonymous user not supported

Resource: MySQL v5 Reference Manual

32

BioWarehouse Demo Queries
…to get you started





Query 1: What are the datasets available in
PublicHouse?
Query 2: How many pathways are there for
the EcoCyc dataset?


33

Example Biologically Meaningful Query Of BioWarehouse:
For a Given Pathway, Return Proteins Involved Pathway
and Their Molecular Weight

SELECT D.Name as PathwayName,J.WID AS
ProteinWID, J.Name AS ProteinName,
J.MolecularWeightCalc AS MolecularWeightCalc
FROM Pathway D,PathwayReaction F, Reaction G,
EnzymaticReaction H, Protein J
WHERE D.WID = F.PathwayWID AND
F.ReactionWID = G.WID
AND G.WID = H.ReactionWID and H.ProteinWID =
J.WID
AND D.DataSetWID=19
AND D.Name LIKE "%lipopolysaccharide%"
ORDER BY ProteinName

34

Exercises
Together:



1. How many datasets are there in PublicHouse?
2. What is the number of genes in S. aureus
(SAUR158878Cyc)?

(10 min)
On your own:
3. List the coding region start and ends for all genes that
code for proteins in the SAUR158878Cyc dataset
 4. How many biochemical reactions are there in each
pathway (of any type) in the EcoCyc (=E. coli) dataset?
(20 min)



35

In Summary…




Knowing the db schema is essential
SELECT statement all you need to know
Remote databases good for exploring a schema at
low cost




No installation…

But:




Performance can be poor
Restrictions on data set
Better to install locally if “real work” to be performed

Remember: SQL gives you the power to return results
directly into your favorite tool!


36

Don’t Forget The
Class Evaluation


Resources


38

Setting-Up for
Internet SQL
Querying

39

Setting Up Data Source Names
Steps
1. Make sure you have the requisite
driver (next slide)
2. Create a Data Source Name (Windows
only)
3.
4.

Write your query
Get the results back into Excel!
See Lane videorecorded class Managing
Experiment Data Using Excel and Friends:
Digging Out from Under the Avalanche for lots
more details.

40

Step 1: Getting Drivers
Essential for SQL Querying


A driver is a piece of software that lets your
operating system talk to a database


Installed drivers visible in ODBC manager




Each database engine (Oracle, MySQL, etc)
requires its own driver






“data connectivity” tool

Generally must be installed by user

Drivers are needed by Data Source Name
tool and querying programs
Require (simple) installation

41

MySQL Driver: Needed to Query
MySQL Databases




Windows: Download MySQL
Connector/ODBC 3.51 here
Must be installed for direct querying using
e.g. Excel


Not necessary if you are using the MySQL Query
Browser


42

Oracle Driver: Needed to Query
Oracle Databases


Installing “client” software will also install
driver






Windows: Download 10g Client here
Mac: Download 10g Client here
Free Oracle user account required to
download

Must be installed if you are querying
using MS Query or any other query
browser involving Oracle

43

Step 2: Creating a Data Source Name




A Data Source Name (DSN) tells programs
on your PC where and how to query a
database
Populating the fields:





Data Source Name: Unique name of your choice
Description: anything
Server: exactly as given by the database provider
Port number: as specified by database provider


Defaults: MySQL: 3306; Oracle: 1521; MS Access: N/A


44

Resources – SQL



eBook: Beginning SQL
eBook: Learning SQL


45

Lots More Resources From Lane


46


47

How To Get Accounts for Direct
SQL Querying
Direct Querying of Selected Bioinformatics Databases
Database

How?

DB
Engine
MySQL

BioWarehouse

http://biowarehouse.ai.sri.com/
 get account for access to publichouse
(publicly-accessible installation of
BioWarehouse; see
http://biowarehouse.ai.sri.com/PublicHouse
Overview.html

Ensembl

http://www.ensembl.org/info/data/download MySQL
.html

Mouse Genome
Database

Mail mgi-help@informatics.jax.org to ask
for an account


Sybase

48

Example Querying with MySQL Query
Browser






Free
MySQL only
Facilitates writing of a SQL query 
Execute
graphical
statement
Query statement
Get it at http://www.mysql.com/products/tools/querybrowser/


Table descriptions

49

A guided SQL tour of bioinformatics databases

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à A guided SQL tour of bioinformatics databases

Similaire à A guided SQL tour of bioinformatics databases (20)

Plus de Yannick Pouliot

Plus de Yannick Pouliot (10)

Dernier

Dernier (20)

A guided SQL tour of bioinformatics databases

Notes de l'éditeur