Mahendra Mahey, manager of British Library Labs (BL Labs) will examine some of the BL’s digital collections/data & discuss challenges he has had in making the BL's cultural heritage data available openly or onsite at the British Library.
Mahendra will invite delegates to explore data-sets at their leisure, setting a challenge for those who are interested, skilled in exploring, finding patterns and grouping data. They could become data-set authors/creators of derived data-sets, based on pre-existing digital collections/data provided on the day or already available on https://data.bl.uk.
The workshop will conclude with reflections from the delegates and possibly highlighting a number derived data-sets that were generated by participants on the day that could now potentially exist on https://data.bl.uk. If selected, these new derived data-sets will be attributed with the creators' / authors' details and each will have its own cite-able Digital Object Identifier (D.O.I). These new data-sets would then be available for reuse by any researcher in the world.
GUIDANCE FOR THIS WORKSHOP
We strongly recommend you come to this workshop with an appropriate device such as a laptop pre-installed with appropriate tools to analayse different kinds of data-sets, e.g. Microsoft Excel may work with smaller data-sets such as metadata (see other data exploration tools below). If you don't have one, and would still like to attend, please request to 'pair up' with someone who is willing to share and has already signed up.
Other data exploration tools include: Notepad++ (e.g. for viewing text and XML); Open Refine (e.g. for cleaning data); Tableau Public (e.g. for visualising data); Google Fusion Tables (e.g for visualising geo-spatial data); Spacy (e.g. for text and data mining), RStudio (an open source Statistical package), MATLAB (data analysis tool) & NLTK (Natural Language processing).
Please note that this workshop is NOT about training you in using any of these tools, just tools you may be already familiar with to explore and find patterns in our data.
Datatypes you may be examining in this workshop could include: .ZIP, .PDF, .TXT, .CSV, .TSV. .XLS, .XLSX, RDF, .nt, XML (TEI, ALTO and bespoke), .JSON, .JPG, .JPEG, .TIFF and .WARC
Please ensure you are able to read these files on your device before the workshop if you are interested in exploring them during our session.
Slides for session: http://goo.gl/
URL for specific data: http://
Mahendra Mahey tweets at @BL_Labs & @mahendra_mahey
Planning for big data (lessons from cultural heritage)
Similaire à A hands-on data exploration & challenge to become a derived data-set author on the British Library’s open data-set platform (https://data.bl.uk)
Similaire à A hands-on data exploration & challenge to become a derived data-set author on the British Library’s open data-set platform (https://data.bl.uk) (20)
A hands-on data exploration & challenge to become a derived data-set author on the British Library’s open data-set platform (https://data.bl.uk)
1. 1
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
http://www.bl.uk/projects/british-library-labs
Funded by the Andrew W. Mellon Foundation
Running since March 2013
A hands-on data exploration & challenge to become a derived data-set author
on the British Library’s open data-set platform (https://data.bl.uk)
Mahendra Mahey, Manager of BL Labs, British Library, London, UK.
1400 – 1530, Tuesday 25 September 2018
Workshop part of ‘Making Connections’, Digital Humanities Australasia, 2018
(#DHA2018), University of South Australia, City West campus, Adelaide, SA, Australia
2. 2
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Who do we work with?
Researchers
https://goo.gl/WutNyi Artists
http://goo.gl/nNKhQ2
Librarians
Curators
https://goo.gl/9NWZUW
Software Developers
https://goo.gl/7QQ5Tf
Archivists
https://goo.gl/x7b4tg
Educators
https://goo.gl/qh01Mi
Working and Communicating
Entrepreneurs
https://goo.gl/Fx8RG7
3. 3
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Competition
Awards
Projects
Tell us your ideas of what to do with our digital content (2013-16)
Show us what you have already done with our digital content in research,
artistic, commercial, learning and teaching, staff categories
Talk to us about working on collaborative projects
Tell us your ideas of what to do with our digital content
Engagement
• Roadshows
• Events
• Meetings
• Conversations
New! Digital Research Support
How?
4. 4
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Collections – not just books!
> 180*million items
> 0.8* m serial titles
> 8* m stamps
> 14* m books
> 6* m sound recordings
> 4* m maps
> 1.6* m musical scores
> 0.3* m manuscripts
> 60* m patents
King’s Library *Estimates
5. 5
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Have you got X?
https://upload.wikimedia.org/wikipedia/commons/5/50/Real_wuerzburg.jpg
Looking for Physical Content in the British Library
6. 6
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
#bldigital
3 %* digitised
* estimate
Digital
Partnerships
Commercial & Other
Organisations
Bias in digitisation
http://goo.gl/bR9UJL
Sample Generator
15 %* Openly Licensed – most online
85 %* Available onsite only at the moment
Digitisation / Curating Born Digital
costs money, time, resources
http://www.turing.ac.uk
Digital increasing
rapidly
Born Digital
http://www.webarchive.org.uk/ukwa/
7. 7
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Have you got X digitised / in digital form?
http://www.yorkmix.com/wp-content/uploads/2014/04/mr-simms-sweet-shoppe-york.jpg
Looking for Digitised / Digital Content in the BL
8. 8
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Our Audience and Collections
Audience
research &
Digital
interests
Digital
collections we
have
This is where Labs works
It starts with a making connections!
The theme to DHA2018
9. 9
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Finding Open Cultural Heritage Datasets
Collection Guides (219 as of 25/09/2018)
https://www.bl.uk/collection-guides/
Datasets about our collections
Bibliographic datasets relating to our published and archival holdings
Datasets for content mining
Content suitable for use in text and data mining research
Datasets for image analysis
Image collections suitable for large-scale image-analysis-based research
Datasets from UK Web Archive
Data and API services available for accessing UK Web Archive
Digital mapping
Geospatial data, cartographic applications, digital aerial photography and
scanned historic map materials
https://data.bl.uk
Download collections as zips, no API
Each dataset has a Digital Object Identifier (DOI)
can be referenced for research
Not all discoverable via
search engines!
10. 10
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Explore Our Data at http://data.bl.uk!
• CSV of Metadata
https://data.bl.uk/digbks/dig19cbooks-mdata-csv.csv
• 19th Century Books - Book Metadata - 01/09/2013.
https://data.bl.uk/digbks/db21.html
• Digitised Books - Flickr Tag History - Dec 2013 to March 2016. TSV
https://data.bl.uk/digbks/db15.html
• Digitised Hebrew Manuscripts - Metadata
https://data.bl.uk/hebrewmanuscripts/heb1.html
• Digitised Hebrew Manuscripts: Or 2210 - Or 2364
https://data.bl.uk/hebrewmanuscripts/heb8.html
• Theatrical playbills from Britain and Ireland (OCR text only)
https://data.bl.uk/playbills/pb2.html
• Portraits of actors, views of theatres and playbills (covering 1750 - 1821 in a single volume)
https://data.bl.uk/singlesheet/por1.html
• Volumes of Lysons Collectanea (Amusements), comprising broadsides, cuttings, advertisements on
amusements.1660-1840. https://data.bl.uk/singlesheet/ad1.html
11. 11
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
The Story of the Digital Collection…
Digital
Collection
Curator
Who paid for the digitisation?
Who did the digitisation?
Technology used
Born digital?
Published
Unpublished
Where is it?
Access / API?
Can it still be accessed?
Generates income
Reputational risk in using?
Legalities /
Ethics / Morality
Politics when digitised
Personalities involved
Surprises (e.g. gaps)
Descriptive information
Old format not supported
What media was the
digitisation done from?
Is there any background documentation?
No Descriptive information
Inconsistent descriptive information
Still there?
Good to know the background ‘story’ of a Digital Collection
if you want to use it for projects …
12. 12
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
https://goo.gl/qpCLlk
https://goo.gl/wMTS3Z
• Dialogue typically:
– you are ‘lucky’ & we have the digital content
/ data relevant to your research
– we don’t have exactly what your looking for,
but is there anything of interest? Let’s talk…
– engagement is hard work and it’s constantly required to
maintain interest in our digital collections!
• Artists find this dialogue easier…
• We also tend to attract researchers with ‘fuzzier’ research
boundaries and possibly open to more
interdisciplinary / collaborative research
What engagement does the BL have with researchers
wanting use our digital content?
14. 14
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
How do we give access to
onsite-only
Digital Collections
(85% of our Digital Collections)?
15. 15
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
READING
ROOM
ON
SITE
NOT
ONLINE
OPEN
British Library
£
Labs Residency Model
Challenges of access to Digital Collections at the BL
16. 16
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Accessing digital collections onsite
OPEN
£
• Have to be ‘onsite’ (interpretations vary)
• Need to be ‘security cleared’ ‘trusted’ for some collections
– Hence ‘Researcher in Residence Model’
• Permission required (depending on ‘story’ of collection)
• Content could be on various media formats
(not always online)
• 5 - 20 % re-use of material for non commercial research for some collections,
depends on agreements in place
• We are learning ‘pathways’ so that this becomes ‘everyday’ to provide onsite access
to some digital collections in the future
17. 17
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Phases of interaction at BL Labs
Submit idea for
support
Ideas always change
Once people experience the data
and culture of the organisation
19. 19
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
URLs to download sample files not on data.bl.uk
• https://www.data.sa.edu.au/dataset/newspapers-from-british-library/
• https://www.data.sa.edu.au/dataset/
• https://www.data.sa.edu.au/dataset/
20. 20
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Working with British Library Digitised Newspapers
• Digitised through public / private means
• Can use commercial products to look manually for content, with search
interfaces but no APIs, useful starting point though, manual methods can
translate into computational ones
• OCR quality is not great, metadata is OK, but plenty of hidden material,
approaches require to consider this, e.g. ‘Good, Bad and Ugly’ OCR
• You can purchase drives from GALE Cengage with content (dependent on
subscription)
21. 21
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Good, Bad, Ugly Image Quality / OCR
• Original image capture of newspaper images can effect the quality of the OCR
• A poor image, very difficult to re-OCR
• Good image quality much better chance for re-OCR
• Bi-tonal, Grey Scale, Colour can effect the quality of the OCR
• Methodology of working with collection at scale needs to acknowledge OCR and
image quality
23. 23
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Burney Collection
• Gathered by the Reverend Charles Burney (1757- 1817)
• 700 volumes, newspapers and news pamphlets, published in London, English
provincial, Irish and Scottish papers, and a few examples from the American
colonies.
• 1271 titles
• Around 1 million digitised page images – from around 2006 from Microfilm
• OCR quality mixed, used custom XML format
• Bi-tonal
28. 28
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Breakdown of titles
Title No. of Pages
PUBLIC ADVERTISER 60680
LONDON GAZETTE 44463
LONDON EVENING POST 38920
LONDON CHRONICLE 32030
GAZETTEER AND NEW DAILY ADVERTISER 31250
LLOYD'S EVENING POST 28941
ST. JAMES'S CHRONICLE OR THE BRITISH EVENING POST 28130
MORNING CHRONICLE AND LONDON ADVERTISER 27658
DAILY COURANT 25334
GENERAL EVENING POST 23500
12 TITLES WITH 10,000+ PAGES 188266
87 TITLES WITH 1,000+ PAGES 289745
216 TITLES WITH 100+ PAGES 79374
945 TITLES WITH 1 TO 100 PAGES 16816
29. 29
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Example Folders
B0001ORIWEEJO - APPLEBEE''S ORIGINAL WEEKLY JOURNAL - 1715 – 1720
B0018CONTPROC - PROCEEDINGS OF THE ARMY UNDER THE COMMAND OF SIR
THOMAS FAIRFAX – 1645
B0054REPINFCH - REPORT OF THE STATE OF THE GENERAL INFIRMARY AT
CHESTOR - 1754?-1779
B0101PROCPARL - EXACT RELATION OF THE PROCEEDINGS AND TRANSACTIONS
OF THE LATE PARLIAMENT – 1654
B0277INSTRUCT - INSTRUCTOR – 1724
B1381SCOU1717 - SCOURGE (1717, REPRINT) - 1717?
30. 30
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Example files
‘service’ folder contains page level images and corresponding OCR XML
BurneyB0001ORIWEEJO17151119service
31. 31
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
APPLEBEE''S ORIGINAL WEEKLY JOURNAL
FROM SATURDAY NOVEMBER 19 TO SATURDAY NOVEMBER 26 1715
WO2_B0001ORIWEEJO_1715_11_19-0001.tiff
36. 36
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Accessing digitised newspapers
onsite at the BL
Accessing ‘service’ Copy (post processed)
and results of OCR available as XML
40. 40
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Metadata from BL (JISC 1 and 2)
• Title Metadata
– Title, as written
– Normalised title across all
variants
– Standardised title
abbreviation
– Variant titles, with associated
dates
– Place of publication
– Dates of publication
– Genre, such as newspaper
– Sub-collection, such as
Regional Daily
Issue Metadata
Volume Number
Issue Number
Date as printed
Normalised date (YYYY.MM.DD)
Number of pages
The microfilm reel number
The OCR quality
Page image data
The number of the image within that
issue
The filename
The spatial coordinates for the page
within the image
The degree of page skew
41. 41
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Metadata from Gale (JISC 1 and 2)
• Standardised identifier
• Newspaper title
• Standardised title abbreviation
• Project codes
• Digitized collection name
• Issue number
• Date as printed
• Standardised date (Month, DD,
YYYY)
• Standardised date
(YYYYMMDD)
• Day of the week
• Number of Pages
• Copyright holder
Language
Unique ID for publication
Holding Library
Citation of the physical item
Title metadata
Title as recorded in the MARC
Library Catalogue
Dates of publication
Genre, such as newspaper
Conversion credit, usually a vendor
Article
Unique ID
OCR quality
SC, or standardized category of article
Unique ID(s) of page(s)
Unique ID(s) of individual column(s)
Column number
Headline
Article type
42. 42
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Samples for JISC 1
‘master’ contains high res tiff
‘service’ contains post processed tiff and OCR XML
BNWL - The Belfast News-Letter - 1871 - November 14
BNWL - The Belfast News-Letter - 1885 - September 12
DNLN - Daily News - 21 Jan 1846 - 31 Dec 1900
47. 47
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Samples for JISC 2
Lancaster Gazetter, And General Advertiser For Lancashire West
Southampton Herald
Berrows Worcester Journal
A - Contains post processed files
M - Contains JP2
O - Contains ALTO XML
48. 48
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Previous ideas of using collection
• Bob Nicholson – Finding jokes
• Katrina Navickas – Political meetings
• Hannah Murray – Black abolitionist performances
• Jennifer Batt – Finding poetry
• Surendra Singh – Finding suicide articles
• Melodee Beals – Evidence of copy and paste
• Ryan Cordel – Viral Texts
• Paul Fyfe - Snipping out images
50. 50
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Use of Overproof
OCR Correction?
Re-OCR with
ABBY FineReader?
https://www.abbyy.com/en-gb/
http://overproof.projectcomputing.com/
RE-OCR
51. 51
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Virtual Infrastructure for OCR text
OCR text ‘scraped’ from
digitised newspapers
and put in cloud
Jupyter notebook
Write python code and results
in web browser
http://jupyter.org
Access available for researchers ‘in residence’
https://www.docker.com/
52. 52
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
65,000 digitised 19th Century books
Image: Artwork by Alicia Martin 2007 / 2008
Paid for by:
For a full list:
https://goo.gl/HqPQMS
Subjects include:
Philosophy
Poetry
History
Literature
1789 - 1876
53. 53
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Working with the MS Books Collection
• Metadata
• Page level images
• OCR Text
• Flickr Commons - images snipped out and user generated tags for images
• 19th Century Books Collection data
64. 64
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Optically Character Recognised (OCR)
generated Text
Scanned Page
Image on Flickr
Commons
https://goo.gl/AC43vs
65. 65
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Worked better for female faces than men’s
Press
http://mechanicalcurator.tumblr.com
Posts image every 30 minutes
http://www.flickr.com/photos/britishlibrary/
1,020,418 images
need tagging!
Creative uses of images
Face recognition
Algorithms based on photos
Mechanical Curator
with an algorithmic brain
(Circles, Squares and Slanty etc)
http://goo.gl/qPPgxX
Wikimedia
Flickr Commons
Individual URL & API
Snipping out images
from 65,000 Digitised Books*
>1000,000,000* views
>17,000,000* tags
https://goo.gl/FgZ4HM
Work @ BL by Ben O’Steen, Labs
and Digital Research Team*Matt Prior - http://goo.gl/j29Tnx
Since Dec 2013
Tumblr
*Estimates
>More demand to see
physical items
66. 66
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
British Library Flickr Commons
https://www.flickr.com/photos/britishlibrary/
Flickr Commons has items from
Galleries, Libraries, Archives and Museums (GLAM)
(Mostly Public Domain)
68. 68
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Getting an account on Flickr
•Get a Flickr / Yahoo account
(https://login.yahoo.com/account/create)
•You can then tag, organise favourites, make your own
albums and galleries from Flickr images online or uploaded
•You get 1TB for free!
69. 69
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
British Library Flickr Commons
Why Flickr Commons?
• Free!
• Each image has it’s own unique web address, easy to share
• Can Tag images
• Has Application Programming Interface (API)
Late August 2013
70. 70
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Using British Library Flickr Commons
•How do we find things in this collection?
•Remember snipped out images from books with no
description?
•Not straightforward…
72. 72
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Flickr Photostream
https://www.flickr.com/photos/britishlibrary/
Kind of the home page for the collection!
Usually displays images with most recent activity!
73. 73
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Flickr Albums
Curated by the British Library – specifically Nora McGregor
She works with the public to add images or create new ones!
Over 450 Albums as of 25/09/18 – Mostly Maps!
74. 74
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Flickr Faves
Most favorited image first in descending order
To favourite an image requires an account
75. 75
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Flickr Galleries
More useful if you have an account
You can create a Gallery of Flickr images to share with everyone
Gallery is tied to your account
76. 76
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Flickr Groups
Community based – for sharing and discussing images
We might create a group for the competition – watch this space!
84. 84
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Opportunities
– increasing traffic to Library services
You can purchase
a ‘High Res’ Copy
View in the
Library Item Viewer
Download .pdf
All illustrations
in book
Other illustrations in books
Published in same year
View the item in
the Library Catalogue Tags auto generated
User generated
Tag
Grouping for image
101. 101
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Warning – can be large file!
It’s aPDF
You can do Ctrl F in it to find text
But health warning about OCR!
103. 103
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Page numbers don’t always correspond!
Page numbers
Don’t always correspond
Page 132 on Flickr?
Is Page Number in PDF
In PDF of
book
Page number
in book
105. 105
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Plain Text from Books?
Not working
But can be obtained from https://data.bl.uk/digbks/db14.html
106. 106
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
All illustrations in book / books in same year!
All the illustrations in this book Other illustrations books published
in the same year
111. 111
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Tagging a million images
Iterative Crowdsourcing
http://goo.gl/j6fxac
Cardiff University’s
Lost Visions Project
http://www.metadatagames.org/
Metadata Games
James Heald
Mario Klingemann
Chico 45
Use computational methods
Human Tagger
Top British Library Flickr Commons Taggers
18 hard core taggers
How to reward and keep motivated this ‘small group?
Average for ‘crowd’ is 1 tag per person
What kind of ‘task’ can this ‘crowd’ do?
Mobile games for ‘Ships’, ‘Covers’ and ‘Portraits’ Interface for tagging
120. 120
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
19th Century Books Metadata
• 1,9 Million records of 19th Century Books
• Used for Sample generator project
121. 121
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Using the Wikimedia Synoptic Index
• Created to help find all the maps in the books
• Great resource if you want to find things by place!
https://goo.gl/zuxRnG
124. 124
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Alston Index
Internal Document
55-602 - Topical Index
603 - 925 - Pressmark Sequence925 page document of BL /
British Museum Pressmarks
125. 125
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Alston Index
• Internal document (not to be externally shared)
• Published in 1987 – dot matrix printed
• Refers to British Museum and British Library Pressmarks / Shelfmarks
• Shelfmarks are used internally to identify
130. 130
@BL_Labs #DHA2018 @BL_DigiSchol labs@bl.uk
Data Journey
• Choose one or two datasets maximum
• Explore the collection and make notes about any challenges and issues
• See if you can curate a smaller collection from the larger collection
• Tell us what you have done
• We will consider to publish on http://data.bl.uk
Notes de l'éditeur
23 seconds (71 words)
Though the project focusses on working and communicating with Digital Humanities and Digital Scholarship researchers, we have also engaged with amazing Artists, Librarians, Curators, Educators, Entrepreneurs, Archivists, Software Developers and other innovators. Hopefully, I will show you<CLICK>
some inspirational examples of work they have done which have used our digital collections.<CLICK>
I will also reflect on our experiences, challenges and lessons we have learned working with some amazing and pioneering people.
85 seconds
The picture you can see is inside the main building in London, it’s the King’s Library – King George the Third’s personal library! Sometimes known as the ‘stack’, I walk past this everyday and I sometimes forget that the collections the British Library have are truly staggering! We currently estimate them to exceed <click>150 million items, representing every age of written civilisation and every known language. Our archives now contain the earliest surviving printed book in the world, the Diamond Sutra, written in Chinese and dating from 868 AD….
So some big numbers…
Over …<click>14 million books
<click>60 million patents
<click>8 million stamps
<click>4 million maps
<click>3 million sound recordings
<click>1.6 million music scores
<click>over .3 million manuscripts
<click>0.8 million serials titles (which are of course made up of many many volumes/editions), this is where a lot of our content is, just in case you thought the numbers didn’t add up!
28 seconds (85 words)
This what I imagine it feels like for a researcher looking for our physical collections. <CLICK>
Everything is on an industrial scale and it can feel overwhelming. Sometimes it isn’t always straightforward to find our items, as there are many that are not on our digital library catalogue, e.g. still on card catalogues and some items are in the secret and very secure parts of the Library where you would need very special permission because the items are extremely valuable and fragile for example.
24 seconds (72 words)
The BL are world renowned experts in digitising materials from our physical holdings. One common misconception that many people have is that much if not all of our collections are digitised. So, the actual proportion of our collections that are digitised surprises many<CLICK>
The figure is around 3% of our physical collections.<CLICK>
Much of our digitisation activity happens through partnerships with commercial, philanthropic, charitable and foundation partners<CLICK>
What is for certain, is the amount we are digitising is increasing rapidly. Our new programme called Heritage Made Digital for example prioritises those collections for digitisation where there is a clear researcher demand.<CLICK>
One important thing we have learned is that researchers need to take heed when doing research based on our digital collections, as they are rarely complete, having gaps and not necessarily being representative of our physical collections.
36 seconds (109 words)
Our digital offering is perhaps like this.<CLICK>
Imagine entering a boutique sweet shop. We have some lovely things to tempt you, but it’s much smaller than the hypermarket you just visited. The shop keeper tells you there are some things behind the back door in a giant warehouse. However, you will need special access to enter that space. She also states that there are rooms in that warehouse, even she isn’t allowed to look. She isn’t even allowed to share the full list of stock because there are items on there she may never be able to be see because they were meant to be secret.
12 seconds (37 words).
In another way, we are trying to match our audiences research needs and digital interests <CLICK>
With the digital collections we have<CLICK>
It is at this intersection where Labs works best and it usually starts with a conversation.
41 words (125 seconds)
Our work in Labs has taught us that it always pays for researchers to know the back ‘story’ of a digital collection especially if they want to use it for research and analysis.<CLICK>
There are too many things to consider right now, but a few highlights are such as, ‘are there gaps in the collection?’, ‘can they still be accessed?’, but perhaps most important of all is whether the curator or a human being who knows about the collection is still around who could be asked about it. Our experience has told us that so much will probably be in their head that isn’t written down, information that could be vital, important and useful for knowing about before carrying out research or re-use.
<click>The British Library faces many challenges of access to our Digital collections!
<click> Sometimes digital content is only available onsite due to license restrictions,
<click>or even only on a specific computer in a reading room! Technically there are very few reasons why digital content can’t be online
<click> though it might be too big or hasn’t been transferred from other digital storage media.
<click>Sometimes access is through a paywall. Finally,
<click>some content is in the happy sunny place, online, open and freely available.
The real reasons why there are challenges to accessing digital content are of course human. They require different approaches from the Library and may often involve an honest, open dialogue and negotiation with the publishers.
The Labs project has tried to address this problem my creating a ‘residency model’ for researchers to work intensively with a digital collection on-site, so as to not infringe access conditions, I will say more about this later.
24 seconds (72 words)
Let’s look a little further at the types of interactions we have with our researchers. We have summarised these phases as ‘Exploration’ where people often ‘rethink’ their ideas of what they want to do with the data, ‘Query-Focused’ where they often have to iterate to come up with a realistic proposal of what they want to do and a ‘Wrap-up’ phase to end their project with us, if it is relevant.
970 files from a selection of 19th century newspaper titles from the BL corpus for us to correct using the overProof post-OCR correction software
The best way to measure the improvement made by the correction process is to compare the OCR'ed text and the automatically corrected text with a perfect correction made by a human (known as the "ground truth").
Hannah-Rose's 5 small human-corrected samples are show as green dots. These are not only smaller than the other files, but their raw error rate is much lower at 13.3%. OverProof was measured as reducing this to 5.4%, a removal of almost 60% of errors.
The red dotted-line indicates the correction "break-even" point: the further under the line, the better the quality of the document after correction.
In the graph below, the grey line shows distribution of files across error rates before correction and the green line after correction.
Posts small illustrations taken almost at random from the digitised book corpus to a Tumblr blog.
This experiment with undirected engagement was a by-product of work to uncover the hidden wealth of illustrations within the digitised pages.
50 seconds
Here is the anatomy of a Flickr record, importantly we have created links to many of the Library’s services <click>some of this lovely traffic is going back to the Library and hopefully generating more interest in our services, from downloading a pdf of the book to purchasing a high res scan of the image.
<click>Tags are added from the original book record, including the approximate page number the image came from<click>users of Flickr can add their own tags, and I have mentioned they have already started doing it.