Presentation given at eHumanities Group, Meertens Institute, Amsterdam (Sept. 2013)

1. WebART project
Web Archive RetrievalTools
Jaap Kamps, Richard Rogers, Arjen deVries
Paul Doorenbosch, RenéVoorburg,Victor-JanVos
Anat Ben-David, Hugo Huurdeman,Thaer Sammar
Flickr: LucViatour
eHumanities Group,“NewTrends in eHumanities”, Sept. 19 2013, Meertens Institute

2. WebART project
Web Archive RetrievalTools
Jaap Kamps, Richard Rogers, Arjen deVries
Paul Doorenbosch, RenéVoorburg,Victor-JanVos
Anat Ben-David, Hugo Huurdeman,Thaer Sammar
Flickr: LucViatour
Data Diggin’ @ KB
eHumanities Group,“NewTrends in eHumanities”, Sept. 19 2013, Meertens Institute

3. Contents
•The WebART project & KB Web archive
•Data Diggin’ @ KB
•Analysis
•DiggingTowards the Future

4. 2012-2016

5. Thaer Samar
PhD/programmer
Hugo Huurdeman
PhD researcher
Anat Ben-David
Postdoc
Arjen deVries Jaap Kamps Richard Rogers
Paul Doorenbosch
Hildelies Balk
Victor-JanVos
RenéVoorburg

6. WebART Goals
•Evaluating current curation and selection
procedures of Web archives
•Getting insights into current use of Web
archives
•Developing new methods and tools for
research using Web archives

7. What are Web archives for?

8. Flickr: koninklijkebibliotheek
KB:Web archive since 2007
Statistics:
•4,000+ websites
•17,000+ harvests
•7+TerabyteSelective approach

9. KB:Web archive since 2007
Statistics:
•4,000+ websites
•17,000+ harvests
•7+TerabyteSelective approach
Original image:A N P

12. ”Wayback Machine” interface

13. Data Diggin’ @ KB
•DMI Summer School (2012)
• analysis of selection lists KB
•DMI Winter School (2013)
• use of nu.nl daily harvests KB dataset
•Workshop: Sept ‘11 Day (2013)
• use of full Web archive KB dataset

14. DMI Summer School (2012)Data digging, part 1
Selection lists KBData:
Toolset: Web-based tools
Flickr: Silvertje

15. DMI Summer School (2012)

16. • Digital Methods Winter School (Jan. ’13)
• Co-design workshop (“Living Lab”)
• New Media researchers & developers
• first use WebARTist
Data digging, part II
nu.nl daily harvestsData:
Toolset: Full-text search Web-based tools

17. • Full-text search:WebARTist (pilot - beta 1)
• Initial dataset (corpus)
• 432 crawls, 16 months (13.64 GB)
KB CommonCrawl+
nu.nl
(Dutch news aggregator)
Full-text searchData digging, part II

18. Full-text search

19. Full-text search

20. Full-text search

21. Full-text search

24. Word frequency analysis
0
100
200
300
400
500
600
700
800
17/05/2011 25/08/2011 03/12/2011 12/03/2012 20/06/2012 28/09/2012 06/01/2013

25. Co-Word Analysis

26. 1
abcnews.go.com1
brucespringsteen.net
1
theverge.com
1
sportamerika.nl
1
reuters.com
1
ebird.org
1
googleblog.blogspot.co.uk
1
presscentre.sony.eu
1
project.wnyc.org
1
bbc.com
1
poynter.org
1
abclocal.go.com
1
en.wikipedia.org
1
nhc.noaa.gov
1
nypost.com
2
earthcam.com
2
maps.google.com
3
hp.com
4
google.org
4
edition.cnn.com
Syria
Sandy
7
wired.com
7
allthingsd.com
7
abcnews.go.com
7
thesun.co.uk
7
allesoversterrenkunde.nl
8
volkskrant.nl
9
fd.nl
9
nos.nl
9
mobiel.nuvideo.nl
9
guardian.co.uk
10
bit.ly
10
billboard.biz
10
cbsnews.com
11
usmagazine.com
11
variety.com
12
theverge.com
12
people.com
13
Rutte enVerhagen leggen schuld bij PVV
13
telegraaf.nl
14
washingtonpost.com
18
edition.cnn.com
19
bbc.co.uk
20
youtube.com
20
nytimes.com
21
styletoday.nl
21
bloomberg.com
24
thesistools.com
26
hollywoodreporter.com
30
online.wsj.com
30
deadline.com
33
poll.nupubliek.nl
34
spaarrente.nl
39
gamer.nl
48
reuters.com
52
tmz.com
57
open.spotify.com
78
peil.nl
93
gezondheidsnet.nl
US Election
4
1
blogs.aljazeera.net
1
youtube.com
1
worldpressphoto.org
1
wikileaks.org
1
washingtonpost.com
1
eubusiness.com
1
vesti.bg
1
trouw.nl
1
#NAME
1
en.wikipedia.org
1
l
1
sana.sy
1
hosted.ap.org
1
shariah4belgium.com
1
nrc.nl
1
guardian.co.uk
1
geopolicity.com
1
nctb.nl
1
rt.com
1
kaspersky.com
2
todayszaman.com
2
volkskrant.nl
2
spaarrente.nl
2
reuters.com
2
peil.nl
2
hrw.org
2
uk.reuters.com
2
cbsnews.com
3
telegraph.co.uk
3
maps.google.nl
4
bbc.co.uk
5
edition.cnn.com
5
aljazeera.com
english.alarabiya.net
7
maps.google.com
Outlink Analysis

30. Geomapping location Wire service

31. Temporal Image Analyses

32. Timeline

33. DMI “9/11 Day” (2013)Data digging, part III
Full KB ArchiveDatasets:
Toolset: Web-based tools
nu.nl “host+1”
Full-text search+ Geo-index

34. Full-text search+

35. Full-text search+

36. Full-text search+

37. •New Media researchers’ interests:
• “derive periodizations of the Web” (Web history)
• “source hierarchy” (dominant sources in archive)
• “keyword uptake” (terms over time)
• e.g.‘geenstijl language in archive’
• “accidental”/“incidental” archiving
• e.g.‘the guilty pleasures of the Web of innocence’
DMI “9/11 Day” (2013)Data digging, part III

38. 2009 2010 2011 2012

39. 2009 2010 2011 2012

40. 2009 2010 2011 2012

41. 2009 2010 2011 2012

42. Analysis (1)
• studying the ‘archive’ vs. the ‘archived content’
• researchers’ (un)familiarity with temporal (archive)
search
• “conditioned” to Google-style searching
• high demand for export functions and aggregation
features

43. Analysis (2)
•“data is still a crucial factor”
• quantity & quality: inherent incompleteness &
inconsistencies
• not always clear what’s in & what’s out
• crawl settings (e.g depth), temporal gaps
• “researchers always want what isn’t there”

44. Digging towards the future
Full KB ArchiveDatasets:
Toolset: “Toolmaker’s tools”
++

45. A step further...
•Build customizable systems, or,
toolmakers’ tools
•Provide building blocks

46. A step further...
use “Hadoop” computing power to build custom dataset, perform high-level analysis, etc.

47. New tools: examples
•select,“clean”, filter & process dataset
•employ complex queries & search strategies
•search, summarize, aggre-
gate & share

48. Moving beyond mere “search”
Wayback
Machine
Search
engine
“Research” engine
explicit support for
full research task,
including analysis
and synthesis steps

49. Summary
•The WebART project
•Data Diggin’ @ KB
•Analysis
•DiggingTowards the Future
Summary

51. webarchiving.nl
@webart12

52. WebART project
Web Archive RetrievalTools
Jaap Kamps, Richard Rogers, Arjen deVries
Paul Doorenbosch, RenéVoorburg,Victor-JanVos
Anat Ben-David, Hugo Huurdeman,Thaer Sammar
Flickr: LucViatour
eHumanities Group, NewTrends in eHumanities, Sept. 19 2013, Meertens Institute