The United States Patent and Trademark Office wanted a simple, lightweight, yet modern and rich discovery interface for Chinese patent data. This is the story of the Global Patent Search Network, the next generation multilingual search platform for the USPTO. GPSN, http://gpsn.uspto.gov, was the first public application deployed in the cloud, and allowed a very small development team to build a discovery interface across millions of patents. This case study will cover: • How we leveraged Amazon Web Services platform for data ingestion, auto scaling, and deployment at a very low price compared to traditional data centers. • We will cover some of the innovative methods for converting XML formatted data to usable information. • Parsing through 5 TB of raw TIFF image data and converting them to modern web friendly format. • Challenges in building a modern Single Page Application that provides a dynamic, rich user experience. • How we built “data sharing” features into the application to allow third party systems to build additional functionality on top of GPSN.

1. Building a Lightweight
Discovery Interface for
Chinese Patents
Chinese Patents
Strata 2014 Santa Clara
Eric Pugh | epugh@o19s.com | @dep4b

2. Who am I?
•
Principal of OpenSource Connections
- Solr/Lucene Search Consultancy
http://bit.ly/OSCCommercialSummary
•
Member of Apache Software
Foundation
•
SOLR-284 UpdateRichDocuments
(July 07)
•
Fascinated by the art of software
development

3. ex
N
n
tio
di
tE
M
!
ar
Co-Author

4. Agilista

5. Selected Customers

6. war
^
Telling some stories

7. •
First USPTO application in
“the cloud”
•
•
Simple, and discoverable
Expresses our philosophy of
“Cloud meets Ocean”

9. Risks
•
•
•
•
Cloud new at USPTO
Discovery is tenuous concept
Conflicting User Goals
Fixed Budget: trade scope for
budget/quality

10. Telling some stories
➡How to inject “Discovery” into your
app
• The Cloud to the Rescue (sorta!)
• Parsers and Parsers and Parsers
• Don’t be Afraid to Share!

11. Flow of understanding
Data
Data
Information
Information
Understanding
Understanding

12. Building “Discovery”
UX
UX
Tensio
n
Data
Data
Engine
Engine

13. UX
UX
User Interviews
Card Sorting
Scenarios/Personas
Data
Data
Grok data at gut level
Look for outliers
brainstorm
brainstorm
brainstorm
brainstorm
Surveys
Mockups
Proof of concept

14. Where to spend time?
UX
UX
Engine
Engine
Data
Data
40%
20%
40%
40%
40%
20%
We
spent

15. Walk through results
http://gpsn.uspto.gov

16. Telling some stories
• How to inject “Discovery” into your app
➡The Cloud to the Rescue (sorta!)
• Parsers and Parsers and Parsers
• Don’t be Afraid to Share!

17. Boy meets Girl Story

18. Boy meets Girl Story
Content
Files
Ingest
Pipeline
Metadata
Discovery
UX

19. How we built it

20. Lessons Learned

21. Don’t Move Files
• Copying 5 TB data up to S3 was very
painful.
• We used S3Funnel which is “rsync like”
• We bought more network bandwidth for
our office

22. Never
underestimate
the bandwidth of
a station wagon
full of tapes
hurtling down
the highway.
–Andrew Tanenbaum, 1981

23. Data Size
277871

24. Think about Data Volume
•
Started with older dataset, and tasks like TIFF -> PNG
conversion became progressively harder. Map/Reduce nice,
need more visibility into progress..
•
Should have sharded our Search Index from the beginning
just to make indexing faster and cheaper process (500 gb
index!)
•
•
8 shards dropped time from 12 hours to 2 hours.
Merging took 5!
We had too many steps in our pipeline

25. Building a Patents Index

26. Cloud meet Ocean

27. More prosaically…
Server
Server
$
Database
Database
Server
Server
$
$
Client
Client
Client
Client
Server
Server
$
Client
Client

28. Telling some stories
• How to inject “Discovery” into your app
• The Cloud to the Rescue (sorta!)
➡Parsers and Parsers and Parsers
• Don’t be Afraid to Share!

29. Morphlines
Why so many pipelines?

30. Tika as a pipeline?

31. Lot’s of File Types
• Sometimes in ZIP archives, sometimes not!
• multiple XML formats as well as CSV and
EDI
• Purplebook,Yellowbook,
Redbook,Greenbook, Questel, SIPO…

32. Tika as a pipeline!
• Auto detects content type
• Metadata structure has all the
key/value needed for Solr
• Allows us to scale up with
Behemoth project (and
others!).

33. Detector to pick File
public class GreenbookDetector implements Detector {
private static Pattern pattern = Pattern.compile("PATN");
@Override
public MediaType detect(InputStream stream, Metadata metadata) throws IOException {
MediaType type = MediaType.OCTET_STREAM;
InputStream lookahead = new LookaheadInputStream(stream, 1024);
String extract = org.apache.commons.io.IOUtils.toString(lookahead, "UTF-8");
Matcher matcher = pattern.matcher(extract);
if (matcher.find()) {
type = GreenbookParser.MEDIA_TYPE;
}
lookahead.close();
return type;
}
}

34. Telling some stories
• How to inject “Discovery” into your app
• The Cloud to the Rescue (sorta!)
• Parsers and Parsers and Parsers
➡Don’t be Afraid to Share!

35. Your solution isn’t
perfect
• Allow users to export data
• Most business users want to work in Excel!
Accept it!
• Allow other applications to build on top of
it.

36. GPSN has
•
•
Lots of easy “Print to PDF” options.
Data stored in S3 as:
•
•
•
•
•
individual patent files
chunky downloads.
Filtering to expand or select specific data sets.
Permalinks: simple, very sharable URLs.
Underlying Solr service is exposed to public via firewall.
You can query Solr yourself.

37. One more thought...

38. Measuring the impact
of our algorithms
changes is just getting
harder with Big Data.

39. e
W
Quepid: Give your Queries
some Love
e
ne
d
t
be
a
s!
er
us
www.quepid.io

40. Office Hours Thurs 10:50 AM
Whats Up with the Lucene
Community?
Community?

41. Questions?
Questions?
Questions?
Nervous about
epugh@o19s.com
•
speaking up? Ask me
• @dep4b
later!
• www.opensourceconnections.com
• slideshare.com/o19s