1. Building
specialized
applica/ons
using
Solr;
Migra/on
from
FAST
ESP
Rahul
Agarwalla
Head
of
Interna/onal
Business
Uchida
Spectrum
Inc.
©2011 Uchida Spectrum, Inc. All rights reserved.

2. Uchida
Spectrum
Overview
SoDware
License
Business
1995
~
• So)ware
License
Sales
• License
Management
Repor:ng
• License
Procurement
System
• License
Adjustment
Consul:ng
Network
Technology
Services
Enterprise
Search
Business
1997
~
• Network
System
Consul:ng
Services
2002
~
― Ac:ve
Directory
Network
• Enterprise
Intelligence
Applica:on
― Exchange
Messaging
Network
― SMART
InSight
G2
Enterprise
• License
Management
System
Consul:ng
― SMART
InSight
G2
Professional
― So)ware
Management
Server
• Search
PlaRorm
Consul:ng
&
Support
• Portal
System
Consul:ng
― FAST
ESP
― Share
Point
Portal
Server
― Lucene/Solr
― Websphere
Portal
Server
― Lucid
Works
Enterprise
©2011 Uchida Spectrum, Inc. All rights reserved. Page-2 Page-2

3. Some
of
Uchida
Spectrum’s
customers
©2011 Uchida Spectrum, Inc. All rights reserved. Page-3 Page-3

4. SMART/InSight
History
Customers
in
Japan,
China
&
India:
• 2
of
top
3
Japanese
car
manufacturers
• Top
consumer
electronics
company
• Large
financial
ins8tu8ons
• China’s
biggest
eCommerce
firm
2005:
SMART
InSight
1.1
2004:
PlaRorm
for
custom
solu:ons
2003:
FAST
Alliance
©2011 Uchida Spectrum, Inc. All rights reserved. Page-4 Page-4

5. What
is
today’s
buzz
word?
Smart Phone
• Extreme
scalability
• Flexibility
&
Extensibility
• Feature
rich
search
©2011 Uchida Spectrum, Inc. All rights reserved. Page-5 Page-5

6. What
I
learnt
from
the
Japan
catastrophe
©2011 Uchida Spectrum, Inc. All rights reserved. Page-6 Page-6

7. The
power
of
community
Japanese
Government
Japanese
People
[Closed/big
brother]
[Open
community]
• Slow,
behind
the
curve
• Quick
response
• Legacy/CYA
• Disclose
/
Share
• Confusion
• Prac:cal
Impact
Power shift
Driver
of
innova/on
©2011 Uchida Spectrum, Inc. All rights reserved. Page-7 Page-7

8. Lessons
from
FAST
ESP
Migra/on:
advantage
LWE/Solr
• Key
Issues:
1. Smaller
record
and
index
size
enable
faster
index
maintenance
2. #
of
records
per
node:
rule
of
thumb
10m
vs.
2m
3. Licensing
&
Maintenance
Cost:
less
than
½
• Scalability:
5x
• Cost
Performance:
10x
• High
Flexibility
• Lower
Opera/ons
Cost
• Faster
Innova/on
©2011 Uchida Spectrum, Inc. All rights reserved. Page-8

9. Enterprise
Search
expecta/ons
• Big
data
scale
• Security
is
important
• Disparate
data:
geography,
systems,
languages,
format,
structures
• KM
is
good
to
have,
databases
are
cri:cal
• Support
different
users
&
usage:
department,
role,
tasks
• High
recall
©2011 Uchida Spectrum, Inc. All rights reserved. Page-9 Page-9

10. Lessons
from
FAST
ESP
Migra/on:
Filling
the
gaps
• Security
• ACL
security:
complex
requirements
• File
System:
file
&
folder
level
control
• CRM/ERP…
:
Keeping
ACLs
up-­‐to-­‐date
• Content
aggrega/on
• Connectors
• Normaliza:on
• Open
source
op:ons
for
ESP
pipeline
• Openpipeline
• Pypes
©2011 Uchida Spectrum, Inc. All rights reserved. Page-10

11. Building
specialized
applica/ons:
Content
fusion
• Content
fusion
from
disparate
data:
• Single
index
≠
integra:on
• Modeling
of
content
rela:onships
is
essen:al
©2011 Uchida Spectrum, Inc. All rights reserved. Page-11

12. Virtual
integra/on
based
on
search
Applica/on
layer
Content
sets
and
inter-­‐rela/onships
Content store
Big
table,
flat
index
Search Index
Search Index
Search Index
©2011 Uchida Spectrum, Inc. All rights reserved. Page-12

13. Virtual
integra/on
based
on
search…2
Search
Service
Content
Append
Pipeline
Tagging
Pipeline
Result
Pipeline
Query
Pipeline
Security
• Data
transforma:on:
.
.
.
.
.
.
- key:key,
key:value,
field
names
Boos:ng
• Query
&
Result
transforma:on
Transform
• Boos:ng
/
Relevancy
algorithm
• Security
• Mul:-­‐Language
support
LWE
Adapter
SolrAdapter
……
Other
• Federa:on
&
mashups
Search Index ……
LWE Solr
©2011 Uchida Spectrum, Inc. All rights reserved. Page-13

14. Building
specialized
applica/ons:
Personaliza/on
• Applica/on
flow
depends
on
the
task
• Data
Personaliza/on
increases
produc/vity
• SMART
InSight
approach:
Task
based
UI
• Schema
independent
widgets
for
analy:cs
&
visualiza:on
• Portalized
• Personalized:
widgets,
func:ons,
content,
fields
©2011 Uchida Spectrum, Inc. All rights reserved. Page-14

15. Knowledge
Center:
made
possible
by
Solr
Scalability
and
low
TCO
gives
us
ability
to
build
new
features
• Knowledge
Centre
has
logs
of
all
user
ac:vity
in
SMART
InSight
• This
would
be
too
costly
with
a
commercial
Search
Engine
and
would
not
be
feasible
in
a
database
Using
this
rich
data
we
can:
• Profile
users,
groups
and
networks
• Personalize
Recommenda:ons
• Create
social
ranking
algorithms
• Usage
analy:cs
©2011 Uchida Spectrum, Inc. All rights reserved. Page-15

16. Overview
of
SMART
InSight
for
Automo/ve
Task
based
UIs
NHTSA
Internet
Page
Widgets
Ajax
Portal
Personaliza/on
Benchmarking
EDR
Virtual
Integra/on
Convergent
Knowledge
Repair
Framework
Framework
Dealers
Knowledge
SA
Contents
Set
Centre
Recommend
Data
Chain
SA
Design
Claims
Profiling
Parts
Catalog
SA
Engineering
Metadata
Analysis
PLM
SA
Claims
Specs
Knowledge
Log
CAD
Internal
Management
&
Security
Early
Defect
Warning
Content
Model
Claim
Analysis
©2011 Uchida Spectrum, Inc. All rights reserved. Page-16

17. :
Interac/ve
Click
Log
Analysis
System
• >
$50
Billion
sales
/
year
• >
800
Million
Items
• >
370
Million
Users
• Billions
of
clicks
per
day
Access
Log
Solr
Hadoop
Solr,
Hadoop
+
SMART/InSight
G2
xxxxxxxx
Xxxx
Xxxx
Xxxx
Xxxx
xxxx
©2011 Uchida Spectrum, Inc. All rights reserved. Page-17

18. :
Global
Research
Community
• Top
Academic
Ins/tutes:
• Faculty,
Research
Fellows
&
Post
graduate
students
• Govt.
Departments
&
Corporate
R&D
• Scien:sts
and
researchers
Research
Discovery
&
Collec/ve
Intelligence
(Knowledge
Centre)
• >
270
content
sources:
Socie/es,
Broadcast
Associa/ons,
Publishers
&
Open
Search
• IEEE,
ACM…
• Elsevier,
Wiley,
Springer…
Dynamic
Result
Merging
Real
/me
indexing
Solr
©2011 Uchida Spectrum, Inc. All rights reserved. Page-18

19. Demonstra/on
©2011 Uchida Spectrum, Inc. All rights reserved. Page-19

20. Contact
Details
Rahul
Agarwalla
Head
–
Interna/onal
Business
rahul@spectrum.co.jp
www.spectrum.co.jp
©2011 Uchida Spectrum, Inc. All rights reserved. Page-20