This technical talk describes the usage of Apache based open source software (Apache Lucene/SOLR, Apache Nutch, Apache Tika and Apache ServiceMix) within the implementation of an enterprise search and retrieval platform. The platform is the result of years of experience with enterprise search technologies combined with enterprise application integration and semantic (web) technologies, both within commercial and open source based environments. The talk will dive into the conceptual architecture of a typical search solutions based upon a real world use case, and will then present the accompanying framework that makes easy and swift implementations of enterprise search solutions possible, based upon this architecture. The architecture describes an innovative enterprise search solution, specifying all necessary components for collecting and indexing content (known in the architecture as the collection process, which consist of inbound, splitting, validating, filtering, enriching and indexing components) and publishing the content (known in the architecture as the publication process, which consists of inbound, validating, request enriching, searching, grouping, response enriching and presentation components). The framework can be seen as an orchestrator framework and contains all tools, components and default configurations and flow descriptions necessary to build enterprise solutions according to this architecture. The framework is entirely based upon open source technologies and are mainly Apache based
Portal Kombat : extension du réseau de propagande russe
Open source enterprise search and retrieval platform
1. Datum 21 augustus 2010
Enterprise Search
EAI
Semantic Web
Open Source
Search & Retrieval
Platform
Marc Teutelink
2. How Apache open source software is used
during the implementation of an
Enterprise Search and Retrieval Platform
(Lucene/SOLR, Nutch, Tika, ServiceMix/Camel, Felix/Ace)
3. Marc Teutelink
marc.teutelink@luminis.eu
@mteutelink
•Software architect at Luminis
•15+ years experience in software development; specialized in
Enterprise Search, Enterprise Application Integration and
Semantic Web technology
•Currently writing “Enterprise Search in Action” for Manning
(Mid-2011)
4. Agenda
•Enterprise Search
• What is Enterprise Search: Functions and features
• Challenges
• Logical Architecture
•Enterprise Search Solution
• Technology Stack
• Collection Process
• Publication Process
• Enricher framework
• Deployment
•Conclusion
5. What is Enterprise Search?
“Enterprise Search offers a solution for searching,
finding and presenting enterprise related information
in the larger sense of the word”
Enterprise search is all about searching through documents from
any type and format from any sources located anywhere with the
upmost flexibility
• Web search: limited to public documents on the web
• Desktop search: limited to private documents on the local machine
• Enterprise search: no limitations on document type and location
6. Enterprise Search
(features)
•Information Sources and Types
• Wide range of sources: local and remote filesystems, content repositories,
e-mail, databases, internet, intranet and extranet
• Type not limited: any type ranging from structured to unstructured data, text
and binary formats and compound formats (zip)
•Usage
• Not limited to interactive use automated business processes
•Security
• Integrations with enterprise security infrastructure
•User Interaction and personalization
• Identity enables more personalized search results
7. Enterprise Search
(features)
•Extended metadata
• More metadata better and more precise search results
• More control over schema (for example Dynamic Fields)
•Ranking
• More control over ranking: personalized ranking (group)
•Data extraction and derivation
• Extract data using various techniques: Xpath, Xquery
• Derive data: using external knowledge models: RDBMS, RDF Store, Web Services
• Conditional extraction & derivation
•Managing and monitoring
• On-the-fly management (JMX)
• Real time monitoring
8. Enterprise Search
(features)
•User Interfaces
• Web search
• All about selling advertisements to the mass
• Generalistic & minimalistic screens; focus on adds
• Enterprise search
• All about finding: rich navigation; focus on quick find
• Small targeted audience
• Specialized and customized screens (use of ontologies, taxonomies
and classifications)
• Use of identity (results customized to user) and web 2.0
• Grouping
• field collapsing, faceted search & clustering
9. Enterprise Search
(Challenges)
•Performance and scalability
•Rich functions and features
•Managebility
•Flexibility
•Easy maintenance
•Quick issue and problem solving
•Reduce total cost of ownerschip
10. Enterprise Search
(Challenges)
•Performance and scalability
•Rich functions and features
•Managebility
•Flexibility
•Easy maintenance
•Quick issue and problem solving
•Reduce total cost of ownerschip
Commercial Search Engines?
11. Enterprise Search
(Challenges)
•Performance and scalability
•Rich functions and features
•Managebility
•Flexibility
•Easy maintenance
•Quick issue and problem solving
•Reduce total cost of ownerschip
Apache Based (Open Source)
Search & Retrieval Platform
30. Luminis Enricher Framework
•Custom Enricher Framework
• Existing ESB & SOLR enricher capabilities not sufficient.
• Enriching = one or more actions (extraction, enhancing &
filtering) performed on documents with fields
• Same enricher to be used for:
• Collection process:
• Documents enriching, filtering & splitting
• Publication process:
• Search requests’first-components’ searchcomponent
• Search response’last-components’ searchcomponent
31. Luminis Enricher Framework
•Custom Enricher Framework
• Existing ESB & SOLR enricher capabilities not sufficient.
• Enriching = one or more actions (extraction, enhancing &
filtering) performed on documents with fields
• Same enricher to be used for:
• Collection process:
• Documents enriching, filtering & splitting
• Publication process:
• Search requests’first-components’ searchcomponent
• Search response’last-components’ searchcomponent
Content Indexer
Content Inbound
2
1
Documents
Message
N
D
Document
Messages
D D
Lucene/Solr
INDEX
SOLR Indexer
(Channel Adapter)
Lucene/SOLR
(SOLRJ)
D
SOLR Document
Message
Splitter
Channel
Content Validation Content Enrichment
Enricher
Content Filter
Content Enricher
Syntactic Validation
(Channel Purger)
Push Inbound
(Message Endpoint)
Semantic Validation
(Channel Purger)
Invalid Message
Channel
!
?
Invalid Message
ChannelChannel
32. Luminis Enricher Framework
•Custom Enricher Framework
• Existing ESB & SOLR enricher capabilities not sufficient.
• Enriching = one or more actions (extraction, enhancing &
filtering) performed on documents with fields
• Same enricher to be used for:
• Collection process:
• Documents enriching, filtering & splitting
• Publication process:
• Search requests’first-components’ searchcomponent
• Search response’last-components’ searchcomponent
Content Indexer
Content Inbound
2
1
Documents
Message
N
D
Document
Messages
D D
Lucene/Solr
INDEX
SOLR Indexer
(Channel Adapter)
Lucene/SOLR
(SOLRJ)
D
SOLR Document
Message
Splitter
Channel
Content Validation Content Enrichment
Enricher
Content Filter
Content Enricher
Syntactic Validation
(Channel Purger)
Push Inbound
(Message Endpoint)
Semantic Validation
(Channel Purger)
Invalid Message
Channel
!
?
Invalid Message
ChannelChannel
<<SearchHandler>>
RequestHandler
"first-components" "components" "last-components"
<<XML>>
Response
<<SearchComponent>>
query
<<SearchComponent>>
facet
<<SearchComponent>>
mlt
<<SearchComponent>>
highlight
<<SearchComponent>>
stats
<<SearchComponent>>
debug
<<SOLRQueryRequest>>
Query
<<XSLT>>
XML2HTML
<<QueryResponseWriter>>
XSLTResponseWriter
<<(X)HTML>>
Resultaat
33. Luminis Enricher Framework
(architecture)
•Pipe-and-filter architecture
• Documents flow through series of actions
• Output from one action is input to another action
• Fields from input document can be used in action’s clauses: values in
expressions filled by replacing velocity type patterns with field values
•Conditional flows supported
•Reuse of flows & Subflows supported
34. Luminis Enricher Framework
(architecture)
•Pipe-and-filter architecture
• Documents flow through series of actions
• Output from one action is input to another action
• Fields from input document can be used in action’s clauses: values in
expressions filled by replacing velocity type patterns with field values
•Conditional flows supported
•Reuse of flows & Subflows supported
Action
(select C where ${B})
Action
(remove A2)
Document
[[A1,A2],[B]]
Document
[[A1],[B]]
Document
[[A1],[B],[C1]]
If [B=3]
YES
Action
(select C where ${A})
Document
[[A1],[B],[C2]]
NO
35. Luminis Enricher Framework
(Configuration)
•Enricher flow and expression configuration via XML based DSL
• Conditional: if-then-else & switch-case-else (with regex support)
• Actions: Add & remove fields and field values using expressions
• Expression handlers currently supported:
• Field
• Function (execute methods via Java Reflection)
• HttpClient (retrieve content by URL described by field values)
• Xslt, Xpath, Xquery (external XML databases)
• JDBC
• SparQL (OpenRDF)
• Apache Lucene/Solr
• Apache Tika (Meta and Text extraction)
40. Luminis Enricher Framework
(Technology)
•Enricher and expresion handlers are Java based OSGi
services:
• Hot pluggable and updatable
• Flow and expression configuration changes no restart
• Extendible: New expression handlers immediatly available in
actions after installing OSGi bundle
•Runs in Apache Felix
• Collection Process: ServiceMix contains OSGi container
• Publication Process: Custom OSGi loader for Lucene/Solr
•Centralized & transactional provisioning (Apache Ace)
‑ Components & Configuration