1. Using full text search in OpenERP
Holger Brunn
Therp BV
03 july 2013

2. Intro PostgreSQL full text search capabilities OpenObject FTS
Outline
1 Intro
2 PostgreSQL full text search capabilities
3 OpenObject FTS
Holger Brunn Therp BV
Using full text search in OpenERP

3. Intro PostgreSQL full text search capabilities OpenObject FTS
Searching content of text fields is
very slow for large amounts of data
finds only fields containing the exact string
you filled in
restricted to just this string
Full text search gives us
fast searches
lexeme level fuzzy search
logical operators
Holger Brunn Therp BV
Using full text search in OpenERP

4. Intro PostgreSQL full text search capabilities OpenObject FTS
Demo
imported part of the project gutenberg database as
documents
10.000 books, every one as its own document,
resulting in a lot of text in different languages
doing several searches there with standard OpenERP
search makes you a serious coffee addict
Holger Brunn Therp BV
Using full text search in OpenERP

5. Intro PostgreSQL full text search capabilities OpenObject FTS
Basics
documents are the lump of text you are searching through
tokens are parts of a document: words, but also i.e. urls,
numbers
lexemes are roughly the core of a word: ‘searching’,
‘search’, ‘searched’ are all derived from the
lexeme SEARCH
stop words are words that don’t make sense to store and are
to be ignored, think of ‘the’
dictionaries define lexemes, synonyms and stop words of a
language
more info http://www.postgresql.org/docs/current/
static/textsearch.html
Holger Brunn Therp BV
Using full text search in OpenERP

6. Intro PostgreSQL full text search capabilities OpenObject FTS
Basics II
tsvector is the datatype postgres uses to represent a
tokenized, lexemized text
tsquery is the datatype to represent a query against a
tsvector, with operators
‘&’ and
‘|’ or
‘!’ not
‘(’, ‘)’ brackets
:* substring search
@@ is the operator to match a tsquery against a
tsvector
Holger Brunn Therp BV
Using full text search in OpenERP

7. Intro PostgreSQL full text search capabilities OpenObject FTS
Examples
select to tsvector(’english’, ’This is an english text. Pay
attention how it is broken up’);
select to tsvector(’french’, ’This is an english text. Pay
attention how it is broken up’);
select plainto tsquery(’english’, ’this’);
select plainto tsquery(’french’, ’this’);
Holger Brunn Therp BV
Using full text search in OpenERP

8. Intro PostgreSQL full text search capabilities OpenObject FTS
Untouched topics
ranking
highlighting
proximity search
parser tuning
dictionary tuning
configurations
index tuning
Holger Brunn Therp BV
Using full text search in OpenERP

9. Intro PostgreSQL full text search capabilities OpenObject FTS
Full text search support for OpenERP
is available on launchpad:
https://launchpad.net/openobject-fts
is a collection of addons that enable full text search for
several models
takes care of creating appropriate indexes and triggers for
underlying database tables and columns
can be misused to provide a unified search over several
models
Holger Brunn Therp BV
Using full text search in OpenERP

10. Intro PostgreSQL full text search capabilities OpenObject FTS
Installation / Configuration
on installation, a cronjob is created to create the initial
index
watch your logs, there is an info on when index creation
starts and when it ends
only then, this model is searchable via full text search
this can take hours if you have enough data
the initial configuration is ‘simple’, this won’t get you
good results
set the parameter fts base tsconfig to english, french, etc.
to change that
then go to Configuration / Fulltextsearch configuration to
recreate the index
Holger Brunn Therp BV
Using full text search in OpenERP

11. Intro PostgreSQL full text search capabilities OpenObject FTS
Hacking it
depend on fts base
inherit fts base
set properties model and indexed column
the class fts base is extensively documented, read that
note that this are not ORM models
to change the combined search view, inherit fts.proxy’s
search view
Holger Brunn Therp BV
Using full text search in OpenERP

12. Intro PostgreSQL full text search capabilities OpenObject FTS
Contributions welcome
hook into search methods of full text indexed models to
also use the full text search in their own searches
hook into fts.proxy’s treeview and override edit/form
actions to get rid of the awkward extra buttons currently
used
hack 7.0’s search widget to propose lexemes as you type
dig deeper into caching mechanisms for smoother scrolling
query available configurations from postgres and offer
then in the confguration wizard
Holger Brunn Therp BV
Using full text search in OpenERP