Introducing Apache Lucene with two demos for analyzing similarity among android applications from Android App market. In this case, I used 20k apps for demo.

1. Apache Lucene
& two demo with PyLucen

2. Lucene family
●
●
●
●
Lucene Core
○ flagship sub-project
○ provides Java-based indexing and search technology, as well as
spellchecking, hit highlighting and advanced analysis/tokenization
capabilities.
Solr
○ a high performance search server built using Lucene Core
○ with XML/HTTP and JSON/Python/Ruby APIs, hit highlighting,
faceted search, caching, replication, and a web admin interface.
Open Relevance Project
○ a subproject with the aim of collecting and distributing free
materials for relevance testing and performance.
PyLucene
○ a Python port of the Core project.

3. Content model of Lucene
●
●
●
Document
○ Lucene’s atomic unit of indexing and searching.
○ It’s a container that holds one or more fields containing the real
content.
Flexible schema
○ Unlike a DB, Lucene has no notation of a fixed global schema.
○ Each document you add to the index is a blank slate and can be
completely different from the document before it.
○ Lucene requires you to flatten, or denormalize, your content when
you index it.
Denormalization
○ a Lucene documents are flat. (no recursion and no joins, just
flat)

4. The indexing process
●
●
●
Extracting text and creating the document
○ You must extract plain text from data
○ Store plain text to each field
○ Tika helps you to convert data into plain
text (e.g. PDF, Excel)
Analysis
○ Lucene first analyzes the text, a process
that splits the textual data into a
stream of tokens, and performs a
number of optional operations on them.
Adding to the index
○ “Which documents contain word X?”

5. IndexSearcher
●
●
●
Creating an IndexSearcher
Performing searches
Working with TopDoc

6. First Demo
a brief demo with PyLucen

7. Simple index and search
https://github.com/sangheestyle/bisonlucene

8. Second Demo
a brief demo with PyLucen

9. Understanding Lucene scoring
(quoted from Lucene in Action: Chap 3.3)
‘This score computes how similar the document is to the query, with higher scores reflecting
stronger similarity and thus stronger matches.’
‘The score is computed for each document (d) matching each term (t) in q query (q).’
--
-‘The larger the similarity score, the better the match of the document to the query.’

10. Score
3.12660646439
compared to itself
title: Neon Sign Dodol Theme
2.25216460228
compared to another
title: Meditation at Noon dodol
theme

11. Why? Explanation by explain() method based on
Lucene scoring
description
explanation
Be seduced by the neon sign that shines in the dark.â ¥Apply
the theme to transform your background screen, icon and
launcher widget! :D***How to apply a theme*** - Install Dodol
Launcher (if you haven't yet). - Press the home button and
set the Dodol Launcher as the default home launcher by
selecting [Use as default value for this task] or [Always]. Open the launcher menu by swiping the home screen
upwards, select the [Theme] menu, then select and apply the
theme of your choice.***Precaution*** - Is available only in
Android ver. 4.0.3 and above (ICS, Jellybean) - Some
functions cannot be applied in certain devices.***Customer
Support*** - http://m.help.naver.com/serviceMain.nhn?
falias=mo_launcher_app&type=faq - http://blog.naver.
com/dodolhome - http://www.facebook.
com/dodolhome***Why Dodol Launcher is so special*** Cute, simple, vibrant, sophisticated, cuddly and warm themes
are continuously updated - The default widget offers
convenient functions, including quick switch and memory
cleaner - Apps/widget can be used easily with the dock and
alert shortcut list - Offers functions that can be used to
decorate fonts/ringtones/keyboards - Offers detailed functions
for screen rotation/individual icons/folders etc... - Supports
functions in a convenient and stable manner by copying the
home screen and creating backups of settings
3.1266062 = (MATCH) sum of:
0.15624778 = (MATCH) weight(description:launcher in 2699)
[DefaultSimilarity], result of:
0.15624778 = score(doc=2699,freq=6.0 = termFreq=6.0
), product of:
0.15802452 = queryWeight, product of:
5.1668243 = idf(docFreq=340, maxDocs=21998)
0.030584458 = queryNorm
0.98875654 = fieldWeight in 2699, product of:
2.4494898 = tf(freq=6.0), with freq of:
6.0 = termFreq=6.0
5.1668243 = idf(docFreq=340, maxDocs=21998)
0.078125 = fieldNorm(doc=2699)
0.1313615 = (MATCH) weight(description:functions in 2699)
[DefaultSimilarity], result of:
0.1313615 = score(doc=2699,freq=5.0 = termFreq=5.0
), product of:
0.15165158 = queryWeight, product of:
4.958452 = idf(docFreq=419, maxDocs=21998)
0.030584458 = queryNorm
0.866206 = fieldWeight in 2699, product of:
2.236068 = tf(freq=5.0), with freq of:
...

12. 5.76816177368
compared to itself
title: Robert Downey Junior
News
1.05290555954
compared to another
title: Rehab News

13. Check Scores with bisonlucene
https://github.com/sangheestyle/bisonlucene

14. Check Scores with bisonlucene
https://github.com/sangheestyle/bisonlucene

15. Indexing some fields by lucene
{
"title": "The Economist",
"playStoreURL": "https://play.google.com/store/apps/details?id=uk.co.economist",
"category": "News & Magazines",
"price": "Free",
"datePublished": "August 29, 2013",
"version": "Varies with device",
"operatingSystems": "Varies with device",
"ratingsCount": "5789",
"rating": "3.3852134",
"contentRating": "Low Maturity",
"creator": "The Economist Newspaper Limited",
"creatorURL": "https://play.google.com/store/apps/details?id=uk.co.economist",
"extendedInfo": {
"installSize": "Varies with device",
"downloadsCount": "1,000,000+",
"downloadsCountText": "1,000,000 - 5,000,000",
...

16. 4.99658298492
Case 1:HandWriting
Description has been changed
Finger Letter :
https://play.google.com/store/apps/details?id=com.demoros.finger
Kim Hongsik
ì
ì ¼ë¡ ê¸ ì ¨ë¥¼ ì ¨ì
문ì , í ¸ì
í °,í
ì ´ì ¤ ë¶ ë ±ì ¼ë¡ ë³´ë ´ê±°ë í ì ¼ë¡ ì ì ¥í ´ì
ë
¸í ¸ì² ë ¼ ì ¬ì ©í ì
ì ì µë ë ¤.~
..
Hey Stop! - Screen Freeze
https://play.google.com/store/apps/details?id=com.cluster.screenfreeze
Inerve
ì ¬ë¯¸ì
ë
ê² ì ì
ê² ì
ë °ì ´í °ë¥¼ ë° ë ë ° ê° ì
기 í ë©´ì ´ êº¼ì ¸ì
ì¤ ë ¨ë ê±°ë
í °ì¹ ë¥¼ ì 못í
´ì
ì¤ ë ¨ë
ë
ê²½ì °!ì§ ì¦ ë
ì
¨ì£ ? ã
ã
ì ´ ì ±ì
ê·¸ë ° ì ì¸ ì ì ´ì
ì
ë
ì ´ì
ë° ì
í ì§ ëª»í
ê² í 기ì í
ì ¬ ë§ ë ¤ì ´ì§ ì ±ì
ë ë ¤!ì ´ ì ±ì
ì ¤í í
ì
ë§ ì
í ´ë í °ì
물리ì ë² í ¼ì
ì ì ¸í
모ë í °ì¹ ì ¤í ¬ë¦°ì
í °ì¹ í ì
ì
ê² ë§ ë -ë ë ¤.ê·¸ë 기 ë 문ì
ê²°ì½ ì§ í
ì¤ ì ´ë
ì ì
ì ´ êº¼ì§ ì§ ì
ê² ë°©ì§ í ì
ì ì µë ë ¤.ì ê¸ ì
í ´ì í 기 ì í ´ì ë
'ë ¤ë¡ ê° ê¸° ë² í ¼'ì
0.15ì´ ì
ì
ë¹ ë¥´ê² ë ë² ì °í
í ì
ì ¼í ©ë ë ¤.í ì§ ë§ ê· ì°®ì ¼ì ë¶ ë ¤ì
ê·¸ë ¥ í ë² í ¼ í ë² ê¾¹ ë ë ¬ì£¼ì ê¸°ë§ í ´ë
ë 구ì
ã
ã
(ë ¤ë§ ì ´ ê²½ì ° ë ¤ì
ì ê¸ í
ì
ë ì
ë
ì¤ ë³µì ¼ë¡ ë
ë² ì ê¸ ë ©ë ë ¤)í ì
ì ½ê² ì ¤í í ì
ì
ê² ì ±ì
ì ¤í í ì
ì í
í
ì
ì¤ ì
ì ë¦¼ì ´ ë
ì
ì§ ë ë ¤.ê·¸ ì 림ì
ë 르ì ë©´ ì ½ê² ë ¤ì
ì ê¸ ì
í ¬ì
ì
ì µë ë ¤.ê·¸ë ¼ ì ¬ë ¬ë¶ ì
0.15ì´ ì ê° ë ½ ë ¬ì ¬ë¥¼ 기ë í ê² ì µë ë ¤!(?)----ê° ë°
ì
ì °ë ½ì² :...

18. Case 2: Company names are just little bit different or
changed. (Same company)
4.37417459488
Kids Swap & Turn Christmas
https://play.google.com/store/apps/details?id=com.divmob.kidsswapandturnnoelgl2hang
divmobvn
Kids Swap & Turn is a kind of jigsaw, which develop your kids imagination, patience and aesthetic sensitivity. It requires children to be
persistent and focused, as well as able to plan their activities.Features:- 2 modes: Swap & Turn- 3 difficulty levels
..
Kids Swap & Turn #3
https://play.google.com/store/apps/details?id=com.divmob.kidsswapandturnhang
divmob
Kids Swap & Turn is a kind of jigsaw, which develop your kids imagination, patience and aesthetic sensitivity. It requires children to be
persistent and focused, as well as able to plan their activities.Features:- 2 modes: Swap & Turn- 3 difficulty levels- 40 cute pictures

20. Case 3-1: Different companies but almost same description
4.15391206741
York College of PA Crib Sheet
https://play.google.com/store/apps/details?id=com.cc.cribsheet.ycp
YCP Mobile
Created by York College of Pennsylvania for our alumni, "Crib Sheet" is your way to stay current with campus news, sports highlights, alumni
events and benefits, and much more. The app is also your "real world" crib sheet on topics like money, housing, etiquette, health insurance, and
others.
..
Calvin College Crib Sheet
https://play.google.com/store/apps/details?id=com.cc.cribsheet.calvin
Calvin Alumni Association
Created by the Calvin Alumni Association for Calvin College alumni, "Crib Sheet" is your way to stay current with campus news, sports highlights,
alumni events and benefits and much more. The app is also your "real world" crib sheet on topics like money, housing, etiquette, health
insurance and others.

22. Case 3-2: Different companies but almost same description
3.36474967003
ISAPS 21st Annual Congress
https://play.google.com/store/apps/details?id=com.eventpilot.isaps12
ATIV Software
EventPilot® conference app is your full featured guide to manage your conference attendance. App features include: ⠢ Native app: No wifi connection
required to access the conference program, schedule or animated maps. â ¢ Now: Stay informed about hot issues, event program changes, your upcoming
sessions and organizer messages. â ¢ Program: Browse the entire event program to build your personal schedule, bookmark sessions or speakers, or
access session handouts as available. â ¢ Take notes and email them as part of your trip report for reference. â ¢ Exhibitors, Maps, related conference
info and much more.
..
ABA Section of Intl Law
https://play.google.com/store/apps/details?id=com.eventpilot.abaenterprise
American Bar Association
EventPilot® conference app is your full featured guide to manage your ABA SIL event attendance. App features include: ⠢ Native app: No wifi
connection required to access the conference program, schedule or animated maps. â ¢ Now: Stay informed about hot issues, event program changes,
your upcoming sessions and organizer messages. â ¢ Program: Browse the entire event program to build your personal schedule, bookmark sessions or
speakers, or access session handouts as available.
â ¢ Take notes and email them as part of your trip report for reference. â ¢ Exhibitors, Maps,
related conference info and much more.

24. Case 4: One of similar apps was removed
3.36554980278
Funny Pictures for Whatsapp
https://play.google.com/store/apps/details?id=es.smartphonereligion.humorgrafico
Smartphone Religion
Recopilación de las fotos más divertidas que nos han mandado por whatsapp. Las puedes reenviar desde la aplicación de una manera muy rápida.
Aprovecha esta app gratuita para divertirte con tus amigos.y no quedarte en casa jugando a pou on al pouf pedos.Hemos recopilado las mejores fotos de
frikis del mundo, las personas más raras, el humor amarillo o verde, la guasa de animales⠦ todas las imágenes que nos han sorprendido las hemos
puesto aquÃ- en esta aplicación de humor gráfico para whatspp al estilo desmotivaciones.Fotos divertidas para enviar por whatsapp gratis. Humor y
bromas para que te puedas reir con tus amigos al enviarles estas fotos graciosas. Puedes también guardarlos para ponértelos como estado en
facebookLa mejor aplicación gratuita de fotos y humor gráfico para enviarlo a tus amigos por Facebook, tuenti, line o por la conocida whatsapp. Te reirás
con cada una de las fotos e imágenes divertidas que aquÃ- hemos recogido.Ola k Ase, si tienes ganas de fiesta, puedes empezar a reÃ-rte con los chistes
gráficos que aquÃ- te hemos recopilado. ...
..
Funny Jokes 2
https://play.google.com/store/apps/details?id=com.evensoftware.fwkimages.humor2
Lights
Recopilación de las fotos más divertidas en español e inglés que nos han mandado por whatsapp. Las puedes reenviar desde la aplicación de una
manera muy rápida a tus amigos. Aprovecha esta app gratuita para divertirte con tus amigos.Hemos recopilado las mejores fotos e imágenes que nos han
enviado por whatsapp para que las disfrutes con tus amigos, familiares o tu suegra⠦ las personas más raras, el humor amarillo o verde, una guasa de
fotos que puedes compartir tanto por whatsapp, como como estado de tuenti o facebook.Si te gusta la aplicación, mira nuestras otras aplicaciones de humor
gráfico para whatsapp y te morirás de la risa. Ve a nuestra página y descargate todas nuestras aplicaciones. Somos especialistas en hacer reir y sonreir al
estilo desmotivaciones o como las frases de José Mota o el ola k ase.La mejor aplicación gratuita de fotos y humor gráfico para enviarlo a tus amigos por
Facebook, tuenti, line o por la conocida whatsapp. Te reirás con cada una de las fotos e imágenes divertidas que aquÃ- hemos recogido. Fotos divertidas
para enviar por whatsapp gratis. Humor y bromas para que te puedas reir con tus amigos al enviarles estas fotos graciosas.Aprovecha tus últimos
momentos del whatsapp gratuito y envÃ-ale a tus grupos de amigos estas divertidas fotos desde cualquiera de las plataformas de comunicación más
extendidas como tuenti, ...

26. Case 5: Almost same app but different user opinions
3.37851405144
Sea Turtle Live Wallpaper
https://play.google.com/store/apps/details?id=com.epestov.turtle1
Booom Soft
This is high quality animated live wallpaper of the underwater world! Sea turtle is swimming just under the glass of your smartphone, very realistic and really
amazing!Enjoy!Installation instructions:Home -> Menu -> Wallpapers -> Live WallpapersScroll down the list, find out the wallpaper and setup it.Note: It is live
wallpaper so you can't open the app, you will need to follow the instruction above in order to set the wallpaper. Also slower/older devices (shipped with Android
OS below 2.1) can't run it too.
..
Underwater World
https://play.google.com/store/apps/details?id=com.epestov.aquadream
N&T Group
This is high quality animated live wallpaper of the underwater world! Fishes and sea turtle are swimming just under the glass of your smartphone!Enjoy!
Installation instructions:Home -> Menu -> Wallpapers -> Live WallpapersScroll down the list, find out the wallpaper and setup it.Note: It is live wallpaper so you
can't open the app, you will need to follow the instruction above in order to set the wallpaper. Also slower/older devices (shipped with Android OS below 2.1)
can't run it too.

31. Reference
●
●
Apache Lucene: http://lucene.apache.org
Lucene in action 2nd