Web Crawler with Email Extractor and Image Extractor

1. ABHINAV GUPTA (9910103413)
NITISH PARIKH (9910103407)
RISHABH SINGH (9910103544)
Web Crawler with Email Extractor
and Image Extractor

2. Web Crawler
 Web Crawler is a program that, given one or more seed URLs, downloads the web
pages associated with these URLs, extracts any hyperlinks contained in them, and
recursively continues to download the web pages identified by these hyperlinks. Web
crawlers are an important component of web search engines, where they are used to
collect the corpus of web pages indexed by the search engine
 Web Crawler gives the list of links where the specific word is present in a particular
Website and its pages. A Web crawler is an Internet bot that systematically browses
the World Wide Wide, typically for the purpose of Web indexing. A Web crawler may
also be called a Web spider, an ant, an automatic indexer.

3. How Web Crawler Works ?
 A Web crawler starts with a list of URLs to visit, called the seeds. As the crawler visits
these URLs, it identifies all the hyperlinks in the page and adds them to the list of
URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited
according to a set of policies.

4. Email Extractor
 Email extracting is the process of obtaining lists of email addresses using various
methods for use in bulk email or other. You may need to harvest email addresses when
you are conducting a marketing campaign, or when you want to find out something, or
send an email to a massive, but targeted, audience. This program is a spider that will
detect emails in web sites, through search engines, or just from a file saved on your
computer.

5. How Email Extractor Works ?

6. Software Used
 Eclipse:
In computer programming, Eclipse is a multi-language Integrated development
environment (IDE) comprising a base workspace and an extensible plug-in system
for customizing the environment. It is written mostly in Java. It can be used to
develop applications in Java and, by means of various plug-ins, other programming
languages including C, C++, JavaScript, PHP, Python. Development environments
include the Eclipse Java development tools (JDT) for Java, Eclipse CDT for
C/C++ and Eclipse PDT for PHP, among others.

7. Screenshots

11. Image Extractor
 Interest in the potential of digital images has increased enormously
over the last few years, fuelled at least in part by the rapid growth of
imaging on the World-Wide Web. Users in many professional fields are
exploiting the opportunities offered by the ability to access and
manipulate remotely-stored images in all kinds of new and exciting
ways. However, they are also discovering that the process of locating a
desired image in a large and varied collection can be a source of
considerable .
 frustration. The problems of image retrieval are becoming widely
recognized, and the search for solutions an increasingly active area for
research and development.

12. PROBLEM STATEMENT
 Since the last decade, Features-Based Interactive Image Retrieval was a
hot topic research. The computational complexity and the retrieval
accuracy are the main problems that FBIIR systems have to avoid.
 The aim of this project is to research and implement the potential for
using Features-based Image Retrieval methods for querying large-scale
image databases. More specifically, the project seeks to identify image
features that serve as accurate, yet low dimensional compact,
descriptors. In extension it should find methods that have general good
retrieval performance that are well suited for scaling. That means that
they must be efficient not only in terms of query time but also
extraction complexity and storage demands.

13. OVERALL ARCHITECTURE WITH COMPONENT DESCRIPTION
ARCHITECTURAL STRATEGIES

14. Color Histogram

Color is the most widely used feature because it is the
intuitive feature compared with other features and easy
to extract from image. However, CBIR system based on
color feature often result in disappointment, because it
uses global color feature which cannot capture color
distributions or textures within the image sometimes.
To improve the preferment of the color extraction
FBIIRS divides color histogram feature into global and
local color extraction. Local color histogram can give
some sort of spatial information, however the cons with
that it use very large feature vectors.

15. Geometric Moments
 This feature use only one value for the feature vector,
however, the performance of current implementation
isn’t well scaled, [2] which means when the image
size become large, it takes very long time to
computer the feature vector. The pros of using this
feature combine with other features such co-
occurrence, which can provide a better result to user.

16. Average RGB
 The objective of using this feature is to filter out
images with larger distance at first stage when
multiple feature queries involves. Another reason of
choosing this feature, because it uses a small number
data to represents the feature vector and it also use
less computation compare to others. However, the
accuracies of query result could be significantly
impact if this feature isn’t combined with other
features.

17. Color Moments
 This feature has very reasonable size of feature
vector, and the computation isn’t expensive, [4]
Colour Moments are measures that can be
differentiate images based on their feature of colour,
however, the basic of colour moments lays in the
assumption that the distribution of colour in an
image can be interpreted as a probability
distribution. On pros of it is its skewness can be used
to measure of the degree of asymmetry in the
distribution.

18. Persistence Module
 This module (component) takes care the transaction
and persistent of the image features with database. It
provides a clear-cut programming interface to other
components. Consequently, other module in the
system will effortlessly deal with database (such as
Feature Extraction and Query module).
 FeatureInfo Id Feature name file path vector

19. Image Represenation in Java

20. Requirements
 Software Items
 Window 7/8/8.1 Stability
 Mac Stability
 Java
 Java Runtime Environment & Development Kit
 Netbeans

 Hardware Items
 Colored Screen
 Good Screen Resolution

21. ScreenShots

22. ScreenShots

23. ScreenShots

25. LIMITATION OF THE SOLUTION
 As the results we see that -:
 „h System is not capable of searching the colored image on
the bases of the sketch of that image.
 „h If the database is very large (like lacs of images) then it
will take lot of time in extracting features of each and every
image.
 „h System sometimes hang due to loss of connection to
database.
 „h If single algorithm is used instead of multiple algorithms
the accuracy will come out to be poor.

26. FINDINGS
 1.Index more efficient
 This system index 1000 sample images in 5 minutes whereas other systems like QBIC
almost took 10 minutes for indexing same number of images.
 2. Statable
 This system more statable as compared to other existing systems.
 3. Reusable
 Compare with other systems, they provide limited sample image, query from limited
image database, but this system can query any sample image, can index any image folder,
more reusable
 4. Compare with other systems, this provides more searching features.
 5. Feedback query
 This system provides User feedback Query, user can research from result, increase the
accuracy.

27. CONCLUSION
 The extent to which FBIR technology is currently in routine use is clearly still very
limited. In particular, FBIR technology has so far had little impact on the more general
applications of image searching, such as journalism or home entertainment. Only in very
specialist areas such as crime prevention has FBIR technology been adopted to any
significant extent. This is no coincidence – while the problems of image retrieval in a
general context have not yet been satisfactorily solved, the well-known artificial
intelligence principle of exploiting natural constraints has been successfully adopted by
system designers working within restricted domains where shape, color or texture
features play an important part in retrieval. FBIR at present is still very much a research
topic. The technology is exciting but immature, and few operational image archives have
yet shown any serious interest in adoption. The crucial question that this report attempts
to answer is whether FBIR will turn out to be a flash in the pan, or the wave of the future.
It is not as effective as some of its more ardent enthusiasts claim – but it is a lot better
than many of its critics allow, and its capabilities are improving all the time. Most current
keyword-based image retrieval systems leave a great deal to be desired.

28. FUTURE WORK
 The success of proved both that image retrieval application can be
implemented in Java programming language with high performance
and Feature-based image retrieval could be a feasible technology in the
future. Nevertheless, the project is at basic level thus, many great
images retrieval techniques hasn’t implemented, yet. Here is a list of
area that can be improved in the future.
 Adopting a better cache technique for result image caching, so that
the latency of display images will be minimized, as well as using lesser
computation and resources.
 Implementing a superior ranking algorithm for result image ranking
 Getting more visual features extraction module (for example, BEMD
filtering for Sketch Detection)

29. Thank You !
Submitted by:
Abhinav Gupta 9910103414
Nitish Parikh 9910103407
Rishabh Singh 9910103544
B.Tech, Cse, 4th year
JIIT-128