Thesis Licenseplaterecognitionincomplexscenes Fulldocument

LICE NSE PLATE RE COG NITIO N IN COr.ifP LEX SCENES

BY

DHAWAL S WAZALV,TAR

Submit t ed in partial fulfillment of t he

requirement s for t he degree of

Master of Science in Electrical Engineering

in t he Gra duate College of t he

Illinois Institute of Technology

Approved ----("Lf-f/;Io.q---;4/----/;4~'---

Chicago, Illinois
Decemb er 2011

ACKNOWLEDGMENT

I would like to sincerely thank my advisors Dr Erdal Oruklu and Dr Jafar
Saniie for their constant encouragement and support throughout my thesis work.

They always ensured that my research work is heading in proper direction and always

guided me in planning each stage of my research work. I would also like to thank Dr

Tricha Anjali for her encouragement and insightful comments during thesis review

meeting.

I would like to thank my family and friends for their constant support through-
out my course.

iii

TABLE OF CONTENTS

Page

ACKNOWLEDGEMENT . . . . . . . . . . . . . . . . . . . . . . . . . iii

LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi

LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii

LIST OF SYMBOLS . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x

CHAPTER
1. INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . 1

1.1. Typical license plate recognition system . . . . . . . . . . 2
1.2. Problem statement . . . . . . . . . . . . . . . . . . . . 3
1.3. Thesis structure . . . . . . . . . . . . . . . . . . . . . 3

2. LITERATURE REVIEW AND PROPOSED ARCHITECTURE 5

2.1. License plate localization . . . . . . . . . . . . . . . . . 5
2.2. Character segmentation . . . . . . . . . . . . . . . . . . 7
2.3. Character recognition . . . . . . . . . . . . . . . . . . . 8
2.4. Proposed license plate recognition system . . . . . . . . . 10

3. LICENSE PLATE LOCALIZATION . . . . . . . . . . . . . . 13

3.1. Selection of edge detection method . . . . . . . . . . . . 13
3.2. Morphological dilation operation . . . . . . . . . . . . . 16
3.3. Region growing segmentation . . . . . . . . . . . . . . . 19
3.4. Criterion for license plate detection . . . . . . . . . . . . 22

4. CHARACTER SEGMENTATION . . . . . . . . . . . . . . . 28

4.1. Image preprocessing . . . . . . . . . . . . . . . . . . . 28
4.2. Threshold operation . . . . . . . . . . . . . . . . . . . 32
4.3. Morphological erosion operation . . . . . . . . . . . . . . 35
4.4. Region growing segmentation . . . . . . . . . . . . . . . 35

5. FEATURE EXTRACTION AND CHARACTER RECOGNITION 37

5.1. Feature extraction . . . . . . . . . . . . . . . . . . . . 37

iv

5.2. Artiﬁcial neural networks . . . . . . . . . . . . . . . . . 40

6. IMPLEMENTATION IN MATLAB AND OPENCV . . . . . . 47

6.1. MATLAB implementation . . . . . . . . . . . . . . . . 47
6.2. OpenCV . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.3. Comparison of MATLAB and OpenCV . . . . . . . . . . 48

7. EXPERIMENTAL RESULTS . . . . . . . . . . . . . . . . . 50

7.1. Database for testing of license plate recognition . . . . . . 50
7.2. Results . . . . . . . . . . . . . . . . . . . . . . . . . . 51

8. CONCLUSION AND FUTURE WORK . . . . . . . . . . . . 54

8.1. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 54
8.2. Future recommendation . . . . . . . . . . . . . . . . . . 55

APPENDIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

A. MATLAB CODE FOR LICENSE PLATE RECOGNITION . . . 56

B. OPENCV CODE FOR LICENSE PLATE LOCALIZATION . . 66

BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

v

LIST OF TABLES

Table Page

6.1 Comparison of MATLAB and OpenCV execution time for license
plate detection of a single frame . . . . . . . . . . . . . . . . . 49

7.1 Summary of license plate localization and detection results . . . . . 51

7.2 Number of frames detected per vehicle . . . . . . . . . . . . . . 52

vi

LIST OF FIGURES

Figure Page

1.1 A typical license plate recognition system . . . . . . . . . . . . 2

2.1 Proposed license plate recognition system . . . . . . . . . . . . 11

3.1 (a) Sample image with distinct contrast; (b) Prewitt edge detection;
(c) Sobel edge detection . . . . . . . . . . . . . . . . . . . . . 15

3.2 (a) Sample image with partial contrast; (b) Prewitt edge detection;
(c) Sobel edge detection . . . . . . . . . . . . . . . . . . . . . 17

3.3 Wavelet Mexican hat . . . . . . . . . . . . . . . . . . . . . . 18

3.4 Mexican hat edge detection output for partial contrast case . . . . 18

3.5 (a) Blur image; (b) Edge detection output; (c) Plate detection with-
out dilation; (d) Dilation output; (e) Plate detection with dilation 20

3.6 Pixel connectivity types . . . . . . . . . . . . . . . . . . . . . 21

3.7 Region growing segmentation output for different seed points (a)
Sample image; (b) Seed point=200; (c) Seed point=230; (d) Seed
point=245 . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.8 Region growing segmentation output with different stopping rule
(a) Original image ; (b) Stopping rule=25; (c) Stopping rule=55;
(d) Stopping rule=75 . . . . . . . . . . . . . . . . . . . . . . 24

3.9 Region growing for license plate detection (a) Output after dilation;
(b) Region growing output . . . . . . . . . . . . . . . . . . . 25

3.10 (a) Failure case for Criterion 2; (b) Criterion 3 output for part (a)
case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.1 Design flow for character segmentation . . . . . . . . . . . . . . 29

4.2 Various forms of noise in an image (a) Original image; (b) Gaussian
Noise; (c) Salt pepper noise; (d) Speckle noise . . . . . . . . . . 31

4.3 (a) Original license plate image; (b) Enhanced version after image
preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.4 (a) Sample image; (b) Global thresholding; (c) Local thresholding . 34

4.5 Steps in character segmentation (a) Image with shadow; (b) Local
thresholding; (c) Erosion operation; (d) Segmented characters . . . 36

vii

5.1 Diﬀerent versions of character ’G’ . . . . . . . . . . . . . . . . 38

5.2 Basic feed forward model, Perceptron . . . . . . . . . . . . . . 42

5.3 Three layer Perceptron . . . . . . . . . . . . . . . . . . . . . 43

7.1 Types of images included in license plate database (a) Type 1; (b)
Type 2; (c) Type 3; (d) Type 4; (e) Type 5 . . . . . . . . . . . . 53

viii

LIST OF SYMBOLS

Symbol Definition

σ Standard deviation

E Euler number of binary image

Xcoef f icient Reflection coefficient about X axis

Ycoef f icient Reflection coefficient about Y axis

ix

ABSTRACT

License plate recognition is considered to be one of the fastest growing tech-
nologies in the field of surveillance and control. In this project, we present a new

design flow for robust license plate localization and recognition. The algorithm con-

sists of three stages: i) license plate localization ii) character segmentation and iii)

feature extraction and character recognition. The algorithm uses Mexican hat opera-

tor for edge detection and Euler number of a binary image for identifying the license

plate region. A pre-processing step using median filter and contrast enhancement is
employed to improve the character segmentation performance in case of low resolution

and blur images. A unique feature vector comprised of region properties, projection

data and reflection symmetry coefficient has been proposed. Back propagation artifi-

cial neural network classifier has been used to train and test the neural network based

on the extracted feature. A thorough testing of algorithm is performed on a database

with varying test cases in terms of illumination and different plate conditions. The
results are encouraging with success rate of 98.10% for license plate localization and

97.05% for character recognition.

x

1

CHAPTER 1

INTRODUCTION

License plate recognition is one of the most rapidly evolving technologies in
the field of video based surveillance and control. It is sometimes also referred to as

Automatic number plate recognition (ANPR) or automatic vehicle detection system.

As the name implies, it first detects the license plate from the input frame and

recognizes the vehicle as well its owner based on its license plate information. Since

its first use in 1976 by UK Police, this technology has gone through a lot of changes

and is now being extensively used in law enforcement field.

One of the key applications has been in access control for gated communities

and other such restricted access locations. In such places, license plate information of

all the authorized vehicles is stored in a database and only these vehicles are allowed

to enter the premises. This can be extremely useful in safeguarding intelligence agen-

cies and other such key government buildings from any external threat. Automated

traffic management is another field where license plate recognition has contributed
immensely. It is widely used for toll collection, speed control and accident scene

reconstruction.

The success for fixed camera applications encouraged authorities to use this

technology for mobile camera based surveillance. The use for mobile applications had

its own challenges especially in terms of power efficiency since vehicle battery was used

as a power source. Another challenge was processor speed as all the systems need
to be real time. In recent years, the semiconductor technology has grown immensely

which has contributed in improving power performance of electronic equipments as

well as reducing its size. This has helped in developing portable surveillance systems

which can be easily mounted on vehicle top. This has proven to be an important tool

for police officials in solving traditional criminal investigations. Recently, a Maryland

2

state police officer traveling along I-70 nabbed alleged drug dealer with the help of

license plate technology.

1.1 Typical license plate recognition system

A generic representation of license plate recognition system is shown in Figure

1.1. It mainly has following key stages.

Op cal Character License Plate
Image Frame
Recogni on Informa on
Acquisi on Processing
Engine Database

Figure 1.1. A typical license plate recognition system

1.1.1 Image acquisition. Surveillance cameras are used to capture real time video

or images of the front or rear of the vehicle with its license plate. Nowadays, mostly

videos are captured and then a frame grabber is used as an interface to make the

required frame available for processing to the software. Cameras used are specially

designed which have important features like faster frame rates, weatherproof and
better power efficiency. For better performance in low vision or night times, infra-

red (IR) illumination is used. A lot of companies constantly work on developing high

end intelligent cameras for better image acquisition. For example PIPS technology,

one of the key vendors of license plate recognition systems, uses a dual lens camera

with triple flash technology to produce highest quality image.

1.1.2 Frame processing. After acquiring the input frame of vehicle with its license
plate, a series of image enhancement and processing steps are performed to detect

the license plate region. The complexity of these steps depends upon the application

requirements. Once we obtain the license plate, we need to segment its individual

characters which then can be sent to optical character recognition engine.

1.1.3 Optical character recognition engine. For recognizing the segmented

3

characters, we extract some of its key features and use statistical classifiers to differ-

entiate and therein recognize the character information.

1.1.4 License plate database. This is also an important part of overall system

wherein we store the output obtained from OCR engine in a database. This database

information then can be used by police officials or other law enforcement authorities

to use for various applications like stolen vehicle identification, speed checks etc.

1.2 Problem statement

Although many license plate recognition systems already are in market and

a lot of research has been done in this field, all these systems perform efficiently
only in a controlled environment. For example, in case of license plate detection,

most of the systems claim to have an accuracy of above 95 %. However, the actual

system performance was not as per the expectations when dealing with practical cases

like presence of shadow/dirt on license plates, sensitivity to license plate dimensions

and many such other factors. Our emphasis in this project has been to come up

with solutions which will help to improve performance of license plate detection and
recognition in dealing with real time practical scenarios.

The performance analysis of license plate recognition systems is also an issue

since there is no common benchmark. In this thesis, system performance is evaluated

on a real time demo video taken from City Sync license plate recognition camera.

This video has all the practical scenarios with varied license plate sizes, night/twilight

conditions and dirty/distorted license plate images. Our focus has been on dealing
with performance issues related to fixed camera applications; however the algorithms

can also be extended for mobile camera surveillance systems with some modifications.

1.3 Thesis structure

Chapter 2 gives an insight on the previous research work done in the license

4

plate recognition ﬁeld. In Chapter 3, a design ﬂow for robust license plate recognition

has been presented and performance of license plate localization for real time cases
is analyzed. The character segmentation aspect and all the associated steps with it

are explained in Chapter 4. The challenging issue of feature extraction and character

recognition is explored in Chapter 5. Software implementation part in MATLAB

and OpenCV is explained in Chapter 6. Performance evaluation of license plate

localization and recognition on test database is explained in Chapter 7. A detailed

summary with some future recommendations is given in Chapter 8. Finally, MATLAB
and OpenCV codes are provided in Appendix A and B for reference purpose.

5

CHAPTER 2

LITERATURE REVIEW AND PROPOSED ARCHITECTURE

License plate recognition can be seen as a three stage process: i) License
plate localization; ii) Character segmentation; and finally iii) Feature extraction and

character recognition. The first two stages are mostly based on image processing and

enhancement techniques, while the final stage requires use of statistical classifiers.

A detailed survey of all previous implementation based on issues like performance,

execution time, and the platform used has been done in [20].

2.1 License plate localization

License plate localization is very crucial step in that the overall system ac-

curacy depends on how accurately we can detect the exact license plate location.

The input can be in the form of still images or video frames from surveillance cam-

eras. Based on the type of processing method used, it can be classified in following

categories.

2.1.1 Binary image processing. All these methods mostly use edge information
and mathematical morphological operations to detect the license plate region. For

instance, in [16] a morphological operator called ‘top-hat’ is used to locate small

objects of significantly different brightness. This method had a success rate of about

80%, however it was sensitive to license plate dimensions. Another method based on

edge statistics and morphological operation was proposed in [8]. This method had

an accuracy of about 99.6 % but requires prior information about candidate region
being in a specific position. Connected component analysis (CCA) which uses pixel

connectivity information for labeling the image is one of the widely used techniques in

binary image processing. It scans the image data and depending on whether we use

4-connected or 8-connected methods it labels the entire image pixel by pixel. Spatial

6

measurements like area, perimeter, and aspect ratio are used for extracting the license

plate region. These methods are relatively fast in execution as the processing is done
at binary level. However, sensitivity of algorithm to unwanted edges is an issue.

2.1.2 Gray level processing. In these methods, processing is done at gray

level and the algorithm uses the local contrast information to differentiate between

possible regions in the image. For example, in [6] the algorithm continuously scans

the image and any region which has contrast changes on a scale of 15 pixels or more

is detected as license plate region. The value 15 was selected based on experimental

analysis and depends on distance between vehicle and the camera. Instead of full
image scanning, a method based on partial image analysis was proposed in [1] . A

certain threshold value is assumed and if the number of edges is greater that value, it

is considered license plate region. This method is comparatively faster as it requires

scanning of few rows only. Another method based on block based gray level processing

is presented in [13] . In this blocks with high edge magnitude or high edge variance

are identified as possible license plate region.

2.1.3 Color processing. These methods use color information of the license

plate characters and the plate background for differentiating it from the rest of the

image. One such method based on using unique color combination of license plate

region was proposed in [2]. Some methods make use of fuzzy set theory for locating

license plate. In [19], segmentation method incorporates the fuzzy sets of yellowness

and texture to identify license plate region. The accuracy of this method was about
75.4%. A Korean license plate recognition system [3], uses an edge detection technique

which responds to only three types of edges, black-white, red-white, and green-white.

It then transforms RGB model into hue-saturation-intensity (HIS) model and the

respective H, S and I maps are generated. After integrating these individual maps in

a single map M, region with locally maximal values is identified as probable license

7

plate region. This method was fairly successful with 97.9% detection accuracy. Color

histogram information with Gaussian weighted algorithm [27] can also be used to
locate license plate region. All these color processing methods are relatively more

accurate as compared to standard gray level or binary level implementations. The

issue lies in the processing speed as the color data is huge and so real time processing

is difficult. Another important aspect is the color information is not stable in different

lighting conditions. This affects robustness of the system since most of these systems

perform well only in the originally defined environment.

2.2 Character segmentation

After detecting the license plate region, the next important task is to obtain

clean and isolated individual characters. The method used depends on the application

requirements and sometimes combination of two or more techniques is used to get

the desired results.

2.2.1 Horizontal and vertical projection data based techniques. The basic

idea is to add up image columns or rows and obtain a vector whose minimum values
allows us to segment characters. In [28], a method using vertical projection data was

presented, however this technique requires prior knowledge of license plate size as well

as the number of characters in it. This sensitivity to license plate dimensions was

reduced to an extent by employing region growing segmentation method along with

pixel connectivity information [15].

2.2.2 Binary algorithms and morphology based techniques. CCA used
in conjunction with binary object measurements such as height, width, area and

orientation can also effectively segment the license plate characters. In order to

tackle complex scenes like degraded plates, a morphology based adaptive approach

was suggested in [21]. Building on this technique, a novel approach of thickening and

8

pruning algorithm to identify noise region was proposed in [7]. All these methods had

prior knowledge of the number of characters in the license plate.

2.2.3 Contour based techniques. In [11], an algorithm based on the contour

tracking principle called ‘bug following’ was implemented for character segmentation.

A shape driven active contour model along with calculation of a special fast marching

algorithm was proposed in [24]. Gradient, curvature and other such shape description

was embedded with these contour models to perform accurate character segmentation.

2.2.4 Gray level histogram based techniques. Threshold operation is a key

aspect of character segmentation algorithm. In [6], intensity-gradient scattergrams
or histogram valley technique was employed to determine threshold value. Similar

method was proposed in [1], where the threshold value was calculated a formula

that employs the entropy values of background and foreground or by cumulative

distribution functions.

2.2.5 Local/Adaptive thresholding and transformations. Binarization of

a gray level image is a key aspect in character segmentation and researchers have
been using both local as well as global thresholding as a means of achieving it. The

image can be divided in several blocks of mxn size and a separate threshold value

can be obtained based on certain image properties. One such method based on Otsu

binarization technique is implemented in [18].

2.3 Character recognition

After isolating the individual characters, it is important we extract the unique
features of it and develop a mechanism to recognize them. For this, normally we go

for either statistical/hybrid classifiers or artificial neural networks.

2.3.1 Statistical/Hybrid classifiers. Statistical classification is a machine learn-

ing problem where we try to specify a relationship between the explanatory variables

9

and the dependent variable. A hidden Markov model for each character was employed

in [15] with a complex procedure for preprocessing and parametrization of the region
of interest (ROI). This method performed well with a predefined effective distance

between the vehicle and camera. The authors in [12] then proposed a system based

on support vector machine model. Four separate models to recognize upper numerals,

upper characters, lower characters and lower numerals were designed and success rate

was about 97.2%.

Instead of using single statistical model for classification, researchers came

up with hybrid models. Horizontal and vertical projection data, zoning density and
contour features were combined to form a feature vector [4] with an accuracy of 95.7%.

In this two stage hybrid recognition system, firstly four statistical sub-classifiers( SC1,

SC2, SC3 and SC4) separately recognize the input character and then results are

combined using Bayes method. If the characters obtained in first stage output are

different, a second stage is initiated in which structural features are fed to a decision

classifier and final decision is made. Recently, a method based on the concept of
stroke direction and elastic mesh [26] was proposed with an accuracy in the range of

97.8- 99.5%.

2.3.2 Artificial neural networks (ANN). Artificial neural network is non-linear

adaptive statistical data models which can be trained by using already know infor-

mation and then can be used to classify unknown test data. The model type, number

of hidden layers and the training method determine the overall performance of ANN.
In [19], A multi layered neural network that contains 24 input, 15 hidden, and 36

output neurons was trained to recognize 36 Latin alphabet characters. This method

was on binary input data and was reported to have accuracy of 98.5%. In order to

deal with problematic cases (I/1, B/8 and O/0) [23], training data included had more

such cases so that ANN can be effectively trained.

10

Adaptive resonance theory (ART), an unsupervised learning technique, which

dynamically changes the number of hidden layers based on convergence requirements,
was presented in [11]. To further improve the system performance, a unique method

based on Kohonen’s self organized feature maps was utilized in [3] and this worked

extremely well for difficult cases like bent/deformed license plates. Probabilistic neu-

ral networks (PNN) were first used for license plate recognition in [20]. The main

advantage was these models required only one time training, which saved a lot of

execution time. Two separate models for alphabet and number recognition were de-
signed and were reported to have success rate of about 99.5 % in general cases and

of about 90% in complex scenes like tilted/deformed license plates.

2.4 Proposed license plate recognition system

A significant research has already been done, however the system should work

effectively to different types of plate conditions, weather conditions and illumination

effects. Another important aspect is most of previous implementations are either

region or country specific. Therefore, in order to use these systems at different lo-
cation, redesign of whole system is required at times. Ideally, while designing itself

care should be taken that algorithm remains generic and can be easily used at any

location. A new design flow for license plate recognition in complex scenes is shown

in Figure 2.1.

The system can be categorized into three key stages i) license plate localization;

ii) character segmentation and iii) character recognition. The main objective of license
plate localization stage is to come up with a near perfect algorithm which will address

to some of the shortcomings in previous implementations like i) presence of another

text block in an image, ii) existence of partial contrast between the license plate

region and surrounding area, iii) license plate region located anywhere in the frame. In

character recognition stage, the emphasis is on obtaining clean and isolated characters.

11

Creating and training a
Input frame Median filtering
neural network

Edge detection using Contrast enhancement
Mexican hat operator Probable character
string

Local thresholding

Morphological dilation
operation

Subtracting
morphological erosion
output No
Is character S, 5, 0
or D
Region growing
segmentation

Region growing
segmentation
Yes

No License
Checking for specific plate
Criterion 3 using Euler property of S, 5, 0 or details
No D
number of binary image
Probable character
region

Yes
Yes
Correction in obtained
Segmented license string
plate region Segmented characters

License plate Character Character
detection segmentation recognition

Figure 2.1. Proposed license plate recognition system

12

Practical issues like presence of shadow or dirt on license plate, blur images need to

be considered. In character recognition, the main focus is on obtaining unique feature
vector as well as on training the back propagation artiﬁcial neural network classiﬁer.

13

CHAPTER 3

LICENSE PLATE LOCALIZATION

Plate localization is the first stage for any license plate recognition system.
It is an important step as the success rate at this level will decide the performance

of rest of the system. The key issue is to achieve a robust system which will work

efficiently in general as well as difficult scenarios like varying illumination and license

plate conditions. The main steps involved are edge detection, morphological dilation

operation and region growing segmentation.

3.1 Selection of edge detection method

An edge can be defined as a set of pixels which define a boundary between

two regions. Edge detection is one of the widely used techniques in detecting mean-

ingful discontinuities in an image. A detailed study of edge detection techniques has

been made in [17], where a comparison based on well known measuring metrics has

been made. The performance of any edge detection technique depends on lighting

conditions, the presence of similar intensities objects and noise content in an image.
Manual adjustment of masks and threshold value for edge consideration has to be

done in order to run edge detector with different set of data. Edge detection can be

performed at gray level as well color level. Gray level is faster as it has to operate

only on one channel as compared to three channels in color processing. However,

color edge detection provides more information about the scene since it can detect

edges in region with high color variation but low intensity variations. Some of the
widely used edge detection techniques at gray level are Canny edge detector, Mexican

hat operator based edge detection and also Boolean function based edge detection.

Color edge detection using Euclidean distance and vector angle is also used in some

cases.

14

In case of license plate detection, edge detection has an important role since

we need to extract clean and continuous edges to reduce any possibility of false license
plate detection. The performance of edge detectors need to be analyzed by considering

two important practical cases.

1. Clear and distinct contrast between the license plate and surrounding region.
In such cases, simple gradient based methods like Prewitt, Sobel operators can

be used to perform the edge detection task. These operators are used to obtain

gradient components along x and y direction. For Prewitt operator, the gradient

components are

Gx = (z7 + z8 + z9 ) − (z1 + z2 + z3 )Gy = (z3 + z6 + z9 ) − (z1 + z4 + z7 ) (3.1)

In case of Sobel operator, a weight value of 2 is used to achieve better smoothing

and noise suppression characteristics. The gradient components in this case are

Gx = (z7 + 2z8 + z9 ) − (z1 + 2z2 + z3 )Gy = (z3 + 2z6 + z9 ) − (z1 + 2z4 + z7 ) (3.2)

The final approximation is then obtained by performing

▽ ≈ |Gx| + |Gy| (3.3)

Figure 3.1(a) shows a sample image where we can visually identify the contrast

difference between license plate and region around it. Figure 3.1(b) and Figure

3.1(c) shows the result of application of Prewitt and Sobel mask on the sample
image respectively. It can be clearly seen that license plate edges are properly

separated from surrounding region and no further modification is required.

2. Partial contrast between the license plate and surrounding region. Consider a
case shown in Figure 3.2(a), where the contrast difference is less as compared

to case shown in Figure 3.1(a). In this situation, the performance of general

15

(a)

(b)

(c)

Figure 3.1. (a) Sample image with distinct contrast; (b) Prewitt edge detection; (c)
Sobel edge detection

16

gradient operators like Prewitt and Sobel is not as desired, see Figure 3.2(b)

and Figure 3.2(c). It can be seen the license plate edges are still connected to
region around it; this can cause license plate detection failure at later stage.

In order to deal with such cases, advanced operators like Mexican hat operator

have to be used. It is called so because of its similarity in appearance to a Mexican

hat, see Figure 3.3. Mexican hat operator ﬁrst performs smoothing operation and

then extracts edges. This function is also called as Laplacian of Gaussian (LoG) and

can be mathematically expressed [7] as

r 2 − σ 2 −r2 2
▽2 h(r) = − e 2σ (3.4)
σ4

Masks of diﬀerent sizes 3x3, 5x5 and 9x9 were tested and the best results were

seen for 9x9 operator.

Figure 3.4 shows the edge detection output using Mexican hat operator for

Figure 3.2(a). As compared to Prewitt and Sobel operator output, the edge detection

output using Mexican hat seems to have proper and continuous edges. Therefore, we
have selected Mexican hat operator for edge detection in our implementation.

3.2 Morphological dilation operation

In general, mathematical morphological operations are used to extract com-

ponents which describe a particular region, such as boundaries, skeletons, and the

convex hull. Dilation is morphological operation which can be considered as a local

maximum operator used to add pixel information to boundaries in an image. Dilation

operation can be mathematically expressed [7] as

X ⊕ B ≡ {X : Bx ∩ X = φ} (3.5)

where X is the object and B is the structuring element.

17

(a)

(b)

(c)

Figure 3.2. (a) Sample image with partial contrast; (b) Prewitt edge detection; (c)
Sobel edge detection

18

Figure 3.3. Wavelet Mexican hat

Figure 3.4. Mexican hat edge detection output for partial contrast case

19

If the license plate image is blurred or distorted, edge detection may not be

sufficient to obtain proper edges. Figure 3.5(a) shows one such case with a blur license
plate image. The Mexican hat operator based edge detection output can be seen in

Figure 3.5(b). The discontinuity in license plate edges result in improper detection

output, see Figure 3.5(c). In order to prevent such false detections and make these

edges thick and continuous, a dilation operation is performed after the edge detection

stage. The improved results edge detection and the corresponding detection output

can be seen in Figure 3.5(d) and Figure 3.5(e) respectively.

The size and nature of structuring element is important since smaller size can
negate the desired effect and larger size can cause the license plate regions to be

connected to the surrounding area. For our algorithm, best results were achieved

with 3x3 ones matrix as a structuring element.

3.3 Region growing segmentation

In computer vision, segmentation is used to partition the digital image into

distinct segments by using characteristic properties of an image like color, intensity or
texture. For example, in medical imaging applications, segmentation can be used to

locate tumors, tissues which then can be used for diagnosis purpose. Segmentation is

widely used to locate objects and boundaries in a digital image. Image segmentation

can be classified based on two basic properties of image: discontinuity and similarity

[7]. In first type, abrupt changes in an image like edges are first detected and then

a edge linking procedure is used to locate meaningful objects in the image. Hough
transform is an example of this category and can be used to segment regular curves

like lines, circles, ellipses. The second category divides the image into different sub

regions that are similar based on certain predefined criterion. Thresholding, region

growing and region splitting are some of the techniques which belong to this category.

20

(a)

(b)

(c)

(d)

(e)

Figure 3.5. (a) Blur image; (b) Edge detection output; (c) Plate detection without
dilation; (d) Dilation output; (e) Plate detection with dilation

21

(a) 4 pixel connectivity (b) 8 pixel connectivity

Figure 3.6. Pixel connectivity types

For detecting a license plate, first we have to obtain all probable license plate

regions in an image. The output of dilation operation is a binary image and so it

will be easy to grow regions based on pixel connectivity information. Region growing

segmentation is ideally suited for this purpose. In region growing, we group pixels

based on some predefined pixel connectivity information to form sub regions. De-
pending upon the accuracy requirements, we can either go for 4-pixel connectivity

Figure 3.6(a) or 8-pixel connectivity Figure 3.6(b). Although ideally we would like to

use 8-pixel connectivity every time, on occasions it can be computationally extensive.

The performance of region growing algorithm depends on the starting point ie

the seed point and the stopping rule selected for the implementation.

3.3.1 Seed point selection. This is an important aspect of region growing

especially when only one starting point is defined for the implementation. Consider
a sample image of fluid scatter (splash). For this image, we have to segment the

splash part from the rest of image using region algorithm. Figure 3.7 shows original

image and region growing segmentation output by using different seed points. It

22

can be visually confirmed that seed point=230 gives the best segmentation results as

compared to seed point=200 and 245.

3.3.2 Stopping rule. In region growing, we need to define a stopping rule which

will terminate the region growing process. The selection of rule depends on the

image characteristics like size, and likeliness of candidate pixel. The stopping rule

will determine the execution time required for region growing process. For instance,

consider a medical image, where we need to segment the tumor part from rest of the

image. Suppose, the stopping criterion used is maximum number of pixels in the

image, M. Figure 3.8 shows the original image with the results for M=25, 55 and 75.
Based on the application requirements, a tradeoff between accuracy and computation

time is made and suitable stopping rule is selected.

For our implementation, we have to find all possible license plate regions.

This means region growing has to be performed on the entire image. For every bright

pixel, its neighboring four pixel connectivity information was checked and region was

grown based on this information. In order to avoid region growing for noise part in
the image, we defined a threshold which decides whether the region is object part

or noise content. A lot of analysis was done and it was seen that regions with area

less than 75 pixels need not be considered. In order to increase the algorithm speed,

sub-sampling was used, where operation was performed for every two pixels, instead

of performing for each and every pixel in the image. The final output will be several

sub regions labeled based on 4-pixel connectivity information, see Figure 3.9.

3.4 Criterion for license plate detection

Once we obtain all probable regions in an image after performing region grow-

ing segmentation, we need to define a criterion to select accurately the license plate

part from it. Following different criteria can be used for this purpose.

23

(a) (b)

(c) (d)

Figure 3.7. Region growing segmentation output for diﬀerent seed points (a) Sample
image; (b) Seed point=200; (c) Seed point=230; (d) Seed point=245

24

(a) (b)

(c) (d)

Figure 3.8. Region growing segmentation output with diﬀerent stopping rule (a)
Original image ; (b) Stopping rule=25; (c) Stopping rule=55; (d) Stopping rule=75

25

(a) (b)

Figure 3.9. Region growing for license plate detection (a) Output after dilation; (b)
Region growing output

3.4.1 Criterion 1 License plate dimensions. Most of the countries/states

generally fix license plate dimensions for all vehicles. This information can be used
to separate license plate region from rest of the probable segments. It is very easy

to implement and computationally less extensive. However, by using this criterion,

we make algorithm sensitive to license plate dimensions, which depend on the dis-

tance between the vehicle and camera. In real time scenarios, this distance can vary

depending upon the application used. Practically it is extremely important that al-

gorithm works at least on 15% tolerance range of license plate dimensions. Therefore,
this criterion does not meet performance requirements.

3.4.2 Criterion 2 Euler number of binary image. For a digital image, Euler

number helps to find the structure information of object without relating to the

geometric shape of object [14]. It is defined as total number of objects in an image

’C’ minus the number of holes in those objects ’H’.

E =C−H (3.6)

26

Characters used in license plate region are alphanumeric (A-Z and 0-9). If we

carefully observe these characters in an image, this can be seen as closed curves like
in 0, 9, 8, P etc. Thus clearly license plate region will have more closed curves than

in any other part of an image. Euler number of an image can be used to distinguish

the different regions obtained after the region growing output. Experimentally, it

was found that in alphanumeric regions, value of Euler number is zero or negative,

while for other regions it has a positive value. Using Euler number criterion will

overcome the sensitivity issue encountered in using Criterion 1. This criterion works
accurately and efficiently for majority of test cases. The only failure case was seen

when the vehicle image has alphanumeric characters in more than one region, see

Figure 3.10(a).

3.4.3 Criterion 3 Combination of Criterion 1 and Criterion 2. In this

implementation, we have used a combined approach of Euler number criterion as

well as license plate dimensions to overcome problem raised in case of Criterion 2.

Instead of predefining exact license plate dimensions, we specify a range of acceptable
minimum and maximum license plate sizes. This meets the real time performance

requirements of algorithm working for cases with 15 % tolerance over exact license

plate dimensions. Figure 3.10(b) shows the application of this criterion on failure

case of criterion 2. It can be seen that license plate is accurately detected using this

criterion.

27

(a) (b)

Figure 3.10. (a) Failure case for Criterion 2; (b) Criterion 3 output for part (a) case

28

CHAPTER 4

CHARACTER SEGMENTATION

After locating the license plate, the next important step is to segment each
character individually, avoiding any possibility of joint characters. The main chal-

lenge is ensuring performance in complex scenes like blur image, tilted license plate

and presence of shadow/dirt on the license plate region. Figure 4.1 shows all the

necessary steps that need to be performed in order to ensure accurate character seg-

mentation. Firstly, the license plate image obtained after localization stage needs to

be preprocessed for better accuracy. Threshold operation is then performed in order
to convert the gray level image into binary format for further processing. Morpholog-

ical erosion operation is done to remove the noise content and ﬁnally region growing

segmentation is used to segment the character region.

4.1 Image preprocessing

Even though we assume that the image acquired during acquisition stage is

perfect, it may not be the case always. The image may have some noise content,
which can cause performance issue in later analysis.

4.1.1 Image noise. The quality of imaging sensors used during the image acqui-

sition and the variations in environment conditions decides the noise content in the

acquired image. In Figure 4.2, some of the common forms of noise that can be seen

in a digital image are shown on image of Lena, most widely used test image for image

processing algorithms.

4.1.1.1 Gaussian Noise. Figure 4.2(b) shows Lena image with Gaussian noise in
it. In this case, each pixel will be distorted by small amount compared to the original

image. If we plot a histogram for amount of distortion of a pixel with respect to its

frequency content, it can be seen that it follows a Gaussian distribution.

29

Median filtering

Contrast enhancement

Local thresholding

Subtracting morphological erosion output

Region growing segmentation

No

Probable character region

Yes

Segmented characters

Figure 4.1. Design ﬂow for character segmentation

30

4.1.1.2 Salt and pepper noise. In this case, we can see dark and white dots on

the image everywhere and so it is called as salt and pepper noise, see Figure 4.2(c).
This is usually caused because of dust flecks inside the camera or faulty image sensing

elements.

4.1.1.3 Speckle noise. Speckle noise is a granular noise usually seen in the form

of increased mean gray level of a local area, see Figure 4.2(d). It is usually present in

cases where we acquire images from multiple sources and combine all the images in

one.

4.1.2 Median filtering. In order to improve character segmentation performance,
we need to first remove the noise content from the image and reconstruct a better

version of original image. While doing so, we need to make sure that we preserve

important characteristics of the image like boundaries, edges. Median filter is a non-

linear spatial filter and is used widely for image denoising [25], [5]. In median filtering,

we replace the original pixel data by median of pixels contained in a prescribed

window. The median filter output for 3x3 window can be obtained by performing
following steps.

1. Compute the mean value y0 of the pixels in the filter window.

2. Subtract y0 from each pixel within the filter window.

3. Find the median ym of the nine modified points.

ˆ
4. The final output is X = ym + y0 .

In order to ensure that the edges of the characters are preserved, analysis
was done to determine the optimal median filter size. In this case, best results were

obtained for 3x3 filter window.

31

(a) (b)

(c) (d)

Figure 4.2. Various forms of noise in an image (a) Original image; (b) Gaussian Noise;
(c) Salt pepper noise; (d) Speckle noise

32

(a)

(b)

Figure 4.3. (a) Original license plate image; (b) Enhanced version after image pre-
processing

4.1.3 Contrast enhancement. If the image acquired is blurred, it will be diﬃcult

to process it further, since most of its character information is not properly visible.

Therefore, we have to locally enhance it in smaller regions while preserving its global

contrast. This can be done by increasing the dynamic range for pixels in certain

intensity range. After observing the pixel data in the license plate region, it was

found that the character information for most of the images in database is usually
in the intensity range of 0-0.4 on a scale of 1. In order to obtain further contrast

enhancement, we increased the dynamic range for pixels in this (0-0.4) intensity range.

Figure 4.3 shows one sample license plate image and enhanced output after

preprocessing step. The improved contrast enhancement results can be seen in Figure

4.3(b).

4.2 Threshold operation

Thresholding is a segmentation process used to separate operate object and
background information. It is also used for converting grayscale image into a binary

33

image. There are basically two types of threshold operation.

4.2.1 Global thresholding. In this case, we select one single threshold value for
entire image. Mathematically it can be represented as



1, if f (x, y) ≥ 0




g(x, y) = (4.1)


0, otherwise.




where f(x,y) is the original grayscale image, g(x,y) is the threshold output

image and T is the threshold value obtained. This operation is easy to implement

and also less time consuming. However, it is sensitive to illumination conditions and

fails in cases if there is a brighter or darker region other than object. One such case is

shown in Figure 4.4(b), where we can see character ‘O’ and ‘2’ are connected because
of unwanted dark region in original grayscale image, see Figure 4.4(a).

4.2.2 Local thresholding. In the local threshold operation, we divide the grayscale

image into several parts by using a windowing operation and then perform the thresh-

old operation separately for each case. Since, we perform threshold operation for

each region separately; a single brighter or darker region cannot corrupt the thresh-

old value. Local thresholding ensures that in most cases, we get clear and separated
characters. In Figure 4.4(c), we can see all the characters are separated compared to

Figure 4.4(b), where global thresholding was performed. The performance of local

threshold operation will vary depending on the selected window size. Experimentally,

best results were obtained when the window size was chosen approximately equal to

the general individual character size in the license plate image.

For our algorithm, the threshold value for local region is obtained using Otsu’s
method for gray level histogram [22]. In this method, optimal threshold value is

obtained such that the intra class variance between the two classes(foreground and

34

(a)

(b)

(c)

Figure 4.4. (a) Sample image; (b) Global thresholding; (c) Local thresholding

35

background) is minimal.

4.3 Morphological erosion operation

In certain situations if some dirt or shadow is present on the license plate
region, the output of local threshold operation may not be suﬃcient. For example in

Figure 4.5(b), it can be seen even after the local threshold operation some amount of

unwanted noise is still present near character ‘4’ and ‘X’. This noise can be removed

by performing morphological erosion operation see Figure 4.5(c). Mathematically,

erosion operation can be represented [7] as

X ⊖ B ≡ {X|Bx ⊆ X} (4.2)

where X is the object and B is the structuring element. The size and na-

ture of structuring element should be selected with an aim of keeping the character

information intact and only removing unwanted parts in the binary image.

4.4 Region growing segmentation

The pixel connectivity information can be used to separate the characters.
Region growing segmentation along with four pixel connectivity approach is used in

this case. This is the same algorithm which was used for license plate detection pur-

pose. Figure 4.5(d) shows the separated characters after performing region growing

segmentation.

36

(a)

(b)

(c)

(d)

Figure 4.5. Steps in character segmentation (a) Image with shadow; (b) Local thresh-
olding; (c) Erosion operation; (d) Segmented characters

37

CHAPTER 5

FEATURE EXTRACTION AND CHARACTER RECOGNITION

Character recognition, often referred in general as optical character recogni-

tion, has been a major revolution in the electronics world. It is widely used for

scanning handwritten notes, online exams evaluation and techniques such as machine
translation, text to speech conversion and text mining. The accuracy requirements

depend on the intended application and in our case it is used to recognize the seg-

mented characters. It can be considered as a two stage process; firstly we have to

extract key features associated with the characters and secondly, we need to define a

mechanism to recognize these characters based on the extracted feature information.

For the latter purpose, we use statistical classifiers in general and in our case we have
used artificial neural networks.

5.1 Feature extraction

Feature extraction is a process in which we define and characterize the available

data by a set of features called as feature vector. Following are two important points

we need to consider while selecting feature vector for character segmentation.

1. The robustness of character recognition algorithm depends on how well it can

recognize different versions of a single character. In Figure 5.1, all the images

are for same character G of size 24x10, however it varies in terms of spatial

information, noise content and thickness of edges. This is a common case espe-
cially when dealing with different illumination conditions. The feature vector

should be such that algorithm can still accurately recognize all these varied

representations of a single character.

2. Sometimes it is difficult to distinguish between characters like D and 0 or 5 and

S because of the similarity in their appearance as well as spatial information. It

38

Figure 5.1. Diﬀerent versions of character ’G’

is critical that our feature vector contain certain characteristics which will help

us diﬀerentiate between these complex cases.

5.1.1 Selection of feature vector . In order to deal with these challenges,

our algorithm uses a combination of spatial information as well region properties.

Following is the description of these key features.

1. Feature 1: Analysis of variation of all region properties with respect to the

characters was performed and from them six properties were selected. Following

are the image properties selected in feature vector.

(a) Area: This gives the actual number of pixels in the region.

(b) Perimeter: Perimeter is calculated as the distance between each adjoining
pair of pixels around the border of the region.

(c) Minor-major axis length: These specify the length of pixels associated

with the minor and major axis of the ellipse that has the same normalized
second central moments as the region.

(d) Orientation: Orientation is the angle between the x-axis and the major

axis of the ellipse that has same secondary moments as that of region.
Orientation value is the angle between the horizontal dotted line and the

major axis.

39

(e) Euler number of binary image: This gives a value which gives the number

of objects minus the number of holes present in the object [14].

2. Feature 2. The second part of feature vector includes projection information

about X and Y axes. Projection is mapping of binary image into one dimensional

array. A horizontal projection is sum of white pixels (background being black)

projected onto vertical axis x, while vertical projection is the sum of white pixels

onto horizontal axis y. Projection data forms major part of feature vector and

is unique for each character.

3. Feature 3: Some of the characters like 0, M, N are symmetric about X-Y axes.

This information can be used to distinguish different characters especially 0

and D. This symmetry information can be quantified by a coefficient called

as reflection symmetry coefficient. For 2D images, a way of calculating this

coefficient is proposed in [20]. In this implementation for a character of size

24x10, we calculate measure of symmetry about X and Y axis in following way.

(a) Consider Cin as the original character matrix where information is stored

in binary format. The center of matrix is considered as the origin. Cx and

Cy are the transformed matrix obtained after flipping about the X and Y

axes respectively.

(b) Calculate the area of original matrix Cin .

(c) Determine the union of transformed matrixes Cx and Cy with original
matrix Cin .

(d) Calculate the areas of regions (Cin ∪ Cx ) and (Cin ∪ Cy ).

(e) The reflection coefficient can be determined as

Area(Cin ) Area(Cin )
Xcoef f icient = ; Ycoef f icient = (5.1)
Area(Cin ∪ Cx ) Area(Cin ∪ Cy )

40

Feature 3 has significantly improved algorithm’s performance in dealing with

complex cases like D and 0. Experimentally it was found that for 0, the value of
Xcoef f icient and Ycoef f icient are approximately similar. On the other hand, since D has

symmetry only along X axis, value of Xcoef f icient is larger compared to Ycoef f icient .

5.2 Artificial neural networks

5.2.1 Basis concept and applications. Humans have the capability to solve

complex perceptional problems at high speed. This is mainly possible because of

biological neural networks which give them ability to learn adaptively, inherent con-

textual information and massive parallelism capability. Inspired by these performance
characteristics of biological neuron, a mathematical model called artificial neural net-

work based on interconnection of artificial neurons, which can be trained by flow of

external and internal information was developed. Since its first use, ANN models has

proven to be powerful tool for solving complex engineering problems [9] like

1. Pattern classification. In these problems, we first define a feature vector which

represents the input pattern and then finally we use this feature information to
assign it to a set of predefined output classes. Character recognition, speech

recognition, blood cell classification are some examples in this category.

2. Clustering/categorization. Clustering refers to grouping of input data based

on some predefined similarity criterion. This falls in an unsupervised leaning

paradigm as there is not data defined for training purposes. Common examples

are data mining, data compression.

3. Function approximation. Function approximation is used to represent target

data with set of well defined available functions. Interpolation, extrapolation,

regression analysis, and curve fitting can be considered to be function approxi-

mation problems.

41

4. Prediction/forecasting. In this, for the given set of data at a particular time

instant we need to predict sample value at future time interval. Machine learn-
ing techniques applied in stock market prediction is well known example in this

category.

5.2.2 Classification of artificial neural networks. Based on the connection

pattern in neural network architecture, artificial neural networks are classified into

two categories.

1. Feed forward neural networks. These are the simplest form in which data flow is
only in one direction. It flows from input nodes to the hidden nodes (if any) and

then finally to the output nodes. Multi-layer Perceptron, radial basis function

nets are examples of this category.

2. Feedback/recurrent neural networks. Contrary to the feed forward neural net-

works, these architectures are bidirectional and have loops in it. Kohonen’s

SOM, Hopfield networks are some examples.

For our implementation, we have used feed-forward based neural network de-

sign.

5.2.3 Feed forward neural networks.

5.2.3.1 Basic model Perceptron. Figure 5.2 shows a simplest form of feed-

forward neuron model is called as ‘Perceptron’. In this model, (x1, x2,...xn) are the

input vectors, (w1, w2,...wn) are the weights associated with the input vectors, h is

the summation of all inputs with its weights and y is the output. Activation function
decides the amplitude range for output. For example, if step function is used, the

range is between 0 and 1, while if a signum function is used, the output lies between -1

42

and 1. The selection of activation function is decided by the complexity of proposed

problem. For our design, we have used a log sigmoid activation function.

weights
inputs
x1
w1j

ac va on
x2 w2j func on
net input
net j
∑
Oj
x3
ac va on
. w3j
. transfer
. . func on
. .
. . θj
x4 threshold
w4j

Figure 5.2. Basic feed forward model, Perceptron

The basic algorithm for this model [9] can be explained as follows.

1. Initialize the associated weights and the activation function for the model.

2. Evaluate the output response for given input pattern (x1 , x2 , xn )t .

3. Update the weights based on following rule

wj (t + 1) = wj (t) + η(d − y)xj (5.2)

Based on the requirements, this principle can be extended to multiple layers.

In general, a standard L-layer model has L input nodes, (L-1) hidden layers and

43

connected set of predeﬁned output nodes. A typical three layer feed forward model is

shown in Figure 5.3. It can be seen that the data ﬂow is unidirectional and the outputs
have no connection between the units in the same layer. Multi-layer perceptron

became extremely popular with the use of back propagation concept. In case of back-

propagation model, the error at the output is propagated backwards and accordingly

the weights for inputs are adjusted.

xq wqi(1) wij(2) wjk(l) yk(l)

Inputs Hidden Output
layer layers layer

Figure 5.3. Three layer Perceptron

5.2.4 Design of neural network for character recognition problem. Following
is the description of some of these key design parameters in reference to character

recognition problem.

1. Input Layers. Input layers to the neural network are the feature vectors ex-

tracted for the all the characters. The overall size is decided by the character

set and the feature vector size.

2. Output Layers. Output nodes are the set of all possible characters that are

44

present in license plate information. The number of output nodes depends

upon the numbers of characters that need to be classiﬁed. Typically, it consists
of 26 alphabets ‘A-Z’ and 10 numbers ‘0-9’. However, to reduce algorithm

complexity, certain countries avoid using both 0 and O together. In our test

database, we have all the characters except O, 1 and Q. Therefore, the number

of output layers is 33 in our implementation.

3. Hidden Layers. Hidden layers are the intermediate layers between the input

and output layers. There is no generic rule as such for deciding the number of

hidden layers. In our design, the number of hidden layers was experimentally
found to be 25.

For our implementation, we have used a supervised learning paradigm. We

obtained a set of 50 images for each character from the database images and trained

the neural network using them. For better results, it is important to include images of

all types of character representation in the training dataset. We have used MATLAB

Neural network tool box for training the neural network. Following are some of the
key steps in MATLAB implementation.

%Training the neural network using ideal target vectors

[R,Q] = size(alphabet);

[S2,Q] = size(targets);

P = alphabet;

net = newff(minmax(P),[S1 S2],{’logsig’ ’logsig’},’traingdx’);
net.performFcn = ’sse’;

net.LW{2,1} = net.LW{2,1}*0.01;%Layer weights

net.b{2} = net.b{2}*0.01;%Bias vector

net.trainParam.goal = 0.5;

45

net.trainParam.show = 20;

net.trainParam.epochs = 5500;
net.trainParam.mc = 0.7;

[net,tr] = train(net,P,T);

%Training the neural network using training image data set

netn=net;

netn.trainParam.goal = 0.6;
netn.trainParam.epochs = 300;

%P1=new alphabet;

%T1=Target vector;

[netn,tr] = train(netn,P1,T1);

% Simulating the network performance
% Rp - > Feature vector of the character to be indentified

A2=sim(netn,Rp);

A2=compet(A2);

answer = find(compet(A2) == 1);

%alphamat= set of all possible characters
character=alphamat(answer);

It was observed that for certain characters like 0-D and 5-S, error rate was more

compared to other characters, because of similar spatial information content. In order

to improve the overall accuracy, a two stage detection process for these characters is

used. If the characters are possibly between 5, 0, S and D, then region properties

like orientation, reflection coefficient were again used in the 2nd stage identification

46

process and the ﬁnal possible character was identiﬁed. This two stage process has

improved the overall algorithm accuracy by 1-2%.

47

CHAPTER 6

IMPLEMENTATION IN MATLAB AND OPENCV

The major focus of this project has been to create a software platform for ro-
bust license plate recognition system. This platform then can be used as a framework

for hardware implementation in future. Initially, all the algorithms were implemented

in MATLAB. Once complete system was tested and analyzed in MATLAB, OpenCV

platform was then used with aim of building real time embedded system in future.

6.1 MATLAB implementation

For MATLAB, all algorithms were made from scratch and then tested with the
predefined MATLAB functions. This was done in order to make our implementation

can be ported for future hardware implementation. For character recognition purpose,

we have made use of artificial neural network toolbox. Following are some of the

important leanings from MATLAB implementation.

1. For computer vision and image processing applications, it is always good to start

first with MATLAB implementations because of its built-in function capability.

Most of the functions like segmentation algorithms, morphological operations,
region properties are already there and so ist easy to verify our custom code.

2. While doing this, we need to make sure that we don’t overuse MATLAB func-

tionality. If we do so, the mapping of software and hardware becomes complex.

3. MATLAB is not the ideal platform to analyze algorithm speed.

6.2 OpenCV

OpenCV was first developed by Intel research team with a strong focus of real
time embedded applications. It is an open source library with over 500 functions in

48

the field of computer vision, machine learning and statistical pattern recognition.The

library is written in optimized C language and therefore can be used with C as well
C++ environment.

For our project, we have implemented license plate detection part in OpenCV.

Following are some of the important aspects in OpenCV.

1. OpenCV has primitive data types like Cvpoint(), CvSize(), CvRect()and CvS-

calar. These are used while accessing matrix or image details.

2. OpenCV has a object oriented design, especially C++ inheritance concept is

the base of it. Therefore, IpIImage, which is used for accessing image data is
basically derived from CvMat class, which is used to deal with matrix elements.

3. For our algorithm, we have used in-built functions like morphological dilation

operator, contour finder.

6.3 Comparison of MATLAB and OpenCV

1. OpenCV was specifically designed for image processing applications, while in

comparison MATLAB is generic. Therefore, it is comparatively faster and easier

to use.

2. MATLAB is based on Java and so there is delay associated with respect to inter-

preting MATLAB code and then converting into Java for execution. OpenCV

is written in C and so it has no such issues.

The above points can be confirmed from Table 6.1. It shows execution speed for all

the steps in license plate detection for MATLAB and OpenCV implementation.

49

Table 6.1. Comparison of MATLAB and OpenCV execution time for license plate
detection of a single frame
Task MATLAB code execution time OpenCV code execution time
Edge detection 0.69s 0.4
Dilation 0.05s 0.03s
Segmentation 0.384s 0.3
Total time 1.13sec 0.731sec

50

CHAPTER 7

EXPERIMENTAL RESULTS

Performance evaluation for the license plate recognition algorithm is a chal-
lenge in itself, since there is no common reference point for benchmarking [10]. Many

systems claim an accuracy of more than 95% however truth is all of them work only

on restricted environmental conditions. In fact, some of these systems when used

in real time environment conditions, system performance was actually found in the

range of 60 to 80%. It is important that while evaluating system performance, we

make sure that we test on all practical test cases.

7.1 Database for testing of license plate recognition

It is important that while evaluating system performance, we make sure that

we test on all practical cases. We tested our algorithm on 1500 different frames

obtained from a demo video freely available online, taken from City Sync license

plate recognition camera. The resolution of all frames is 480X640. All these frames

were then classified into following types.

1. Type 1: In these samples, license plate is fully visible and all its characters

are clear and distinct, see Figure 7.1(a). Also, the license plate location in the

image is in the center portion of the image.

2. Type 2: Images in these samples are little blurred due to variations in illumina-

tion conditions; see Figure 7.1(b). This is mainly because of motion blur effect

during image acquisition stage.

3. Type 3: These samples have little dirt or shadows on or near license plate region,

Figure 7.1(c). These include cases where even with presence of dirt/shadow,

license plate information is still visible. Cases where license plate details are

51

hidden need not be considered in analysis, as its impossible to correctly recognize

such cases.

4. Type 4: In these samples, another text block apart from license plate is present

in the frame, Figure 7.1(d).

5. Type 5: License plate details are present in two rows, Figure 7.1(e).

7.2 Results

Table 7.1 summarizes the results obtained for all the above mentioned test

cases.

Table 7.1. Summary of license plate localization and detection results
Image type Images tested Plate detection success Character recognition success
Type 1 879 99.4% 97.81%
Type 2 442 96.3% 96.63%
Type 3 153 94.8% 95.69%
Type 4 44 100% 99.25%
Type 5 11 100% 97%
Total 1529 98.1% 97.05%

Overall, the algorithm has been tested on images of 335 vehicles. On an average

5 frames were available for each vehicle. A summary of license plate detection results

in terms of number of frames per vehicle is shown in Table 7.2, where n is the number

of input frames available for each vehicle (maximum value of n=5).

52

Table 7.2. Number of frames detected per vehicle
Number of frames per vehicle Number of vehicles falling in this category
5 291
3≤N ≤5 32
1≤N ≤3 5
None 7

53

(a)

(b) (c)

(d) (e)

Figure 7.1. Types of images included in license plate database (a) Type 1; (b) Type
2; (c) Type 3; (d) Type 4; (e) Type 5

54

CHAPTER 8

CONCLUSION AND FUTURE WORK

8.1 Conclusion

In this project, a framework has been proposed to localize and recognize license

plate in complex scenes. A detailed study of all previous implementations was per-

formed and key implementation issues at license plate detection as well as recognition

stage were listed. Following are some of the key highlights of our algorithm.

1. We ensured to include all practical test cases in our database including complex

scenarios like tilted license plate, license plate images taken in different illumi-
nation conditions, presence of dirt/shadows on license plate. This serves as a

common benchmarking for future implementations.

2. The use of Mexican hat operator helps to improve the performance of edge

detection step, when there is only partial contrast between license plate region

and region surrounding it.

3. Euler number criterion for a binary image helps to reduce the sensitivity of

algorithm to license plate dimensions.

4. Preprocessing step using median filter and contrast enhancement ensures per-

formance in case of low resolution and blurred images.

5. Reflection symmetry coefficient along with two stage identification process im-

proves the character recognition performance significantly especially in dealing

with complex cases like recognizing 0 and D.

6. Local threshold operation prevents a single brighter or darker region from cor-
rupting the threshold value and thereby improves binarization process.

55

8.2 Future recommendation

8.2.1 Extension of algorithm for mobile license plate recognition systems.
The algorithm can be extended to mobile license plate recognition systems because of

its ability to provide exact license plate location in complex cases. Also, to increase

the intelligence of license plate detection algorithm while working with video input,

motion analysis can be applied to the sequential frames and selection of proper frame

can be done. The performance of algorithm will also improve if higher resolution

input frames are used.

8.2.2 Hardware implementation. This project mainly focused on software
implementation aspect and algorithms were implemented in MATLAB and openCV

platforms. This can serve as a framework for hardware implementation using FPGA.

VHDL codes for key image processing stages like edge detection, morphological dila-

tion and erosion operation need to be developed. The major challenge is to implement

artiﬁcial neural networks in FPGA.

56

APPENDIX A

MATLAB CODE FOR LICENSE PLATE RECOGNITION

57

1. License plate localization

fi=imread(‘Path of input vehicle image with license plate’);

fin=rgb2gray(fi);

d=double(fin);

[r c]=size(d);

%Mexican Filter Operator
filter= ...

[ 0 0 0 -1 -1 -1 0 0 0;

0 -1 -1 -3 -3 -3 -1 -1 0;

0 -1 -3 -3 -1 -3 -3 -1 0;

-1 -3 -3 6 13 6 -3 -3 -1;

-1 -3 -1 13 24 13 -1 -3 -1;
-1 -3 -3 6 13 6 -3 -3 -1;

0 -1 -3 -3 -1 -3 -3 -1 0;

0 -1 -1 -3 -3 -3 -1 -1 0;

0 0 0 -1 -1 -1 0 0 0];

%Creating image matrix for Mexican Hat operator

gm=zeros(r,c);

for i=5:2:r-5
for j=5:2:c-5

gm(i,j)=sum(sum(double(f(i-4:i+4,j-4:j+4)).*filter,2));

end;

end;

%Removing the unwanted edges by using a threshold

fh=gm>1200;
%Dilation Operation

x=1;

58

y=1;

fs=double(fh);
se=ones(3,3);

for x=3:3:r-20

for y=3:3:c-20

if(x+50<=r)

xend=x+50;

else
xend=r;

end;

if(y+100<=r)

yend=y+150;

else

yend=c;
end;

if(sum(fh(x:xend,y))<=35 || sum(fh(x,y:yend),2)<=60)

if((sum(fh(x,y:y+3),2)<=3 && sum(fh(x,y:y+3),2)>2 ))

fs(x-2:x+2,y-2:y+2)=bwmorph(fh(x-2:x+2,y-2:y+2),’dilate’,se);

end;

end;
end;;

end;

% Image with Dilation Performed

f=double(fs);

% Determining the size of an image ie number of rows

% and columns

[ row col ]= size(f);

59

% Initializing a matrix for segmented Image

g=zeros(row,col);
gl=zeros(row,col);

label=1;

n=1;

x=1;

iter=[];

it=0;
ss_prev=0;

nn=[];

sss_mat=[];

for i=1:2:row

for j=1:2:col

r_pt=i;
c_pt=j;

if(g(r_pt,c_pt)==0)

while(true)

% using 4 Neighbor Rule

if(f(r_pt(n),c_pt(n))==1 && g(r_pt(n),c_pt(n))==0)

g(r_pt(n),c_pt(n))=label;
if(r_pt(n)+1<=row)

if( f(r_pt(n)+1,c_pt(n))==1)

r_pt=[r_pt r_pt(n)+1];

c_pt=[c_pt c_pt(n)];

x=x+1;

end;

end;

60

if(c_pt(n)-1>=1)

if(f(r_pt(n),c_pt(n)-1)==1)
r_pt=[r_pt r_pt(n)];

c_pt=[c_pt c_pt(n)-1];

x=x+1;

end;

end;

if(c_pt(n)+1<=col)
if(f(r_pt(n),c_pt(n)+1)==1)

r_pt=[r_pt r_pt(n)];

c_pt=[c_pt c_pt(n)+1];

x=x+1;

end;

end;
if(r_pt(n)-1>=1)

if(f(r_pt(n)-1,c_pt(n))==1)

r_pt=[r_pt r_pt(n)-1];

c_pt=[c_pt c_pt(n)];

x=x+1;

end;
end;

end;

if(n>=x)

break;

end;

n=n+1;

end;

61

y1=min(r_pt);

y2=max(r_pt);
x1=min(c_pt);

x2=max(c_pt);

a1=g(min(r_pt):max(r_pt),min(c_pt):max(c_pt));

f1=d(min(r_pt):max(r_pt),min(c_pt):max(c_pt));

[ra ca]=size(a1);

if(n>=50 )
b1=bwlabel(a1);

ss=regionprops(b1,’EulerNumber’);

sss=struct2array(ss);

sss=min(sss);

sss_mat=[ sss_mat sss];

if(sss < ss_prev && sss<0 && ca<=190 ...
&& ra <=60 && ca>=50 && ra>=15 && mean(mean(f1))<=220 )

x_cor1=x1;

y_cor1=y1;

x_cor2=x2;

y_cor2=y2;

ss_prev=sss;
end;

label=label+1;

else

g(r_pt,c_pt)=0;

end;

end;

x=1;

62

n=1;

it=1;
end;

end;

if( exist(’y_cor1’)==1)

d(y_cor1:y_cor1+2,x_cor1:x_cor2)=255;

d(y_cor2:y_cor2+2,x_cor1:x_cor2)=255;

d(y_cor1:y_cor2,x_cor1:x_cor1+2)=255;
d(y_cor1:y_cor2,x_cor2:x_cor2+2)=255;

end;

%Segmented license plate image

d=mat2gray(d);

lp= d(y_cor1:y_cor2,x_cor1:x_cor2);

2. Character Segmentation

%License plate image, characters of which are to be segmented
lp1=d(y_cor1:y_cor2,x_cor1:x_cor2);

[rl cl]=size(lp1);

% Median Filtering

lp=medfilt2(lp1,[3 3]);

%Contrast enhancement

lpf=imadjust(lp,stretchlim(lp, [0.1 0.5]));
%Creating output image matrix

output=zeros(rl,cl);

%Window for local threshold operation

dis=round(cl/7);

%Local threshold operation

63

for i=1:dis:cl

if(i+dis-1<=cl)
t=threshcal(lpf(:,i:i+dis-1),a);

output(:,i:i+dis-1)=lpf(:,i:i+dis-1)<=t;

else

t=threshcal(lpf(:,i:cl),a);

for z1=2:rl-1

for z2=i+5:cl-5
if(mean(mean(lpf(z1-1:z1+1,z2-5:z2+5)))<=t)

output(z1,z2)=1;

end;

end;

end;

output(:,i:cl)=lpf(:,i:cl)<=t;
end;

end;

%Structuring element for erosion operation

se=[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1];

output=output-imerode(output,se);

[of lab lpdet]=reggrow1(logical(output),number);
%Segmented characters

lpdet=logical(lpdet);

3. Character Recognition

% Output string giving license plate iformation

lpstr=[];

for i=1:lab-1

64

R=lpdet(:,st:st+9);

st=st+10;
b=bwlabel(R);

%Feature extraction

ar=struct2array(regionprops(b,’area’));

or=struct2array(regionprops(b,’orientation’))/90;

eu=struct2array(regionprops(b,’eulernumber’))/10;

pe=struct2array(regionprops(b,’perimeter’));
mi=struct2array(regionprops(b,’minoraxislength’));

ma=struct2array(regionprops(b,’majoraxislength’));

temp=logical(R);

%Reflection X and Y coefficient determination

v1=temp;

v1(:,6:10)=flipdim(temp(:,1:5),2);
vx=(v1 + temp)/2;

vx=vx>=0.5;

xcoef=sum(sum(temp),2)/sum(sum(vx),2);

v2=temp;

v2(1:12,:)=flipdim(temp(13:24,:),1);

vy=(v2 + temp)/2;
vy=vy>=0.5;

ycoef=sum(sum(temp),2)/sum(sum(vy),2);

ed=struct2array(regionprops(b,’equivdiameter’))/100;

[val pos]=max(fa);

vcoeff=pe(pos)/ar(pos);

mcoeff=ed(pos);

Rp=[xcoef/ycoef;pe(pos)/ar(pos);mi(pos)/ma(pos);or(pos);...

65

eu(pos);sum(temp,2)/norm(sum(temp,2));...

transpose(sum(temp,1))/norm(sum(temp,1)) ];
A2=sim(netn,Rp);

A2=compet(A2);

answer = find(compet(A2) == 1);

if(i<=numel(lpnum))

if(alphamat(answer)==lpnum(i))

numrc=numrc+1;
else

answ=find(alphamat==lpnum(i));

err(answ)=err(answ)+1;

end;

end;

lpstr=[lpstr alphamat(answer)];
end;

numc=numc+numel(lpnum);

if(strcmp(lpstr,lpnum)==1)

tr=tr+1;

sr=strcat(num2str(num),’/’,num2str(1),’//’);

casep=[casep sr];
else

fr=fr+1;

sr=strcat(num2str(num),’/’,num2str(1),...

’/’,num2str(answer),’//’);

casen=[casen sr];

end;

66

APPENDIX B

OPENCV CODE FOR LICENSE PLATE LOCALIZATION

67

#include "stdafx.h"

#include <iostream>
using namespace std;

#include <cv.h>

#include <cxcore.h>

#include <highgui.h>

#include <time.h>

#include <windows.h>
#include <vector>

int _tmain(int argc, _TCHAR* argv[])

{

double ts = GetTickCount();

IplImage *img1 = cvLoadImage("Input image path");

IplImage *img=cvCreateImage(cvSize(640,480),IPL_DEPTH_8U,1);
cvCvtColor( img1, img, CV_RGB2GRAY );

IplImage *out=cvCreateImage(cvGetSize(img),IPL_DEPTH_8U,1);

IplImage *out1=cvCreateImage(cvGetSize(img),IPL_DEPTH_8U,1);

CvSize c= cvGetSize(img);

CvScalar a1;

a1.val[0]=0;
IplImage *out2=cvCloneImage(img);



for(int i=0;i<c.height;i++)

{

for(int j=0;j<c.width;j++)

{

68

cvSet2D(out3,i,j,a1);

cvSet2D(out4,i,j,a1);
};

};

CvMat *imdata=cvCreateMat( 9, 9, CV_16UC1);

CvMat *matdata=cvCreateMat( 9, 9, CV_16UC1);

CvMat *outdata=cvCreateMat( 9, 9, CV_16UC1);

int it=0;
double a[]={0, 0, 0, -1, -1, -1, 0, 0, 0,

0, -1, -1, -3, -3, -3, -1, -1, 0,

0, -1, -3, -3, -1, -3, -3, -1, 0,

-1, -3, -3, 6, 13, 6, -3, -3, -1,

-1, -3, -1, 13, 24, 13, -1, -3, -1,

-1, -3, -3, 6, 13, 6, -3, -3, -1,
0, -1, -3, -3, -1, -3, -3, -1, 0,

0, -1, -1, -3, -3, -3, -1, -1, 0,

0, 0, 0, -1, -1, -1, 0, 0, 0,

};

if(i%9==0 && i>0)

cout<<endl;
else

cout << string( 3, ’ ’ );

int Mi = ((int*)(F.data.ptr + F.step*0))[i];

cout<< Mi;

};

*/

69

CvScalar s;
for(int i=4; i < c.height -4; i++ )

{

for( int j=4; j< c.width -4; j++)

{

it=0;

double svalue=0;
for(int k=-4; k<5; k++)

{

for(int l=-4;l<5;l++)

{

s=cvGet2D(img,i+k,j+l);

int n= 9;
*( (double*)CV_MAT_ELEM_PTR( *imdata, k+4, l+4 ) )=s.val[0];

int itval=(k+4)*n+(l+4);

*( (double*)CV_MAT_ELEM_PTR( *matdata, k+4, l+4 ) )=a[itval];

[(k+4)*n+(l+4)]=a[(k+4)*n+(l+4)];

*( (double*)CV_MAT_ELEM_PTR( *outdata, k+4, l+4 ) )=

a[itval]*s.val[0];
it++;

svalue=svalue+(a[itval]*s.val[0]);

};

};

CvScalar sval;

sval.val[0]=svalue;

cvSet2D(out,i,j,sval);

70

};

};
double maxo;

double mino;

double avgm;

cvMinMaxLoc(out,&mino,&maxo);

avgm=(maxo+mino)/2;

cvThreshold(out, out1, avgm, maxo ,CV_THRESH_BINARY);
double t = ((double)GetTickCount() - ts);

cout<<endl;

int pos=1;

IplConvKernel* element =

cvCreateStructuringElementEx(pos*2+1, pos*2+1, pos, pos,

CV_SHAPE_RECT);
cvDilate( out1,out2,element,1);

int label=1;

out4=out2;

cvNamedWindow("Image:",CV_WINDOW_AUTOSIZE);

int n=0;

int x=0;
CvRect rrect;

CvRect crect;

CvSize rect;

for(int i=345;i<c.height;i=i+2){

for(int j=416;j<c.width;j=j+2){

vector<int> rpt (1) ;

vector<int> cpt (1);

71

rpt[0]=i;

cpt[0]=j;
CvScalar pival;

CvScalar pival1;

pival=cvGet2D(out3,i,j);

if(pival.val[0] == 0){

while(true){

pival=cvGet2D(out3,rpt[n],cpt[n]);
pival1=cvGet2D(out4,rpt[n],cpt[n]);

if(pival1.val[0]==255.0 && pival.val[0]==0){

pival.val[0]=label;

cvSet2D(out3,rpt[n],cpt[n],pival);

if(rpt[n]+1<c.height){

pival1=cvGet2D(out4,rpt[n]+1,cpt[n]);
if(pival1.val[0]==255.0){

rpt.push_back(rpt[n]+1);

cpt.push_back(cpt[n]);

x=x+1;

};

};
if(cpt[n]-1>=0){

pival1=cvGet2D(out4,rpt[n],cpt[n]-1);


rpt.push_back(rpt[n]);

cpt.push_back(cpt[n]-1);

x=x+1;

};

72

};

if(cpt[n]+1<c.width){
pival1=cvGet2D(out4,rpt[n],cpt[n]+1);


rpt.push_back(rpt[n]);

cpt.push_back(cpt[n]+1);

x=x+1;

};
};

if(rpt[n]-1>=0){

pival1=cvGet2D(out4,rpt[n]-1,cpt[n]);


rpt.push_back(rpt[n]-1);

cpt.push_back(cpt[n]);
x=x+1;

};

};

};

if(n>=x)

break;
n=n+1;

};

const int rh = *max_element (rpt.begin(), rpt.end());

const int rl = *min_element (rpt.begin(), rpt.end());

const int ch = *max_element (cpt.begin(), cpt.end());

const int cl = *min_element (cpt.begin(), cpt.end());

if((rh-rl) <60 && (rh-rl)>20 && (ch-cl) <190 && (ch-cl)>50){

Thesis Licenseplaterecognitionincomplexscenes Fulldocument

Thesis Licenseplaterecognitionincomplexscenes Fulldocument

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Thesis Licenseplaterecognitionincomplexscenes Fulldocument

Similar to Thesis Licenseplaterecognitionincomplexscenes Fulldocument (20)

Thesis Licenseplaterecognitionincomplexscenes Fulldocument