This document describes an algorithm to solve Wordoku puzzles by processing image elements. The key steps are: 1) Separating the puzzle grid and keyword, 2) Extracting characters from each, 3) Matching characters using classifiers like cross-correlation and support vector regression, 4) Solving the puzzle by filling values in a matrix, 5) Printing the solved puzzle by pasting characters. The algorithm was tested on various puzzles and fonts, achieving a 90% accuracy rate. Extensions to handle real images and optimize character extraction are proposed for future work.

1.
WORDOKU SOLVER
Final Project Report
Surya Sekhar Chandra

2. Final
Project
Report
for
Wordoku
Solver
ii
Table of Contents
1. INTRODUCTION …………………………………………… iii
2. RELATED WORK …………………………………………. iv
3. ALGORITHM ………………………………………………. v
3.1 Steps Involved …………………………………………….. v
3.1.1 Separation ……………………………………………. v
3.1.2 Extraction …………………………………………….. vii
3.1.3 Matching ……………………………………………... ix
a) Classification Tree …………………………………… ix
b) Normalized Cross-Correlation ………………….……. x
c) Support Vector Regression ………………………….. x
3.1.4 Solving ……………………………………………..… xi
3.1.5 Printing …………………………………………….…. xii
4. TESTING/RESULTS ……………………………………..… xiv
5. EXTENDING TO REAL IMAGES ……………………….. xvi
6. CONCLUSIONS ………………………………………….… xvii
7. LIMITATIONS ………………………………………….…. xvii
8. FUTURE WORK ………………………………………..….. xvii
9. REFERENCES ……………………………………….…….. xviii
10. APPENDICES (A-O) (code) …………………….……….. xviii

3. Final
Project
Report
for
Wordoku
Solver
iii
1. INTRODUCTION:
A Wordoku is a variant of the popular number puzzle, Sudoku.
Similar to a Sudoku, a Wordoku consists of a 9x9 grid divided into nine
rows, nine columns and nine 3x3 sub-grids. In addition, a keyword is given
at the bottom of each puzzle consisting of nine different symbols or
characters. The objective of the puzzle is to fill the 9x9 grid with these
characters so that each of the nine rows, nine columns and nine 3x3 sub
grids contain all the characters from the keyword.
Fig.1 – Example of a solved Wordoku
Fig.1 shows a solved Wordoku, where the individual characters of the
keyword “WORLDGAME” were used to fill in the missing elements of the
grid with no repetitions in any row, column or a sub-grid.

4. Final
Project
Report
for
Wordoku
Solver
iv
The aim of this project is use the various image processing tools to create an
algorithm to process the puzzle elements for an unsolved Wordoku image
with any given font accurately, and to output a solved Wordoku image as
shown in Fig.1.
2. RELATED WORK:
My project falls predominantly in the category of Image extraction
and character recognition. Image extraction involves detecting and
separating out the areas of interest from a given image effectively so as to
use them for further computations. Character recognition deals with
techniques used to recognize given set of characters with good accuracy.
Most of the existing work is concentrated in developing ways to solve a
Sudoku with a higher success rate. The techniques used are the use of Hough
transforms for detecting and locating the puzzle elements in the given image.
This method seems to be efficient but requires image taken under good
lighting conditions and with no shadows as they affect the thresholding
process. The digit recognition technique employed incase of Sudoku is
either Feed-Forward Artificial networks[2] or Deep Belief Networks[3].
Both of these methods involve creating a large database for digits from 1-9
as a training set to train the neural network and identify the corresponding
matches with digits extracted from the puzzle. Deep Belief Networks are
faster and have lesser error rate compared to Feed-Forward Artificial
network.
Unlike a Sudoku, a Wordoku can consist of different symbols or characters
and thus creating a database for the purpose of recognition would not be
possible. The detection process to explore would be to extract characters
from within the image to create a dataset and match these characters with the
remaining characters with precision.

5. Final
Project
Report
for
Wordoku
Solver
v
3. ALGORITHM:
3.1 Steps Involved:
• Separation
• Extraction
• Matching
• Solving
• Printing
3.1.1 Separation -
The first step to solving a given Wordoku is to separate the keyword
and the grid to analyze them separately.
Fig. 3a - Test image of unsolved Wordoku [Source: Wikipedia]

6. Final
Project
Report
for
Wordoku
Solver
vi
Fig. 3a will be used as a test image to illustrate the various steps involved
and their outcomes.
The dimensions of the puzzle grid and hence the location of the keyword
varies from puzzle to puzzle. Therefore, it is necessary to detect the
boundary of the grid for the puzzle and extract the keyword outside the
boundary.
Convert the test image to a gray-scale image by applying a threshold. Use
the canny edge operator to detect the edges in the image. Apply a Hough
transform on this edge image to detect the 4 corners of the puzzle grid,
which correspond to the 4 peaks in the Hough image. Also, detect and draw
the lines corresponding to the boundary of the grid (Fig.3b1). (Refer to
Appendix for Matlab source code)

7. Final
Project
Report
for
Wordoku
Solver
vii
Fig.3b1 – Thresholded and boundary detected image.
Find the bottom most row value that corresponds to the bottom line of the
puzzle grid that was extracted so as to avoid detecting the keyword while
drawing lines to the whole grid. This bottom most value of the grid is stored
as maxx. Crop out the image below this bottom most value as shown in
(Fig.3b2).
Fig.3b2 – Cropped image of the keyword
3.1.2 Extraction –
Extract the individual characters from the cropped keyword image.
This can be done using the regionprops command in Matlab. However,
before extracting the characters, it is better to get rid of any small noise in
the image using imopen and imclose commands to avoid false detection.
Fig.3c - Detected characters from keyword
Fig.3d - Extracted and stored characters from keyword
The detected characters as shown in Fig.3c are extracted and separately
stored according to their location value. Thus assigning each character to a
number between 1 and 9 (Refer Fig.3d)
To the cropped grid image without the keyword, apply a hough transform to
detect the 20 grid lines (Fig3e). houghlines gives end points of the lines .
These lines as obtained are found to be in no specific order. The order of
these lines is important to us as it can be used to assign indexes to the
characters extracted from the grid as references and in-turn identify

8. Final
Project
Report
for
Wordoku
Solver
viii
Fig.3e – Houghlines detected for the entire grid
the position of the characters in the grid. The lines are sorted into a set of 10
vertical and 10 horizontal lines. It is desirable to convert these lines to a set
of lines with constant distance between them. Using the polyxpoly
command, the intersection of these sorted lines is computed. From each of
these intersection points to the next intersection points, an image is cropped
Fig.3f - Extracted and stored characters from Grid

9. Final
Project
Report
for
Wordoku
Solver
ix
out and stored separately with the reference of the intersection point as
shown in Fig3f. The region properties of each of these images is also
extracted and stored in a separate array.
3.1.3 Matching –
a) Classification tree:
b) Normalized cross-correlation
c) Support Vector Regression
d) Classification Tree:
Use the region properties obtained (Area, Centroid, Axis
lengths) for the all keyword characters to create a training dataset to train a
classification or decision tree as shown in Fig.3g
Fig.3g – A classification tree generated for the test image
This method fails and gives false detection in cases where the difference in
magnitude of properties like area or centroid between different characters is
very small.

10. Final
Project
Report
for
Wordoku
Solver
x
e) Normalized Cross-Correlation :
Each of the nine keyword characters are cross-correlated with each of
the characters extracted from the grid. The maximum scores obtained in
each case is taken as the match for that particular character. The value of that
character is assigned to the matched character. In this way, a matrix of
numbers is created with all the matching numbers assigned to each
character. Eroding the keyword characters before matching is found to give
better results. The limitation of this method is when one of the characters
could completely fit inside another; its cross-correlated value is maximum
and gives a false detection. In Fig.3h, The ‘P‘ completely fits inside ‘B’ and
hence gives a wrong match.
Fig.3h – A case of false detection
f) Support Vector Regression:
It is a regression technique [5] which maps the training data points
given in a lower dimensional space to higher dimension space, and using
Kernel functions it computes the curve that best describes the data points
and returns this function to the lower dimensional space as weights for each
dimension. With these weights, we can classify any test data point into its
corresponding class (in our case 1-9). We can use the training data points as
region properties of the keyword characters and use the region properties of
grid characters as test data points to classify them to their appropriate match
class. This method has the same limitation as the classification tree because
we are still using the data points (area, centroid and axis lengths).
Fig.3i – Weights obtained
for keyword (1-9)

11. Final
Project
Report
for
Wordoku
Solver
xi
Fig.3j – Inner products found in higher dimensional space using a Gaussian kernel
Solution: By combining the results from Support Vector Regression and
Normalized cross-correlation, we can overcome their limitation. Instead of
find a match for the maximum cross-correlated value, we can store all
matches that satisfy cross-correlated scores > 0.68 as close matches, and
then use support vector regression prediction to further narrow down the
actual match. This significantly increases our accuracy, but in-turn increases
computational time.
3.1.4 Solving –
At the corresponding location in a 9x9 matrix, the match found for
each of the character of the grid is recorded. This has reduced the Wordoku
puzzle to a Sudoku with numbers from 1-9 and 0’s indicating blanks. An
Algorithm found in the mathworks website, [4] solves this matrix recursively
using backtracking to produce a solved matrix. (Fig.3k and Fig.3l) show the
matrices obtained for the above test image.

12. Final
Project
Report
for
Wordoku
Solver
xii
Fig.3k – Unsolved Matrix
Fig.3l – Solved Matrix
3.1.5 Printing –
The solved matrix has the values indicating the location of the
keyword character that has to be pasted in that position in the grid. After
selecting the keyword character to be pasted, we have to move to the
corresponding location on the grid to paste this character to the center of that
square, as each character has a different size. The width (ws) and height (hs)
of each square can be calculated from the distances between lines. The width
(wc) and height (hc) of each character can be extracted from the size of that
image. The position to paste the image in that square can be calculated as,
X = (ws /2) - (wc-/2)
Y = (hs/2) - (hc/2)
From this starting position, we can use the pixels where the keyword has
black color as a reference to change the corresponding pixel value in that
square to our desired color. In this way, the keyword as obtained in the
solved matrix can be precisely pasted to the center.

13. Final
Project
Report
for
Wordoku
Solver
xiii
Fig.3m – Image
showing the pasting
method to the center
Fig.3n – Pasting all
the characters in the
grid

14. Final
Project
Report
for
Wordoku
Solver
xiv
Final Solution:
Fig.3o – Image of the unsolved and solved Wordoku, solved using the above
algorithm
4. TESTING/RESULTS:
Fig.4a – Result for the second test image

15. Final
Project
Report
for
Wordoku
Solver
xv
The algorithm was tested with cases where there are lines missing from the
given puzzle. It could still extract the elements to solve the puzzle and paste
the solution to the correct locations.
Fig.4b – Result for the test image with missing lines.
In this next case, the algorithm was tested with a keyword containing greek
alphabets, but the algorithm could handle this case without any problem.
Fig.4c – Result for the test image with greek symbols

16. Final
Project
Report
for
Wordoku
Solver
xvi
5. EXTENDING TO REAL IMAGES:
Mobile shots of a puzzle were taken from different angles as test
image. The first was to manually select the corners of the puzzle and project
it on to a square template using cpselect.
Fig.5a – Using cpselect on the mobile shot and projecting onto a square template
This causes thesholding issues and the houglines has problems detecting the
lines of the grid.
Fig.5b – houghlines was unable to detect the lines of the grid

17. Final
Project
Report
for
Wordoku
Solver
xvii
Solution: As we are projecting the puzzle onto a square template, we can just
divide the square into 10 horizontal and 10 vertical equal parts and extract
the characters from the grid. The keyword has to be separately extracted.
This method worked for one out of the two images that were tested. The
matching for second imaged failed due to thresholding issues.
6. CONCLUSIONS:
. Algorithm was able to handle a tilt of the puzzle up to 10 degrees without
any user interaction.
. It was able to solve the puzzle for any given font.
. It could solve the puzzle even when some lines were missing in the puzzle.
. It gave a 90 percent (10/11) accuracy. It failed in the case where there was
a size mismatch.
7. LIMITATIONS:
. Keyword characters needed to be separated before extraction.
. It could not handle real images without user interaction.
. Noise/thresholding issues for real images.
8. FUTURE WORK:
. To optimize the algorithm to handle varying size of the keyword and
puzzle elements.
. To extend this algorithm to solve for real images with thresholding
issues.

18. Final
Project
Report
for
Wordoku
Solver
xviii
9. REFERENCES:
[1] Otsu, N., "A Threshold Selection Method from Gray- Level
Histograms," IEEE Transactions on Systems, Man, and Cybernetics, Vol. 9,
No. 1, 1979, pp. 62-66.
[2] A. Van Horn, “Extraction of sudoku puzzles using the hough transform,”
2012.
[3] Wicht, Baptiste, and Jean Hennebert. "Camera-based Sudoku recognition
with Deep Belief Network."
[4] “Recursive Sudoku Solver in MATLAB” by Josin, 11 Nov 2013,
http://www.mathworks.com/matlabcentral/fileexchange/44272-recursive-
sudoku-solver-in-matlab/content/miniSudokuSolver.m
[5] Code for “Support Vector Regression” by Ronnie Clark, 10 Sep 2013,
http://www.mathworks.com/matlabcentral/fileexchange/43429-support-
vector-regression
10. APPENDIX:
A.
%% Wordoku-Solver MAIN PROGRAM ROUTINE
%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%
clear all
close all
% Select the puzzle
I = imread('wordoku_puzzle1.png');
%I = imread('wordoku_puzzle2.png');
%I = imread('wordoku_puzzle3.png');
%I = imread('wordoku_weird2.png');
orig = zeros(size(I,1),size(I,2),3);
orig = I;

19. Final
Project
Report
for
Wordoku
Solver
xix
%% Set the desired flags(=1) to display the corresponding
outputs/images
dispOriginal = 1;
dispLinesDrawn = 0; % Image with all the lines
drawn
dispKeyword = 1; % Image with the keyword
extracted
dispIndvKeys = 0; % Display the keys indvidually
dispCroppedPuzzleWithLines = 0;
dispNewLinesforCroppedPuzzle = 0;
dispAllExtractedElements = 0;
dispTree = 0;
dispUnsolvedSudoku = 0;
dispSolvedSudoku = 0;
dispDebugPasteSolution = 0;
dispFinalSolvedImage = 0;
dispBothSolvedAndUnsolved = 1;
%
useTrees = 0;
%or
useSupportVectors = 0;
%or
useCrossCorr = 1;
valueGrayThresholdCutoff = 100; %Ranges from 0 - 255
valueErrorInclineStLine = 5; % Error or incline estimate of
st.line
USE_ERODE = 0; %SET THE RADIUS OF
STRUCTURING ELEMENT
%% TO CREATE A FUNCTION FOR EVERYTHING BELOW
%flag - stacks all the flags and sends them to solve
flag = zeros(1,30);
flag = [dispLinesDrawn dispKeyword dispIndvKeys ...
dispCroppedPuzzleWithLines dispNewLinesforCroppedPuzzle ...
dispAllExtractedElements dispTree dispUnsolvedSudoku...
dispSolvedSudoku dispDebugPasteSolution
dispFinalSolvedImage...
dispBothSolvedAndUnsolved];
%
value = zeros(1,30);
value = [useTrees useSupportVectors useCrossCorr ...
valueGrayThresholdCutoff valueErrorInclineStLine USE_ERODE];
%Recieve flags and values to display appropriate images as desired
dispLinesDrawn = flag(1);
dispKeyword = flag(2);
dispIndvKeys = flag(3);
dispCroppedPuzzleWithLines = flag(4);
dispNewLinesforCroppedPuzzle = flag(5);

20. Final
Project
Report
for
Wordoku
Solver
xx
dispAllExtractedElements = flag(6);
dispTree = flag(7);
dispUnsolvedSudoku = flag(8);
dispSolvedSudoku = flag(9);
dispDebugPasteSolution = flag(10);
dispFinalSolvedImage = flag(11);
dispBothSolvedAndUnsolved = flag(12);
useTrees = value(1);
useSupportVectors = value(2);
useCrossCorr = value(3);
valueGrayThresholdCutoff = value(4);
valueErrorInclineStLine = value(5);
USE_ERODE = value(6);
%% START OF MAIN PROGRAM
I = rgb2gray(I);
if(dispOriginal ==1)
figure,imshow(I,[]),impixelinfo;
title('Original puzzle image');
end
%Thresholding and detecting Edges,
%Takes : Image,gray value to cutoff
%Returns : Edge Image and bw image
[E,bw] = Threshold_Image(I,valueGrayThresholdCutoff);
%Finding just four corners of the puzzle
%for the four corners of the puzzle , finding the bottom most value
%of x , so we don't detect the keyword while drawing lines
%Takes : bwImage
%Returns : Only the hough lines for the 4 boundaries and max cutoff
...
% to avoid the keyword
[lines,maxx] = Boundary_Maxcutoff(bw);
Draw_Lines(maxx,lines,bw,dispLinesDrawn);
%Finding peaks for all the lines of the puzzle
%Takes : bwImage
%Returns : All the 20 hough lines i.e., for all the lines in the
puzzle
lines1 = Give_All_Lines(bw);
%Draw all the lines on the image specified

21. Final
Project
Report
for
Wordoku
Solver
xxi
%Takes : maxx,lines and image; only if 1 => display the new image
%Returns: if 1 then it shows the new image, else, no display.
Draw_Lines(maxx,lines1,bw,dispLinesDrawn);
%Locate the keyword in the image, extract individuals and store them
%Takes : bwimage,maxx,and dispKeyword,dispIndvKeys which are flags
%Returns : The regionprops of keys, number of keys and set of indv
%images
[keys,num,im] =
Keyword(bw,maxx,dispKeyword,dispIndvKeys,USE_ERODE);
%Crop the puzzle part away from keyword and extract lines
P = Crop_Puzzle(bw,maxx);
%Get lines for the cropped puzzle above
lines2 = Give_All_Lines(P);
Draw_Lines(maxx,lines2,P,dispCroppedPuzzleWithLines);
% % % % %SORT LINES AND GET INFO % % % % %
%The points extracted in the array are all random
%We want to get the indices of the 10 vertical and 10 horizontal lines
in
%increasing order from the origin(top-left corner)
%Also, the Width and height of each block varies..
%orderVertical and orderHorizontal are the indices sorted in order
%WID and HEI are arrays of all the widths and heights of size 9
each
%width, height are the average of all
%widthInBetw and heightInBetw are the middle values between the
thin
%lines instead of thick lines, so we can draw a new set of
lines with
%constant width
[orderVertical orderHorizontal WID HEI...
width height widthInBetw heightInBetw]...
= SortLines_GetInfo(lines2,valueErrorInclineStLine);
%%
%NEW LINES WITH THE MODIFIED WIDTH AND HEIGHT
lines2new = GetNewLines(lines2,valueErrorInclineStLine);
Draw_Lines(maxx,lines2new,P,dispNewLinesforCroppedPuzzle);
[orderVertical orderHorizontal WID HEI...
width height widthInBetw heightInBetw]...
= SortLines_GetInfo(lines2new,valueErrorInclineStLine);
Original = 0;
[intersectX, intersectY,...
puz , numberOfElementsAt , mainRegionProperties ,
elementProperties, Offset ]...
= Extract_Elements_Grid(bw,lines2new,WID,HEI,...

22. Final
Project
Report
for
Wordoku
Solver
xxii
orderVertical,orderHorizontal,dispAllExtractedElements,Original,USE_ERO
DE);
Original = 1;
[intersectXxx, intersectYxx,...
puz , numberOfElementsxx , mainRegionPropertiesxx ,
elementPropertiesxx, Offsetxx ]...
= Extract_Elements_Grid(I,lines2new,WID,HEI,...
orderVertical,orderHorizontal,dispAllExtractedElements,Original,USE_ERO
DE);
%%
%Find matches and return a number
Num_Matrix =
Find_Matches(keys,elementProperties,numberOfElementsAt,...
dispTree,im,puz,useTrees,useSupportVectors,useCrossCorr);
Unsolved_Sudoku = Num_Matrix;
if(dispUnsolvedSudoku == 1)
Unsolved_Sudoku
end
%Sudoku algorithm
S = miniSudokuSolver(Num_Matrix);
Solved_Sudoku = S;
if(dispSolvedSudoku == 1)
Solved_Sudoku
end
%%
%Form
Original = 1;
[intersectX, intersectY,...
puz , numberOfElements , mainRegionProperties , elementProperties,
Offset ]...
= Extract_Elements_Grid(I,lines2new,WID,HEI,...
orderVertical,orderHorizontal,dispAllExtractedElements,Original,USE_ERO
DE);
%% FIND INTERSECTS
intersectX;intersectY;
X = lines2new;
x11 = cellfun(@(X) X(1,1), {X(orderVertical).point1});
y11 = cellfun(@(X) X(1,2), {X(orderVertical).point1});
x12 = cellfun(@(X) X(1,1), {X(orderVertical).point2});
y12 = cellfun(@(X) X(1,2), {X(orderVertical).point2});
x21 = cellfun(@(X) X(1,1), {X(orderHorizontal).point1});
y21 = cellfun(@(X) X(1,2), {X(orderHorizontal).point1});

23. Final
Project
Report
for
Wordoku
Solver
xxiii
x22 = cellfun(@(X) X(1,1), {X(orderHorizontal).point2});
y22 = cellfun(@(X) X(1,2), {X(orderHorizontal).point2});
for i=1:10
for j=1:10
x1 = [x12(j); x12(j)];
y1 = [y11(j)-3; y12(j)+3];
x2 = [x21(i)-10; x22(i)+10];
y2 = [y21(i); y22(i)];
[xi,yi] = polyxpoly(x1,y1,x2,y2);
intersectXnew(i,j) = round(xi);
intersectYnew(i,j) = round(yi);
end
end
%% PASTE KEYWORDS AND SHOW THE SOLUTION
[NEW_FULLSIZE_KEYS_ORIGINAL,bwVersion] =
Get_Fullsize_keys(S,Num_Matrix,...
puz);
[ImageFinal,ImageFinalColor] =
PasteKeys(orig,I,bw,NEW_FULLSIZE_KEYS_ORIGINAL,bwVersion,...
Num_Matrix,Offset,WID,HEI,intersectXnew,
intersectYnew,S,im,dispDebugPasteSolution);
if(dispFinalSolvedImage==1)
figure,imshow(ImageFinal,[]);
title('Solved puzzle image');
end
if(dispBothSolvedAndUnsolved==1)
figure,subplot(1,2,1),imshow(orig,[]),title('UNSOLVED');
subplot(1,2,2),imshow(ImageFinalColor,[]),title('SOLVED');
end
%%%%%%%%%%%%%%%%%%%%%%
%MAIN PROGRAM ENDS
%%%%%%%%%%%%%%%%%%%%%
B.
%% Boundary_Maxcutoff sub-routine
%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%

24. Final
Project
Report
for
Wordoku
Solver
xxiv
function [Puzzle_Boundary Max_Dist] = Boundary_Maxcutoff(bw)
E = edge(bw,'canny',[]);
[H t r] = hough(E);
peaks = houghpeaks(H,4);
lines = houghlines(E,t,r,peaks);
maxx = 0;
% for the four corners of the puzzle , finding the bottom most value
% of x , so we don't detected the keyword while drawing lines
for k = 1:length(lines)
ix = (lines(k).point1(1,2));
if(ix>maxx)
maxx = ix;
end
end
Puzzle_Boundary = lines;
Max_Dist = maxx;
end
%%%%%%%%%%%%%%%%%%%%%%
%Boundary_Maxcutoff subprogram ends
%%%%%%%%%%%%%%%%%%%%%
C.
%% Crop_Puzzle sub-routine
%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%
function P = Crop_Puzzle(bw,maxx);
P = imcrop(bw, [1,1,size(bw,2),maxx+2*round(size(bw,2)/300)]);
end
%%%%%%%%%%%%%%%%%%%%%%
%Crop_Puzzle sub-routine ends
%%%%%%%%%%%%%%%%%%%%%
D.
%% Draw_Lines sub-routine
%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%

25. Final
Project
Report
for
Wordoku
Solver
xxv
function Draw_Lines(maxx,liness,bw,disp)
if(disp==1)
figure,imshow(bw,[]),impixelinfo;
hold on;
for k = 1:length(liness)
xy = [liness(k).point1; liness(k).point2];
if (xy(1,2) > maxx)
xy(1,2) = maxx;
end
if (xy(2,2) > maxx)
xy(2,2) = maxx;
end
plot(xy(:,1),xy(:,2),'LineWidth',2,'Color','green');
% Plot beginnings and ends of lines
plot(xy(1,1),xy(1,2),'x','LineWidth',2,'Color','yellow');
plot(xy(2,1),xy(2,2),'x','LineWidth',2,'Color','red');
end
title('Hough Lines');
hold off;
end
end
%%%%%%%%%%%%%%%%%%%%%%
%Draw_Lines sub-routine ends
%%%%%%%%%%%%%%%%%%%%%
E.
%% Extract_Elements_Grid sub-routine
%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%
function [intersectX, intersectY,...
puz , numberOfElementsAt , regionProperties , elementProperties,
Offset ]...
=
Extract_Elements_Grid(bw,lines2,WID,HEI,orderVertical,orderHorizontal,.
..
dispAllExtractedElements,Original,USE_ERODE)
if dispAllExtractedElements == 1
figure
end
for oh = 1:1:9
kh = orderHorizontal(oh);

26. Final
Project
Report
for
Wordoku
Solver
xxvi
hz = [lines2(kh).point1; lines2(kh).point2];
for ov = 1:1:9
k = orderVertical(ov);
xy = [lines2(k).point1; lines2(k).point2];
x1 = xy(:,1);
y1 = xy(:,2);
x2 = hz(:,1);
y2 = hz(:,2);
[xi,yi] = polyxpoly(x1,y1,x2,y2);
X = lines2;
x11 = cellfun(@(X) X(1,1), {X(orderVertical(ov)).point1});
y11 = cellfun(@(X) X(1,2), {X(orderVertical(ov)).point1});
x12 = cellfun(@(X) X(1,1), {X(orderVertical(ov)).point2});
y12 = cellfun(@(X) X(1,2), {X(orderVertical(ov)).point2});
x21 = cellfun(@(X) X(1,1), {X(orderHorizontal(oh)).point1});
y21 = cellfun(@(X) X(1,2), {X(orderHorizontal(oh)).point1});
x22 = cellfun(@(X) X(1,1), {X(orderHorizontal(oh)).point2});
y22 = cellfun(@(X) X(1,2), {X(orderHorizontal(oh)).point2});
x1 = [x12(1); x12(1)];
y1 = [y11(1)-3; y12(1)+3];
x2 = [x21(1)-10; x22(1)+10];
y2 = [y21(1); y22(1)];
[xi,yi] = polyxpoly(x1,y1,x2,y2);
intersectX(oh,ov) = round(xi);
intersectY(oh,ov) = round(yi);
if(ov<10 && oh <10)
sW = WID(ov);
sH = HEI(oh);
osW = 3*round(sW/25);
osH = 2*round(sH/25);
sizeW = WID(ov)-2*osW;
sizeH = HEI(oh)-2*osH;
Offset{oh,ov} = [osW,osH;sizeW,sizeH];
EX = imcrop(bw, [xi+osW,yi+osH,sizeW,sizeH]);
if(Original==1)
EX = im2bw(EX);
puz{oh,ov} = EX;

27. Final
Project
Report
for
Wordoku
Solver
xxvii
pz = EX;
numberOfElementsAt(oh,ov)=0;
regionProperties{oh,ov} = 0;
elementProperties{oh,ov} =0;
end
if(Original ==0)
[L2 , n2] = bwlabel(EX);
regionProperties{oh,ov} =
regionprops(L2,'centroid','area','solidity','convexArea','majorAxisLeng
th','minorAxisLength');
if USE_ERODE >0
if n2~=0
s = strel('disk',USE_ERODE);
EX = imerode(EX,s);
EX = imclose(EX,s);
[L2 , n2] = bwlabel(EX);
regionProperties{oh,ov} =
regionprops(L2,'centroid','area','solidity','convexArea','majorAxisLeng
th','minorAxisLength');
end
end
[r,c] = find(L2);
pz=EX(min(r):max(r),min(c):max(c));
puz{oh,ov} = pz;
numberOfElementsAt(oh,ov)=n2;
[L3 , n3] = bwlabel(pz);
elementProperties{oh,ov} =
regionprops(L3,'centroid','area','solidity','convexArea','majorAxisLeng
th','minorAxisLength');
end
end
end
end
if dispAllExtractedElements == 1
cou = 1;
for q = 1:9
for w=1:9
subplot(9,9,cou),imshow(~puz{q,w},[]);
title(sprintf('%d,%d',q,w));
cou = cou +1;
end

28. Final
Project
Report
for
Wordoku
Solver
xxviii
end
end
end
%%%%%%%%%%%%%%%%%%%%%%
%Extract_Elements_Grid sub-routine ends
%%%%%%%%%%%%%%%%%%%%%
F.
%% Find_Matches sub-routine
%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%
function nMat =
Find_Matches(props1,props2,n,dispTree,im,puz,useTrees,useSupportVectors
,useCrossCorr)
for i=1:length(props1)
X(i,1) = props1{i}.Area;
%X(i,6) = props1{i}.Centroid(1,1);
%X(i,5) = props1{i}.Centroid(1,2);
%'centroid','area','solidity','convexArea','majorAxisLength','minorAxis
Length'
X(i,4) = props1{i}.Solidity;
X(i,7) = props1{i}.ConvexArea;
X(i,2) = props1{i}.MajorAxisLength;
X(i,3) = props1{i}.MinorAxisLength;
y(i,1) = i; % class label
end
ctree = ClassificationTree.fit(X,y,'MinParent',1);
if(dispTree == 1)
view(ctree, 'mode', 'graph');
end
nMat = n;
[row,col]=find(n~=0);
for i =1:size(row)
xTest(i,1) = props2{row(i),col(i)}.Area;
%xTest(i,6) = props2{row(i),col(i)}.Centroid(1,1);
%xTest(i,5) = props2{row(i),col(i)}.Centroid(1,2);
xTest(i,4) = props2{row(i),col(i)}.Solidity;
xTest(i,7) = props2{row(i),col(i)}.ConvexArea;
xTest(i,2) = props2{row(i),col(i)}.MajorAxisLength;
xTest(i,3) = props2{row(i),col(i)}.MinorAxisLength;
end

29. Final
Project
Report
for
Wordoku
Solver
xxix
if(useTrees==1)
class = predict(ctree, xTest);
end
%% support vector
if(useSupportVectors == 1)
svrobj = svr_trainer(X,y,1000,0.0000025,'gaussian',.0001);
cl = svrobj.predict(xTest);
keyy = svrobj.predict(X);
for j=1 : size(cl)
for k =1:size(keyy)
if((keyy(k) - cl(j))^2 < 0.2)
class(j) = k ;
end
end
end
keyy;
end
%% cross corr
if(useCrossCorr == 1)
for k =1:size(im,2)
for l=1:size(row)
T = im{k};
strell = strel('disk',1);
T = imopen(T,strell);
T = imclose(T,strell);
I2 = ~puz{row(l),col(l)};
k;
row(l);
col(l);
C = normxcorr2(T,I2);
%figure,imshow(C,[]);
% The scores are in an image that is slightly bigger than
the original
% image ... it is expanded by half the size of the template
in all
% directions. So we will crop out the center portion.
Csub = imcrop(C, [(size(T,2)-1)/2+1 (size(T,1)-1)/2+1
size(I2,2)-1 size(I2,1)-1]);
%figure,imshow(Csub,[]),impixelinfo;
cmax(l,k) = max(Csub(:));
end
end
for l=1:size(row)
index = find(cmax(l,:)==max(cmax(l,:)));
class(l)=index;
end

30. Final
Project
Report
for
Wordoku
Solver
xxx
end
%%
for j=1:size(row)
nMat(row(j),col(j)) = class(j);
end
end
%%%%%%%%%%%%%%%%%%%%%%
%Find_Matches sub-routine ends
%%%%%%%%%%%%%%%%%%%%%
G.
%% Get_Fullsize_keys sub-routine
%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%
function [New_Keys,bwVersion] = Get_Fullsize_keys(S,Num_Matrix,...
puz)
[r,c] = find(Num_Matrix~=0);
for i=1:9
for j=1:size(r)
if S(r(j),c(j)) == i
New_Keys{i} = puz{r(j),c(j)};
bwVersion = ~New_Keys{i};
end
end
end
end
%%%%%%%%%%%%%%%%%%%%%%
%Get_Fullsize_keys sub-routine ends
%%%%%%%%%%%%%%%%%%%%%
H.
%% GetNewLines sub-routine
%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%

31. Final
Project
Report
for
Wordoku
Solver
xxxi
function lines2new = GetNewLines(lines2,error)
[orderVertical orderHorizontal WID HEI...
width height widthInBetw heightInBetw] =
SortLines_GetInfo(lines2,error);
lines2new = lines2;
lines2new(orderVertical(1)).point1(1,1) ...
= lines2new(orderVertical(2)).point1(1,1) - WID(2);
lines2new(orderVertical(1)).point2(1,1) ...
= lines2new(orderVertical(2)).point2(1,1) - WID(2);
for i = 4:3:9
lines2new(orderVertical(i)).point1(1,1) ...
= lines2new(orderVertical(i-1)).point1(1,1) +
round((WID(i)+WID(i-1))/2);
lines2new(orderVertical(i)).point2(1,1) ...
= lines2new(orderVertical(i-1)).point2(1,1) +
round((WID(i)+WID(i-1))/2);
end
lines2new(orderHorizontal(1)).point1(1,2) ...
= lines2new(orderHorizontal(2)).point1(1,2) - HEI(2);
lines2new(orderHorizontal(1)).point2(1,2) ...
= lines2new(orderHorizontal(2)).point2(1,2) - HEI(2);
for i = 4:3:9
lines2new(orderHorizontal(i)).point1(1,2) ...
= lines2new(orderHorizontal(i-1)).point1(1,2) +
round((HEI(i)+HEI(i-1))/2);
lines2new(orderHorizontal(i)).point2(1,2) ...
= lines2new(orderHorizontal(i-1)).point2(1,2) +
round((HEI(i)+HEI(i-1))/2);
end
end
%%%%%%%%%%%%%%%%%%%%%%
%GetNewLines sub-routine ends
%%%%%%%%%%%%%%%%%%%%%
I.
%% Give_All_Lines sub-routine
%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%

32. Final
Project
Report
for
Wordoku
Solver
xxxii
function lines1 = Give_All_Lines(image)
E = edge(image,'canny',[]);
% E= im2bw(image);
[H t r] = hough(E);
peaks1 = houghpeaks(H,20);
lines1 = houghlines(E,t,r,peaks1);
end
%%%%%%%%%%%%%%%%%%%%%%
%Give_All_Lines sub-routine ends
%%%%%%%%%%%%%%%%%%%%%
J.
%% Keyword sub-routine
%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%
function [keysProp,n,im] =
Keyword(bw,maxx,dispKeyword,dispIndvKeys,USE_ERODE)
J = imcrop(bw, [1,maxx+2*round(size(bw,2)/300),size(bw,2),size(bw,1)]);
if(USE_ERODE > 0)
%NEW
s = strel('disk',USE_ERODE);
J = imerode(J,s);
J = imclose(J,s);
%NEW END
[L , n] = bwlabel(J);
end
[L , n] = bwlabel(J);
keys =
regionprops(L,'boundingBox','centroid','area','solidity','convexArea','
majorAxisLength','minorAxisLength');
rect = zeros(1,n);
if dispKeyword==1
figure,imshow(J,[]);
for i=1:length(keys)

33. Final
Project
Report
for
Wordoku
Solver
xxxiii
rect=rectangle('Position',keys(i).BoundingBox,'EdgeColor','r');
end
title('Keyword from the bottom of the puzzle')
end
for j=1:n;
[r,c] = find(L==j);
im{j}=J(min(r):max(r),min(c):max(c));
keysProp{j} =
regionprops(im{j},'boundingBox','centroid','area','solidity','convexAre
a','majorAxisLength','minorAxisLength');
end
if(dispIndvKeys==1)
figure,
for q=1:9
subplot(1,9,q),imshow(im{q},[])
title(sprintf('Key %d',q));
end
end
end
%%%%%%%%%%%%%%%%%%%%%%
%Keyword sub-routine ends
%%%%%%%%%%%%%%%%%%%%%
K.
[ http://www.mathworks.com/matlabcentral/fileexchange/44272-recursive-
sudoku-solver-in-matlab/content/miniSudokuSolver.m ]
function solved = miniSudokuSolver(sud)
% Solves Sudoku recursively! Input is a 9x9 grid with zeros as spaces.
% Josip S - 11/11/13. Man, I should be studying.
% If it's empty, return.
if isempty(sud), solved = []; return, end
% Returns the indicies where we can guess/put in numbers.
[i, j] = find(sud == 0);
% If there are no empty squares, we're done!
if isempty(i),solved = sud; return,end
% Has no result by default.
solved = [];

34. Final
Project
Report
for
Wordoku
Solver
xxxiv
% Finds a good square spot to start guessing! (<=2 is good.)
% gridposs = the remaining numbers it could be from the numbers in the
% 3x3 grid. horzVertPos = the remaining numbers it could be numbers
above/below
% and beside the ith and jth square. allPoss is the common numbers of
these two.
for leastGuessIndx = 1:length(i)
% Old code
% gridposs = setxor(sud(ceil(i(leastGuessIndx)/3)*3-
2:ceil(i(leastGuessIndx)/3)*3, ceil(j(leastGuessIndx)/3)*3-
2:ceil(j(leastGuessIndx)/3)*3), (0:9));
% horzVertPoss = intersect(setxor(sud(i(leastGuessIndx), :),(0:9)),
setxor(sud(:, j(leastGuessIndx)),(0:9)));
% allGuesses = intersect(gridposs, horzVertPoss)';
% New code equivalent. (~14x faster, no setxoring.)
gridposs = reshape(sud(ceil(i(leastGuessIndx)/3)*3-
2:ceil(i(leastGuessIndx)/3)*3, ceil(j(leastGuessIndx)/3)*3-
2:ceil(j(leastGuessIndx)/3)*3), 1, 9);
horzVertPoss = [sud(i(leastGuessIndx), :) sud(:,
j(leastGuessIndx))']; set=(0:9);
allGuesses = set(~ismember(set, [gridposs horzVertPoss]));
if isempty(allGuesses), return, end % If any 0 has no possible
moves, there's a contradiction.
if length(allGuesses) <= 2, break, end % If there are less that 2
valid guesses, use that.
end
% Takes a guess from each of the valid possibilities. Recursively calls
% itself on the new guess.
for guess = allGuesses
sud(i(leastGuessIndx), j(leastGuessIndx)) = guess;
result = miniSudokuSolver(sud);
if ~isempty(result), solved = result; return; end
end end
L.
%% PasteKeys sub-routine
%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%
function [ImageFinal,ImageFinalColor] =
PasteKeys(orig,I,bw,New_Keys,bwVersion...
,Num_M,Offset,WID,HEI,intersectX,
intersectY,S,keys,dispDebugPasteSolution)
ImageFinal = I;
ImageFinalColor = orig;
[r,c] = find(Num_M==0);

35. Final
Project
Report
for
Wordoku
Solver
xxxv
if(dispDebugPasteSolution==1)
figure,
end
for i=1:size(r)
ix = r(i);
iy = c(i);
val = S(ix,iy);
x = uint16(intersectX(ix,iy));
y = uint16(intersectY(ix,iy));
if(ix+1 < 10 && iy+1 < 10)
dy = uint16(intersectX(ix,iy+1))-uint16(intersectX(ix,iy));
dx = uint16(intersectY(ix+1,iy))-uint16(intersectY(ix,iy));
else
dx = dx;
dy = dy;
end
him = size(keys{val} ,1);
lim = size(keys{val} ,2);
osH = round((dx-him)/2);
osW = round((dy-lim)/2);
for m=1:size(keys{val},1)
for n=1:size(keys{val},2)
if(ix~=iy)
if((keys{val}(m,n)) > 0)
ImageFinal(y+osH+m,x+osW+n)= 0;
ImageFinalColor(y+osH+m,x+osW+n,:)= [0 55 255];
%pause(.00001);
end
end
if(ix==iy)
if((keys{val}(m,n)) >0)
ImageFinal(y+osH+m,x+osW+n)= 0;
ImageFinalColor(y+osH+m,x+osW+n,:)= [0 55 255];
%pause(.00001);
end
end
end
end
if(dispDebugPasteSolution==1)
imshow(ImageFinal),impixelinfo
%pause(.5);
hold on

36. Final
Project
Report
for
Wordoku
Solver
xxxvi
plot(x,y,'x','LineWidth',2,'Color','red');
plot(x,y+dy,'x','LineWidth',2,'Color','green');
plot(x+dx,y,'x','LineWidth',2,'Color','green');
plot(x+osW,y+osH,'x','LineWidth',2,'Color','yellow');
plot(x+osW+n,y+osH+m,'x','LineWidth',2,'Color','yellow');
pause(0.2)
title('DEBUGGING THE PASTE METHOD')
end
if(dispDebugPasteSolution==1)
hold off
end
end
%%%%%%%%%%%%%%%%%%%%%%
%PasteKeys sub-routine ends
%%%%%%%%%%%%%%%%%%%%%
M.
%% SortLines_GetInfo sub-routine
%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%
function [orderVertical orderHorizontal WIDarray HEIarray...
WidthAvg HeightAvg widthInBetw heightInBetw] =
SortLines_GetInfo(liness,error)
%Sort the lines or get their indices
for i=1:20
lines_array(i,1) = liness(i).point1(1,1);
lines_array(i,2) = liness(i).point1(1,2);
lines_array2(i,1) = liness(i).point2(1,1);
lines_array2(i,2) = liness(i).point2(1,2);
end
indexV = find( abs(lines_array(:,1) - lines_array2(:,1)) < error );
indexH = find( abs(lines_array(:,2) - lines_array2(:,2)) < error );
Vsort= sort(lines_array(indexV),'ascend');
newVind = zeros(size(Vsort));
for m=1:size(Vsort)
val = Vsort(m);
newVind(m,1) = find(lines_array(indexV(:),1) == val) ;
end

37. Final
Project
Report
for
Wordoku
Solver
xxxvii
lines_array(indexV(newVind(:)));
Hsort= sort(lines_array(indexH,2),'ascend');
newHind = zeros(size(Hsort));
for m=1:size(Hsort)
val = Hsort(m);
newHind(m,1) = (find(lines_array(indexH(:),2) == val)) ;
end
orderVertical = indexV(newVind(:));
orderHorizontal = indexH(newHind(:));
%%
width =0;
for aa=1:size(orderVertical,1)-1
width = width + abs(lines_array(orderVertical(aa),1) ...
- lines_array(orderVertical(aa+1),1));
WID(aa)=abs(lines_array(orderVertical(aa),1) ...
- lines_array(orderVertical(aa+1),1));
end
width = round(width/(size(orderVertical,1)-1));
height =0;
for aa=1:size(orderHorizontal,1)-1
height = height + abs(lines_array(orderHorizontal(aa),2) ...
- lines_array(orderHorizontal(aa+1),2));
HEI(aa)=abs(lines_array(orderHorizontal(aa),2) ...
- lines_array(orderHorizontal(aa+1),2));
end
height = round(height/(size(orderHorizontal,1)-1));
WidthAvg = width;
HeightAvg = height;
WIDarray = WID;
HEIarray = HEI;
widthInBetw = WID(2);
heightInBetw = HEI(2);
end
%%%%%%%%%%%%%%%%%%%%%%
%SortLines_GetInfo sub-routine ends
%%%%%%%%%%%%%%%%%%%%%
N.

38. Final
Project
Report
for
Wordoku
Solver
xxxviii
[ http://www.mathworks.com/matlabcentral/fileexchange/43429-support-
vector-regression ]
function svrobj = svr_trainer(xdata,ydata, c, epsilon, kernel,
varargin)
% SVR Utilises Support Vector Regression to approximate
% the functional relationship from which the
% the training data was generated.
% Function call:
%
% svrobj = svr_trainer(x_train,y_train,c,epsilon,kernel,varargin);
% The training data, x_train and y_train must be column vectors.
%
% Example usage:
%
% svrobj =
svr_trainer(x_train,y_train,400,0.000000025,'gaussian',0.5);
% y = svrobj.predict(x_test);
%
if strcmp(kernel,'gaussian')
lambda = varargin{1};
kernel_function = @(x,y) exp(-lambda*norm(x.feature-
y.feature,2)^2);
elseif strcmp(kernel,'spline')
kernel_function = @(a,b) prod(arrayfun(@(x,y) 1 +
x*y+x*y*min(x,y)-
(x+y)/2*min(x,y)^2+1/3*min(x,y)^3,a.feature,b.feature));
elseif strcmp(kernel,'periodic')
l = varargin{1};
p = varargin{2};
kernel_function = @(x,y) exp(-2*sin(pi*norm(x.feature-
y.feature,2)/p)^2/l^2);
elseif strcmp(kernel,'tangent')
a = varargin{1};
c = varargin{2};
kernel_function = @(x,y) prod(tanh(a*x.feature'*y.feature+c));
end
ntrain = size(xdata,1);
alpha0 = zeros(ntrain,1);
for i=1:ntrain
for j=1:ntrain
xi(i,j).feature = xdata(i,:);
xj(i,j).feature = xdata(j,:);
end
end
% *********************************
% Set up the Gram matrix for the
% training data.
% *********************************
M = arrayfun(kernel_function,xi,xj);
M = M + 1/c*eye(ntrain);

39. Final
Project
Report
for
Wordoku
Solver
xxxix
% *********************************
% Train the SVR by optimising the
% dual function ie. find a_i's
% *********************************
% options = optimoptions('quadprog','Algorithm','interior-point-
convex');
options = optimset('Algorithm','interior-point-convex');
H = 0.5*[M zeros(ntrain,3*ntrain); zeros(3*ntrain,4*ntrain)];
figure; imagesc(M); title('Inner product between training data (ie.
K(x_i,x_j)'); xlabel('Training point #'); ylabel('Training point #');
lb = [-c*ones(ntrain,1); zeros(ntrain,1); zeros(2*ntrain,1)];
ub = [ c*ones(ntrain,1); 2*c*ones(ntrain,1);
c*ones(2*ntrain,1)];
f = [ -ydata;
epsilon*ones(ntrain,1);zeros(ntrain,1);zeros(ntrain,1)];
z = quadprog(H,f,[],[],[],[],lb,ub,[],options);
alpha = z(1:ntrain);
figure; stem(alpha); title('Visualization of the trained SVR');
xlabel('Training point #'); ylabel('Weight (ie. alpha_i - alpha_i^*)');
% *********************************
% Calculate b
% *********************************
for m=1:ntrain
bmat(m) = ydata(m);
for n = 1:ntrain
bmat(m) = bmat(m) - alpha(n)*M(m,n);
end
bmat(m) = bmat(m) - epsilon - alpha(m)/c;
end
b = mean(bmat);
% *********************************
% Store the trained SVR.
% *********************************
svrobj.alpha = alpha;
svrobj.b = b;
svrobj.kernel = kernel_function;
svrobj.train_data = xdata;
svrobj.predict = @(x) cellfun(@(u) svr_eval(u),num2cell(x,2));
function f = svr_eval(x)
f = 0;
n_predict = size(x,1);
for i=1:n_predict
sx(i).feature = x(i,:);
end
n_train = size(xdata,1);
for i=1:n_train
sy(i).feature = xdata(i,:);
end
for i=1:n_train
f = f + svrobj.alpha(i)*kernel_function(sx(1),sy(i));

40. Final
Project
Report
for
Wordoku
Solver
xl
end
f = f + b;
f = f/2;
end
end
O.
%% Threshold_Image sub-routine
%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%
function [Threshold_E bw] = Threshold_Image(gray,g)
%Thresholding and detecting Edges
bw = gray < g ;
[Threshold_E] = edge(bw,'canny',[]);
[Threshold_E] = bw;
end
%%%%%%%%%%%%%%%%%%%%%%
%Threshold_Image sub-routine ends
%%%%%%%%%%%%%%%%%%%%%