Motivation – Image registration is an alignment problem • Registration basics • Rigid registration • Non-rigid registration • Example Applications,

1. MEDICAL IMAGE COMPUTING (CAP 5937)
LECTURE 15: Medical Image Registration I (Introduction)
Dr. Ulas Bagci
HEC 221, Center for Research in Computer
Vision (CRCV), University of Central Florida
(UCF), Orlando, FL 32814.
bagci@ucf.edu or bagci@crcv.ucf.edu
1SPRING 2017

2. Outline
• Motivation
– Image registration is an alignment problem
• Registration basics
• Rigid registration
• Non-rigid registration
• Example Applications
2

3. Image Registration Taxonomy
• Dimensionality
– 2D-2D, 3D-3D, 2D-3D
• Nature of registration basis
– Image based
• Extrinsic, Intrinsic
– Non-image based
• Nature of the transformation
– Rigid, Affine, Projective, Curved
• Interaction
– Interactive, Semi-automatic, Automatic
• Modalities involved
– Mono-modal, Multi-modal, Modality to model
3
• Subject:
Intra-subject
Inter-subject
Atlas
• Domain of transformation
• Local, global
• Optimization procedure
• Gradient Descent, SGD,
…
• Object
• Whole body, organ, …

4. Open Source Implementation
• ITK
• ANTS (advanced normalization tools) (PICSL of Upenn)
• CAVASS (MIPG of Upenn)
• Nifty Reg (UCL)
• Elastix (www.elastix.isi.uu.nl)
• FAIR (Modersitzki 2009), mostly matlab.
• 3D Slicer
• FSL
• …
4

5. Modalities in Medical Imaging
• Mono-modality:
5

6. Modalities in Medical Imaging
• Mono-modality:
ü A series of same modality images (CT/CT, MR/MR, Mammogram
pairs,…).
6

7. Modalities in Medical Imaging
• Mono-modality:
ü A series of same modality images (CT/CT, MR/MR, Mammogram
pairs,…).
ü Images may be acquired weeks or months apart; taken from different
viewpoints.
7

8. Modalities in Medical Imaging
• Mono-modality:
ü A series of same modality images (CT/CT, MR/MR, Mammogram
pairs,…).
ü Images may be acquired weeks or months apart; taken from different
viewpoints.
ü Aligning images in order to detect subtle changes in intensity or shape
8

9. Modalities in Medical Imaging
• Mono-modality:
ü A series of same modality images (CT/CT, MR/MR, Mammogram
pairs,…).
ü Images may be acquired weeks or months apart; taken from different
viewpoints.
ü Aligning images in order to detect subtle changes in intensity or shape
• Multi-modality:
9

10. Modalities in Medical Imaging
• Mono-modality:
ü A series of same modality images (CT/CT, MR/MR, Mammogram
pairs,…).
ü Images may be acquired weeks or months apart; taken from different
viewpoints.
ü Aligning images in order to detect subtle changes in intensity or shape
• Multi-modality:
ü Complementary anatomic and functional information from multiple
modalities can be obtained for the precise diagnosis and treatment.
10

11. Modalities in Medical Imaging
• Mono-modality:
ü A series of same modality images (CT/CT, MR/MR, Mammogram
pairs,…).
ü Images may be acquired weeks or months apart; taken from different
viewpoints.
ü Aligning images in order to detect subtle changes in intensity or shape
• Multi-modality:
ü Complementary anatomic and functional information from multiple
modalities can be obtained for the precise diagnosis and treatment.
ü Examples: PET and SPECT (low resolution, functional information)
need MR or CT (high resolution, anatomical information) to get
structure information.
11
BEFORE
AFTER
PET/CT EXAMPLE

12. In other words,…
• Combining modalities (inter
modality) gives extra
information.
• Repeated imaging over time
same modality, e.g. MRI, (intra
modality) equally important.
• Have to spatially register the
images.
12

13. 13
Before Registration

14. 14
After Registration

15. 15

16. 16

17. 17

18. 18

19. 19

20. 20

21. 21

22. 22

23. Summary of Mostly Used Applications
• Diagnosis
– Combining information from multiple imaging modalities
• Studying disease progression
– Monitoring changes in size, shape, position or image intensity over
time
• Image guided surgery or radiotherapy
– Relating pre-operative images and surgical plans to the physical reality
of the patient
• Patient comparison or atlas construction
– Relating one individual’s anatomy to a standardized atlas
23

24. Then, What is Image Registration (formally)?
24

25. Image Registration is a
• Spatial transform that maps points from one image to
corresponding points in another image
25
matching two images so that corresponding coordinate
points in the two images correspond to the same physical
region of the scene being imaged also referred to as
image fusion, superimposition, matching or merge
MR SPECT registered

26. Image Registration is a
• Spatial transform that maps points from one image to
corresponding points in another image
– Rigid
• Rotations and translations
– Affine
• Also, skew and scaling
– Deformable
• Free-form mapping
26

27. Registration Framework
27
Deformation
Model
Optimizat
ion Metho
d
Matching
Criteria (
Objective
Function)

28. Recap: Image Registration Taxonomy
• Dimensionality
– 2D-2D, 3D-3D, 2D-3D
• Nature of registration basis
– Image based
• Extrinsic, Intrinsic
– Non-image based
• Nature of the transformation
– Rigid, Affine, Projective, Curved
• Interaction
– Interactive, Semi-automatic, Automatic
• Modalities involved
– Mono-modal, Multi-modal, Modality to model
28
• Subject:
Intra-subject
Inter-subject
Atlas
• Domain of transformation
• Local, global
• Optimization procedure
• Gradient Descent, SGD,
…
• Object
• Whole body, organ, …

29. Deformation Models
Method used to find the transformation
• Rigid & affine
– Landmark based
– Edge based
– Voxel intensity based
– Information theory based
• Non-rigid
– Registration using basis functions
– Registration using splines
– Physics based
• Elastic, Fluid, Optical flow, etc.
29

30. Deformation Model (Transformation)
• Rigid
– Rotation, translation
• Affine
– Rigid + scaling
• Deformable
– Affine + vector field
• ….
30

31. 31
Deformation Model (linear vs. non-linear)

32. Linear Registration -> Separable
32
RIGID TRANSFORMATION
rotation

33. Linear Registration -> Separable
33
RIGID TRANSFORMATION
rotation

34. Rigid Registration - Rotation
34

35. Example Rigid Transformation Formulation
35

36. Example Rigid Transformation Formulation
36
Old locationnew location

37. Linear Registration -> Separable
37
AFFINE TRANSFORMATION = Rigid + Scaling (+ skew)
9 parameters, Affine = 6 parameters (rotation + translation) + 3 parameters (scaling)
12 parameter, Affine = …+ 3 parameters (skew)

38. Shear in 3D
38

39. Affine Transformation
39
p’ = M p + t

40. Homogenous Coordinates for Transformations
40

41. Homogenous Coordinates for Transformations
41

42. Translation in Homogenous Coordinate System
42

43. Registration is an alignment problem
p = (825,856)
q = (912,632)
q = T(p;a)
Pixel location in first image Homologous pixel location in
second image
Pixel location mapping function

44. Registration is an alignment problem
p = (825,856)
q = (912,632)
q = T(p;a)
Pixel scaling and
translation

45. Similarity Criteria
45

46. Intensity Based
• Method
– Calculating the registration
transformation by
optimizing some measure
calculated directly from
the voxel values in the
images
• Algorithms used
– Registration by minimizing
intensity difference
– Correlation techniques
– Ratio image uniformity
– Partitioned Intensity
Uniformity
46

47. Intensity Based
• Intensity-based methods compare intensity
patterns in images via some similarity metrics
– Sum of Squared Differences
– Normalized Cross-Correlation
– Mutual Information
47

48. Feature Based
• Feature-based methods find correspondence between image
features such as points, lines, and contours.
• Distance between corresponding points
• Similarity metric between feature values
– e.g. curvature-based registration
48

49. Information Theory Based
• Image registration is considered as
to maximize the amount of shared
information in two images
– reducing the amount of information in
the combined image
• Algorithms used
– Joint entropy
• Joint entropy measures the amount
of information in the two images
combined
– Mutual information
• A measure of how well one image
explains the other, and is
maximized at the optimal alignment
– Normalized Mutual Information
49

50. 50

51. 51

52. Simple Code for Joint Entropy Computation
rows=size(x,1);
cols=size(y,2);
N=256;
h=zeros(N,N);
for i=1:rows;
for j=1:cols;
h(x(i,j)+1,y(i,j)+1)= h(x(i,j)+1,y(i,j)+1)+1;
end
end
imshow(h)
end
52

53. 53Because the images are identical,
all gray value correspondences lie on
the diagonal.

54. 54

55. • I(A,B) = H(A) + H(B) – H(A,B)
– Maximizing mutual information is related to minimizing joint entropy
55
Less sensitive to changes in overlap!

56. Registration Algorithm
• Fixed and Moving Images (target and source, …)
• Preprocessing
• Define Similarity Measure (NMI, CC, MSE, …)
• Define Spatial Transformation (Rigid, Affine, Deformable)
• Implementation
1. Initialize
2. Transform (and Interpolate) moving image
3. Measure similarity
4. Optimize (decide parameters of the transform)
5. If converged
STOP
6. Else
7. Go to Transform Step 2. Repeat
56

57. Summary
• Introduction to the medical image registration
• Transformation types
– Rigid, affine, non-rigid
• Mono-modal, multi-modal image registration
• Similarity metric
• Mutual Information
• Next Lecture(s): further details on the topic.
57

58. Slide Credits and References
• Credits to: Jayaram K. Udupa of Univ. of Penn., MIPG
• Sir M. Brady’s Lecture Notes (Oxford University)
• Darko Zikic’s MICCAI 2010 Tutorial
• Bagci’s CV Course 2015 Fall.
• K.D. Toennies, Guide to Medical Image Analysis,
• Handbook of Medical Imaging, Vol. 2. SPIE Press.
• Handbook of Biomedical Imaging, Paragios, Duncan, Ayache.
• Seutens,P., Medical Imaging, Cambridge Press.
• Aiming Liu, Tutorial Presentation.
• Jen Mercer, Tutorial Presentation.
58