Image and video copy detection by sparse representation

1. Reference Image
Query Image(Crop)
Query Image(Conv.)
Query Image(Noise) Query Image(Rot.+Crop)
Query Image(Rot.)

2. Sparse Representation in Image
and Video Copy Detection
Huan-Cheng Hsu

3. Introduction
• With the rapid growth of network and
multimedia technology, a massive amount of
non-original, illegal or duplicated images are
shared via internet.
• In order to protect the copyright, identifying
duplicated images has become an emerging
research topic.
• Original images and videos can be
duplicated/manipulated in various ways.

4. Introduction
• To detect duplicated images, digital
watermarking techniques are developed.
• These techniques require inserting additional
watermarks into images.
• They cannot detect the duplications if
watermarks are not embedded.
I. Cox, J. Killian, F. T. Leighton, and T. Shamoon, “Secure spread spectrum watermarking for multimedia,” IEEE Trans. on Image
Processing, vol. 6, no. 12, pp. 1673–1687, Dec. 1997.
B. Chen and G. W. Wornell, “Quantization index modulation: A class of provably good methods for digital watermarking and
information embedding,” IEEE Trans. Inf. Theory, vol. 47, no. 5, pp. 1423–1443, May 2001.

5. Introduction
• The main idea of content based image copy
detection (CBCD) is to compare the extracted
features of a suspect image with those in a
database.
• An original image can be manipulated in a
various way before the manipulated/copied
image is used.
• This calls for robust feature extraction against
various distortions/attacks.

6. Introduction
Reference Image
Query Image(Crop)
Query Image(Conv.)
Query Image(Noise) Query Image(Rot.+Crop)
Query Image(Rot.)

7. Introduction
• Chang et al. propose a replicated image
detector (RIME) by using wavelet transform
and defining a new color space based on RGB
values.
• Kim compares discrete cosine transform (DCT)
coefficients between duplicated and query
images to detect copies.
E. Chang, J. Wang, C. Li, and G. Wiederhold, “RIME: A replicated image detector for the world-wide-
web,” in SPIE Multimedia Storage and Archiving Systems III, vol. 3527, pp. 58–67, 1998.
C. Kim, “Content-based image copy detection,” Signal Processing: Image Communication, vol.18, no.3,
pp. 169–184, 2003.

8. Introduction
• Wu et al. propose an elliptical division strategy
to solve the in-plane rotation problem.
• Hsiao et al. propose extended feature set (EFS),
which obtains additional features by applying
possible distortions on the original images.
M.-N. Wu, C.-C. Lin, and C.-C. Chang, “Image Copy Detection with Rotating Tolerance,” in CIS, Part I,
LNAI 3801, pp. 464–469, 2005.
J.-H. Hsiao, C.-S. Chen, L.-F. Chien, and M.-S. Chen, “A new approach to image copy detection based on
extended feature sets,” IEEE Trans. on Image Processing, vol. 16, no. 8, pp. 2069–2079, 2007.

9. Sparse Representation
• Recently, sparse representation (SR) in the
context of compressed sensing has received
considerable attention.
• SR is able to describe images based on their
natural sparsity and redundancy to obtain the
most compact representations.
Y D α
1α
kα
ε
J. Mairal, M. Elad, and G. Sapiro, “Sparse representation for color
image restoration,” IEEE Trans. Image Processing, vol. 17, no. 1, pp.
53–69, Jan. 2008.
J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma, “Robust face
recognition via sparse representation,” IEEE Trans. Pattern Anal.
Mach. Intell., vol. 31, no. 2, pp. 210–227, Feb. 2009.

10. SR in Image Copy Detection
Y D α
1α
kα
ε
L. W. Kang, C. Y. Hsu, H. W. Chen, and C. S. Lu, “Feature-based Sparse Representation for Image
Similarity Assessment,” IEEE Trans.on Multimedia, 2011.

11. Our SR in Image Copy Detection
J. Mairal, F. Bach, J. Ponce, and G. Sapiro,
“Online learning for matrix factorization
and sparse coding,” J. Mach. Learn. Res.,
vol. 11, pp. 19–60, Mar. 2010
C.-R. Huang , C.-S. Chen, and P.-C. Chung,
“Contrast Context Histogram - An Efficient
Discriminating Local Descriptor for Object
Recognition and Image Matching,” Pattern
Recognition, vol. 41, no. 10, pp. 3071–
3077, 2008
NKD KM
=ℜ∈ ×
,

12. Proposition : Sparsity

13. Image Copy Detection Using Sparsity Cue

14. Experimental Setting
Airplane Baboon Fruits Lena Peppers
F. Petitcolas, “Watermarking schemes
evaluation,” Signal Processing Magazine, IEEE,
vol. 17, no. 5, pp. 58–64, Sep. 2000.
Types # Description
Convolution Filtering 2 Blurring & sharpening
JPEG 14 Quality 5% ~ 95%
Median Filtering 4 Window size 2 ~ 8
Noise 6 Level 1% ~ 88%
Self-Similarities 3 Color space changing
PSNR 10 Range 10% ~ 100%
Cropping 13 Sizes 20% ~ 95%
Rotation 18 Degrees 1~359
Affine 8 Affine transform
Remove Lines 10 Frequency 10% ~ 100%
Rotation+Rescaling 10 30 ~ 30 degree
Rotation+Cropping 10 30 ~ 30 degree

15. Performance Comparison

16. Performance Comparison

17. 2010 - MIRFLICKR-1M Dataset
• To evaluate our method, we further perform
our method on a larger dataset that contains
one million images.
Query
Image
Sets
Airplane Baboon Fruits Lena Peppers
Mean
Sparsity
3.27 3.23 3.32 3.34 3.32
Mean
Time
0.16 0.17 0.15 0.15 0.15
False Rate
(%)
0.19 0.13 0.04 0.02 0.05
M. J. Huiskes, and M. S. Lew, “The MIR Flickr retrieval evaluation,” in Proc. of ACM Int. Conf. on Multimedia Information Retrieval, pp.
39–43, 2008.
M. J. Huiskes, B. Thomee, and M. S. Lew, “New trends and ideas in visual concept detection,” in Proc. of ACM Int. Conf. on Multimedia
Information Retrieval, pp. 527–536, New York, 2010.

18. Video Copy Detection
• Video copy attributes contain not only various
spatial attacks but also temporal factors.
• Moreover, videos contain much more data
information, the processing time becomes a
very important issue.

19. Video Copy Detection
• Kim et al. propose a sequence matching
method that which uses ordinal measure
between spatial information and temporal
signatures to evaluate similarity.
• Chen et al. propose a method based on
temporal ordinal measurements. This method
divides each frame into many partitions, and
sorts the corresponding partitions along the
time series.
C. Kim and B. Vasudev, “Spatiotemporal sequence matching for efficient video copy detection,” IEEE Trans. Circuits Syst. Video Technol.,
vol. 15, no. 1, pp. 127–132, Jan. 2005.
L. Chen and F. W. M. Stentiford, “Video sequence matching based on temporal ordinal measurement,” Pattern Recognition Letters, vol.
29, no. 13, pp. 1824–1831, Oct. 2008

20. Video Copy Detection
• Chiu et al. use dynamic time warping to select
key frames.
• Esmaeili et al. propose a fast spatio-temporal
fingerprint algorithm called efficiency of
fingerprint matching - discrete cosine
transform (TIRI-DCT).
C.-Y. Chiu, C.-H. Li, H.-A. Wang, C.-S. Chen, and L.-F. Chien, “A time warping based approach for video copy detection,” in Proc. ICPR,
pp. 228–231, 2006.
M. M. Esmaeili, M. Fatourechi, and R. K. Ward, “A robust and fast video copy detection system using content-based fingerprinting,”
IEEE Trans. Information Forensics and Security, vol. 6, no. 1, pp. 213 –226, March 2011.

21. SR in Video Copy Detection
• We describe how to represent a video clip by a set of spatially
and temporally coherent trajectories in this section, it consists
of the following three steps:
– Video shot detection.
– Keypoint localization, matching, and tracking.
– Trajectory description.
C.-R. Huang, Z.-X. Yang, and Y.-Y. Lin, “VIDEO SALIENCY MAP DETECTION USING TRAJECTORY ENTROPY,” 2012.

22. Video Copy Detection Experimental Setting
• We chose 3 reference videos from a 6.1-
hour video that composed by 14
sequence videos.
• Reference videos are transformed into
MPEG-1, 320×240 pixels, and 30 frames
per second (fps).
• Each video is 30 seconds and is derived 12
transformation types.
C. Y. Chiu, H. M. Wang, and C. S. Chen, “Fast min-hashing indexing and robust spatio-temporal matching for detecting video copies,”
ACM Trans. Multimedia Comput., Commun. Applicat., vol. 6, no. 2, Mar. 1, 2010.

23. Video Copy Detection Experimental Setting
Category Type Description
Whole region-preserved spatial
transformation
Brightness Enhance the brightness by 20%.
Compression Set the compression quality 50%
Noise Add 10% random noise.
Equalization Equalize the color histogram.
Resolution change Change the frame resolution to
120×90 pixels.
Partial region-discarded spatial
transformation
Cropping Crop the top and bottom frame
regions by 10% each.
Zooming in Zoom in to the frame by 10%.
Frame number-changed temporal
transformation
Slow motion Halve the video speed.
Fast forward Double the video speed.
Frame rate change Change the frame rate to 15 fps.
Frame order-changed temporal
transformation
Swap Swap the first-half subsequence and
the second-half one.
Insertion/deletion Delete middle 50% of frames and
insert unrelated frames.

24. Performance

25. Conclusions
• We investigate a sparse representation-based
image copy detection method, our method is
computationally efficient while attaining better
or comparable detection performance.
• To further improve the detection efficiency,
other faster descriptors can be used, we also
plan to study web-scale retrieval of digital
images based on the discovered sparsity cue.

26. Thank you for your attention!