1. Super Resolution для
сжатого видео
Моисейцев Алексей
Video Group
CS MSU Graphics & Media Lab

2. Only for
Maxus 
Содержание
 Введение
 Example-based SR
 SRME
 HMRF SR
2
CS MSU Graphics & Media Lab (Video Group)

3. Only for
Maxus 
Введение
3
CS MSU Graphics & Media Lab (Video Group)

4. Only for
Maxus 
Введение
Lossless 2Mbps 1Mbps 256Kbps
Bi-cubic
source
Super-res
Super-Resolving Compressed Video with Large Artifacts, Wen-Yi Zhao, ICPR 4
CS MSU Graphics & Media Lab (Video Group) 2004

5. Only for
Maxus 
Введение
Алгоритмы SR:
 Iterative Backprojection (IBP)
 Projection Onto Convex Sets (POCS)
 Probabilistic Methods
 Maximum a posteriori (MAP)
 Model-based approach (MBSR)
 Example-based
6
CS MSU Graphics & Media Lab (Video Group)

6. Only for
Maxus 
Введение
IBP POCS
C2 C1
C
f=Pg P1g
P2g
C3
P3g
g
7
CS MSU Graphics & Media Lab (Video Group)

7. Only for
Maxus 
Введение
8
CS MSU Graphics & Media Lab (Video Group) Super resolution: an overview, C Papathanassiou and M Petrou, Geoscience
and Remote Sensing Symposium, 2005

8. Only for
Maxus 
Введение
Типичная скорость работы: 0.15-0.4 fps
9
CS MSU Graphics & Media Lab (Video Group)

9. Only for
Maxus 
Содержание
 Введение
 Example-based SR
 SRME
 HMRF SR
10
CS MSU Graphics & Media Lab (Video Group)

10. Only for
Maxus 
Example-based SR
 В SR только по LR-кадрам
существует теоретический
предел качества
 Иногда есть доступ и к
отдельным кадрам в
высоком разрешении
11
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006

11. Only for
Maxus 
Example-based SR
Registration
Для компенсации движения
используется алгоритм Lucas-Kanade:
12
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006

12. Only for
Maxus 
Example-based SR
Registration
В 99% случаев ошибка не превышает 0.15 LR-пикселя и 0.45 HR-пикселя
13
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006

13. Only for
Maxus 
Example-based SR
Построение пар HR-LR
 Используются LR-блоки
8x8
 Рассматриваются
повороты блоков
 Ключ —
10 AC-коэффициентов LR-
блока и 68 граничных
пикселей HR-блока
14
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006

14. Only for
Maxus 
Example-based SR
Восстановление
15
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006

15. Only for
Maxus 
Example-based SR
Восстановление
16
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006

16. Only for
Maxus 
Example-based SR
Результат
source, frame 0 compressed frame 20 SR, frame 20
 MJPEG, Q=50
 Обучение на первом кадре
 Восстановление двадцатого кадра
17
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006

17. Only for
Maxus 
Example-based SR
Результат
compressed frame 20 SR, frame 20 Spatial example-based SR,
frame 20
18
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006

18. Only for
Maxus 
Key-frame based SR
Идея: использовать HR ключевые кадры
19
CS MSU Graphics & Media Lab (Video Group) Super Resolution ofMukherjee, ISCASFrames, Fernanda Brandi, Ricardo de
Queiroz, Debargha
Video Using Key
2008

19. Only for
Maxus 
Key-frame based SR
Восстановление
20
CS MSU Graphics & Media Lab (Video Group) Super Resolution ofMukherjee, ISCASFrames, Fernanda Brandi, Ricardo de
Queiroz, Debargha
Video Using Key
2008

20. Only for
Maxus 
Key-frame based SR
Результаты
21
CS MSU Graphics & Media Lab (Video Group) Super Resolution ofMukherjee, ISCASFrames, Fernanda Brandi, Ricardo de
Queiroz, Debargha
Video Using Key
2008

21. Only for
Maxus 
Key-frame based SR
Результаты
22
CS MSU Graphics & Media Lab (Video Group) Super Resolution ofMukherjee, ISCASFrames, Fernanda Brandi, Ricardo de
Queiroz, Debargha
Video Using Key
2008

22. Only for
Maxus 
Key-frame based SR
Результаты
23
CS MSU Graphics & Media Lab (Video Group) Super Resolution ofMukherjee, ISCASFrames, Fernanda Brandi, Ricardo de
Queiroz, Debargha
Video Using Key
2008

23. Only for
Maxus 
Example-based SR
 Слабая обоснованность метода
 Требуется наличие специфичного
видеопотока
 Зависимость от обучающей выборки
24
CS MSU Graphics & Media Lab (Video Group)

24. Only for
Maxus 
Содержание
 Введение
 Example-based SR
 SRME
 HMRF SR
25
CS MSU Graphics & Media Lab (Video Group)

25. Only for
Maxus 
SRME
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 26
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

26. Only for
Maxus 
SRME
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 27
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

27. Only for
Maxus 
SRME
— апостериорная
вероятность
— интересующее
решение
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 28
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

28. Only for
Maxus 
SRME
Задание вероятностей (1)
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 29
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

29. Only for
Maxus 
SRME
Задание вероятностей (2)
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 30
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

30. Only for
Maxus 
SRME
Задание вероятностей (3)
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 31
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

31. Only for
Maxus 
SRME
Решение (1)
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 32
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

32. Only for
Maxus 
SRME
Решение (2)
Две итерации:
 Компенсация движения
 Построение улучшенного кадра
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 33
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

33. Only for
Maxus 
SRME
source
Y-PSNRSimultaneous motion estimation and resolution enhancement of
= 20.44dB
compressed low resolution video. 34
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

34. Only for
Maxus 
SRME
bilinear
Y-PSNRSimultaneous motion estimation and resolution enhancement of
= 21.98dB
compressed low resolution video. 35
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

35. Only for
Maxus 
SRME
proposed SR
Y-PSNRSimultaneous motion estimation and resolution enhancement of
= 25.64dB
compressed low resolution video. 36
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

36. Only for
Maxus 
Содержание
 Введение
 Example-based SR
 SRME
 HMRF SR
37
CS MSU Graphics & Media Lab (Video Group)

37. Only for
Maxus 
HMRF SR
Standard decompression Smoothed image Sharpened image
Restoration of Compressed Video using Temporal Information, 38
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001

38. Only for
Maxus 
HMRF SR
Restoration of Compressed Video using Temporal Information, 39
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001

39. Only for
Maxus 
HMRF SR
Шум квантования (1)
Restoration of Compressed Video using Temporal Information, 40
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001

40. Only for
Maxus 
HMRF SR
Шум квантования (2)
Restoration of Compressed Video using Temporal Information, 41
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001

41. Only for
Maxus 
HMRF SR
Шум компенсации (1)
Restoration of Compressed Video using Temporal Information, 42
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001

42. Only for
Maxus 
HMRF SR
Шум компенсации (2)
— общее выражение
— независимость шума
Restoration of Compressed Video using Temporal Information, 43
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001

43. Only for
Maxus 
HMRF SR
Условие гладкости
Restoration of Compressed Video using Temporal Information, 44
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001

44. Only for
Maxus 
HMRF SR
— гладкость
— квантование
— компенсация
Restoration of Compressed Video using Temporal Information, 45
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001

45. Only for
Maxus 
HMRF SR
Решение
Градиент:
Restoration of Compressed Video using Temporal Information, 46
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001

46. Only for
Maxus 
HMRF SR
Решение
Упрощение:
Градиентный спуск:
Restoration of Compressed Video using Temporal Information, 47
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001

47. Only for
Maxus 
HMRF SR
Результаты
Restoration of Compressed Video using Temporal Information, 48
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001

48. Only for
Maxus 
HMRF SR
Результаты
Restoration of Compressed Video using Temporal Information, 49
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001

49. Only for
Maxus 
HMRF SR
Результаты
Restoration of Compressed Video using Temporal Information, 50
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001

50. Only for
Maxus 
Заключение
 Рассмотрены Example-based и MAP методы
 Многие методы используют информацию из
потока или требуют дополнительные
данные для работы
 Прямые реализации алгоритмов сложны, но
тем не менее многие из них возможно
распараллелить (например, с
использованием CUDA)
51
CS MSU Graphics & Media Lab (Video Group)

51. Only for
Maxus 
Литература
 Simultaneous motion estimation and resolution enhancement of compressed low
resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
 Restoration of Compressed Video using Temporal Information, Mark A.
Robertson and Robert L. Stevenson, SPIE 2001
 DCT Quantization Noise in Compressed Images, Mark A. Robertson and Robert L.
Stevenson, 2004
 Super-Resolving Compressed Video with Large Artifacts, Wen-Yi Zhao, ICPR 2004
 Z. Lin and H-Y. Shum. Fundamental limits of reconstruction-based
superresolution algorithms under local translation. PAMI, 2004
 Resolution enhancement of low quality videos using a high-resolution frame,
Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
 T.Q. Pham, M. Bezuijen, L.J. van Vliet, K. Schutte, and C.L. Luengo Hendriks.
Performance of optimal registration estimators, SPIE 2005
 Super resolution: an overview, C Papathanassiou and M Petrou, Geoscience and
Remote Sensing Symposium, 2005
 Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de
Queiroz, Debargha Mukherjee, ISCAS 2008 52
CS MSU Graphics & Media Lab (Video Group)

52. Only for
Maxus 
Вопросы
?
53
CS MSU Graphics & Media Lab (Video Group)