This is our oral presentation for ICIP2013 Paper.

1. Video Stabilization with L1-L2
Optimization
Hui Qu, Li Song
Institute of Image Communication and Network
Engineering
Shanghai Jiao Tong University
17 September 2013

2. Outline
 Introduction
 Benchmark work
 Our algorithm
L1-L2 mixed optimization model
Online video stabilization scheme
 Experimental results
 Conclusion

3. Introduction
Video is often shaky

4. Introduction
Video stabilization steps
 Original camera path estimation
shaky

5. Introduction
Video stabilization steps
 Smooth camera path computation
smooth

6. Introduction
Video stabilization steps
 Synthesizing the stabilized video
shaky stabilized

7. Introduction
Video Stabilization methods
 Original camera path estimation
2D camera path——2D linear model
3D camera path——Structure from Motion (SfM)
……
 Smooth camera path computation
Filtering
Optimization
……
 Synthesizing the stabilized video
Cropping——keep the central part
Inpainting——full frame
……

8.  Grundmann et al. 2011
L1 camera path optimization
Integrated into Google ‘s YouTube Editor
Benchmark work
L1 Camera Path Optimization method
Original camera path: 𝐶
Optimal camera path: 𝑃

9.  Objective function:
D means derivative operator
𝜔1, 𝜔2, 𝜔3 are empirical weights
2 3
1 2 31 1 1
( ) ( ) ( ) ( )P D P D P D P
Benchmark work
L1 Camera Path Optimization method
constant path
static camera

10.  Objective function:
2 3
1 2 31 1 1
( ) ( ) ( ) ( )P D P D P D P
Benchmark work
L1 Camera Path Optimization method
path with constant
velocity
panning or
dolly shot

11.  Objective function:
2 3
1 2 31 1 1
( ) ( ) ( ) ( )P D P D P D P
Benchmark work
L1 Camera Path Optimization method
path with constant
acceleration
ease in and out
transition

12. Benchmark work
L1 Camera Path Optimization method
 Inclusion constraint:
stabilized frame

13. Benchmark work
Problem of L1 Path Method
 The method discards information due to cropping
not suitable for videos with important information near the
boundary.

14. Benchmark work
Problem of L1 Path Method
 𝜔1, 𝜔2, 𝜔3 are empirically set
hard to be adaptable to different kinds of videos
sequence 1 sequence 2

15. Our algorithm
L1-L2 mixed optimization model
 Objective function:
2 3
1 1 1 2
( ) ( ) ( ) ( )P D P D P D P P C
L1 part, ensure smoothness
L2 part, ensure proximity to original path
weight, adjust the degree of smoothness and fidelity

16. Our algorithm
L1-L2 mixed optimization model
 Objective function:
2 3
1 1 1 2
( ) ( ) ( ) ( )P D P D P D P P C

17. Our algorithm
L1-L2 mixed optimization model
 Objective function:
2 3
1 1 1 2
( ) ( ) ( ) ( )P D P D P D P P C

18. Our algorithm
L1-L2 mixed optimization model
𝜆 = 0.1 𝜆 = 0.5
𝜆 = 1.0 𝜆 = 2.0

19. Our algorithm
L1-L2 mixed optimization model
𝜆 can be relatively small 𝜆 should be relatively large

20. Our algorithm
Online processing scheme
…… … ……
time
segment 1:N frames
……
segment 2:N frames
K overlapped frames

21. Our algorithm
Online processing scheme
segment 1:
N=150 frames
segment 2:
N=150 frames
K=30 overlapped
frames

22. Our algorithm
Online processing scheme
 Optimal path for the overlapped frames
𝑃𝑡
1
: optimal path of previous segment
𝑃𝑡
2
: optimal path of current segment
𝜐𝑖: weights
 
   1 2
1t i t i tP P P   
0.5 cos 1 0.5i
i
K
 
  
     
  

23. Experimental results
L1 path vs. L1-L2 path
Input shaky video

24. Experimental results
L1 path vs. L1-L2 path
Grundmann et al’s result our result

25. Experimental results
L1 path vs. L1-L2 path
Grundmann et al’s result our result

26. Experimental results
Different values of 𝜆
Input shaky video

27. Experimental results
Different values of 𝜆
𝜆 = 0.1 𝜆 = 0.5
𝜆 = 1.0 𝜆 = 2.0

28. Experimental results
Different values of 𝜆
𝜆 = 0.1
80% crop
𝜆 = 0.5
85% crop
𝜆 = 1.0
90% crop
𝜆 = 2.0
95% crop

29. Experimental results
Speed
 Platform
OS: Windows 7
CPU: Inter Core i5 & 3.1 GHz
 Execution time
About 20 fps ( resolution: 640*360)

30. Conclusion
 Video stabilization method by mixed L1-
L2 optimization
Stabilize & preserve video content
Adjust the degree of stabilization according
to different demands
Able to handle online stabilization and
unlimited length videos

31. Thank you!
Q&A

32. Insert inconsecutive frames of another video
original
stabilized

33. Transition between different scene
original
stabilized

34. Insert some consecutive frames of few features
original
stabilized