Paper and videos: http://www.videofields.com Another interesting project: http://socialstreetview.com Video Fields system fuses multiple videos, camera-world matrices from a calibration interface, static 3D models, as well as satellite imagery into a novel dynamic virtual environment. Video Fields integrates automatic segmentation of moving entities during the rendering pass and achieves view-dependent rendering in two ways: early pruning and deferred pruning. Video Fields takes advantage of the WebGL and WebVR technology to achieve cross-platform compatibility across smart phones, tablets, desktops, high-resolution tiled curved displays, as well as virtual reality head-mounted displays. See the supplementary video at http://video-fields.com.

2. Video Fields: Fusing Multiple Surveillance Videos into a
Dynamic Virtual Environment
Ruofei Du, Sujal Bista, Amitabh Varshney
The Augmentarium| UMIACS | University of Maryland, College Park
{ruofei, sujal, varshney} @ cs.umd.edu
www.VideoFields.com

3. image courtesy: university of maryland, college park
Introduction
Surveillance Videos - Monitoring

4. image courtesy: www.icsc.org
Introduction
Surveillance Videos – Shopping Centers

5. image courtesy: wikipedia
Introduction
Surveillance Videos - Airports

6. image courtesy: wikipedia
Introduction
Surveillance Videos – Train stations

7. image courtesy: university of maryland, college park
Introduction
Surveillance Videos - Campuses

8. image courtesy: university of maryland, college park
Introduction
Surveillance Videos - Conventional

9. image courtesy: theimaginativeconservative.org
Introduction
Surveillance Videos – Cognitive Burden

10. image courtesy: university of maryland, college park
Introduction
Surveillance Videos – Fusing & Interpreting

11. Related Work
Fusing Multiple Static Photographs

12. Related Work
Fusing Multiple Static Photographs

13. Related Work
Fusing Multiple Static Photographs

14. Related Work
Fusing Multiple Static Photographs

15. Related Work
Fusing Multiple Static Photographs

17. Related Work
Fusing Multiple Dynamic Videos

18. Related Work
Fusing Multiple Dynamic Videos
RGB

19. Related Work
Fusing Multiple Dynamic Videos
RGB
RGBD

20. Related Work
Fusing Multiple Dynamic Videos

21. Related Work
Fusing Multiple Dynamic Videos

22. Related Work
Fusing Multiple Dynamic Videos

23. Related Work
Fusing Multiple Dynamic Videos

24. Related Work
Fusing Multiple Dynamic Videos

25. Related Work
Fusing Multiple Dynamic Videos

26. Related Work
Fusing Multiple Dynamic Videos

27. Related Work
Fusing Multiple Dynamic Videos

28. Related Work
Fusing Multiple Dynamic Videos
SIGGRAPH 2016
Wednesday, 3:30-4:00 PM

29. Related Work
Fusing Multiple Dynamic Videos

30. Related Work
Fusing Multiple Dynamic Videos

31. Related Work
Fusing Multiple Dynamic Videos

32. Our Approach?

33. Video Fields

34. Video Fields

35. Introduction
Video Field

37. Introduction
Video Field

39. Conception, architecting & implementation
Video Fields
A mixed reality system that fuses multiple
surveillance videos into an immersive virtual
environment,

40. Integrating automatic segmentation of
moving entities
Video Fields Rendering
Real-time fragment shader processing

41. Two algorithms to fuse multiple videos
Early & deferred pruning
These methods use voxels and meshes respectively
to render moving entities in the video fields

42. Achieving cross-platform compatibility by
WebGL + Three.js
smartphones, tablets, desktop, high-resolution
large-area wide field of view tiled display walls, as
well as head-mounted displays.

43. System Overview

44. Architecture
Video Fields Flowchart

45. Architecture
Video Fields Flowchart

46. Architecture
Video Fields Flowchart

48. Architecture
Video Fields Flowchart

49. Background Modeling
Motivation
• Provide a background texture for each camera
• Identify moving entities in the rendering stage
• Reduce the network bandwidth requirements

50. Background Modeling
Gaussian Mixture Models (GMM)

51. Background Modeling
Advantages [Stauffer and Grimson]
More adaptive with:
• different lighting conditions,
• repetitive motions of scene elements,
• moving entities in slow motion

52. Architecture
Video Fields Flowchart

53. Segmentation
Moving Entities

54. Background Modeling
Gaussian Mixture Models (GMM)

56. Architecture
Video Fields Flowchart

57. Visibility Test
Plus Opacity Modulation

59. Architecture
Video Fields Flowchart

60. Video Fields Mapping
Overview

61. Video Fields Mapping
Challenges
1. Vertex in the 3D models -> Pixel in the texture space
2. Pixel in the texture space -> Vertex on the ground
• The second is useful for projecting a 2D segmentation
of a moving entity to the 3D world

62. Video Fields Mapping
Projection Mapping

63. Video Fields Mapping
Perspective correction

64. Video Fields Mapping
Depth Map / Hashing Function

65. Early Pruning for Rendering
Moving Entities
Voxels

66. Deferred Pruning for
Rendering Moving Entities
Billboards

67. Visual Comparison
Early Pruning vs. Deferred Pruning

68. View-dependent Rendering

69. View-dependent Rendering

70. View-dependent Rendering

71. View-dependent Rendering

72. Experimental Results
Early Pruning vs. Deferred Pruning

74. Experimental Results
Early Pruning vs. Deferred Pruning

75. Experimental Results
Early Pruning vs. Deferred Pruning

76. Visual Comparison
Early Pruning vs. Deferred Pruning

78. Future Work
Scale Up - Hundreds of cameras

79. Future Work
Bandwidth Problem

80. Future Work
Holoportation with RGB cameras

81. Acknowledgement
Augmentarium Lab | GVIL | UMIACS

82. Acknowledgement
NSF | Nvidia | MPower | UMIACS

83. Video Fields
www.Video-Fields.com
Thank you! Questions or comments?
Ruofei Du and Amitabh Varshney
Augmentarium Lab | GVIL | UMIACS
Web3D 2016