By Guillaume Emont. GStreamer 1.0 introduced a new architecture that makes memory management easier. In particular, handling non-system memory and sharing it between elements is made easier. While building a video sink for cairogles, Guillaume wanted to explore the possibility of sharing GPU memory (in his case, GL or GLES textures) between elements of a pipeline. In this talk, Guillaume will explain how he made the various parts fit together, and discuss the challenges that he has met and how he has worked on them.

1. Cairosink, GPU Buffer Sharing
...and other random ramblings
Guillaume Emont - Igalia
guijemont@igalia.com

2. ongoing experiment, I may not have the best answers yet

3. Work sponsored by Samsung Research America
http://www.sisa.samsung.com/

4. I. Technologies, past and present

5. Cairo: 2D graphics
Cairogles: 2D graphics in OpenGL(ES)

6. support for EGL/GLX/WGL

7. Sharing video buffers: before
- Add a specific cap
(e.g. video/x-raw-gl)
- subclass GstBuffer

8. Sharing video buffers: nowadays
- GstContext
- GstVideoGLTextureUploadMeta
- GstAllocator and GstMemory

9. - GstContext, system to share a GL context and more between elements
- GstVideoGLTextureUploadMeta: a way to tell how to upload a buffer to GPU
- GstAllocator and GstMemory: cleaner way to handle non-CPU memory

10. Sharing video buffers: nowadays
"modern" gl(es) sink implementations
known to the author:
- eglglessink (and libgstegl)
- glimagesink

11. II. Integrating cairogles and GStreamer

12. Many possibilities:
- xvimagesink + GstVideoOverlay
- eglglessink
- glimagesink
- make our own sink

13. xvimagesink with overlay: not so fast, I suspect it triggers a download to
CPU to get the image from xvimagesink's Drawable to a cairo-gl surface.
eglglessink: promising, but obviously doesn't support openGL or GLX
glimagesink started work on proper integration, likely will require addition
of a few features there

14. First attempt:
a cairosink that does everything

15. Cairosink: status
- GstAllocator and GstMemory subclasses
- Use either OpenGL or OpenGLES
- Possibility to use PixelBuffer Objects (PBOs)
(inefficiently)

16. allocator and memory not that useful as it's the only element using it. Main
advantage: _could_ allow to start uploading the frame as soon as possible.
PBOs in a nutshell: a way to asynchronously upload a frame to a texture. The
upload is then done through the magic of DMA, without the CPU being involved.
More on why cairosink is currently inefficient at using it later.
FIXME: graph explaining PBO?

17. Cairosink: simplistic benchmark

19. Synchronous / PBO / XV
XV: limited by refresh rate? Looks like its genuine limit (often below 60)
frames 720x400, RGB or YUV

20. Cairosink: difficulties (or rants)
- cairogles can be compiled with
either gl or glesv2 (not both)
- driver bugs and regressions
- cairo only supports RGB
- desktop vs mobile GPUs

21. driver regression: GL+EGL used to work on my machine...

22. Desktop vs mobile GPU
- EGLMakeCurrent() is costly on many mobile GPUs/drivers
- bandwidth differences

23. context acquisition (makeCurrent) slowness is likely to counter the effects of asynchronous
direct upload

24. Cairosink: what's to do
- more flexibility on code path
- compatibility with other elements
(not our own memory/allocator)

25. we want to be able to tailor to various GPUs various "options": is it faster
to do a synchronous or asynchronous upload?

26. Cairosink: other ideas to explore
- libgstgl backend (ongoing experiment)
- wrapper around glimagesink

27. libgstgl issues:
mechanism to MakeCurrent() or save/restore states
share GMainContext? (all in one thread)

28. Thank you
Questions?