This document discusses image processing using FPGAs. It begins with an overview of FPGAs and their components. It then discusses using high-level synthesis to convert C++ code to hardware designs for FPGAs. An example of implementing Sobel edge detection on an FPGA is provided. The implementation was optimized from 40 cycles per pixel to 1 cycle per pixel through pipelining, parallelism, and using block RAM for intermediate storage. Challenges discussed include limited debugging tools and steep learning curves for FPGA development.

1. Image processing on FPGA
Eugene Khvedchenya
https://ua.linkedin.com/in/cvtalks

2. What is FPGA and who needs it ?

3. General implementation
OpenCL
Cache tuning
Multithreading
SIMD (SSE, NEON)
FPGA
Optimization pyramid

4. What’s inside?
LUT
Flip-Flop
ALU
BRAM
IO pads
FPGA

5. Development efforts

6. CPU vs FPGA

7. CPU vs FPGA

8. CPU vs FPGA

9. Development efforts

10. High Level Synthesis
Converts C++ code to hardware design
HLS compiler optimizes your code for FPGA
Automatically optimize RTL and timing
Provides #pragma’s for fine tuning
C++ API for arbitrary precision math
C++ API for stream data processing
Supports C++ 11

11. Things to remember
No branching penalty

12. Things to remember
No dynamic memory allocation

13. Things to remember
Instantaneous BRAM access
Register-level bandwidth 0.5M-bits / second
BRAM bandwidth 23T-bits / second
Numbers above for Xilinx Kintex®-7 410T device

14. Things to remember
Single producer - single consumer

15. Things to remember
Pipelining

16. Things to remember
● No branching penalty
● No cache penalty
● No dynamic memory allocation
● Instantaneous BRAM access
● Single producer - single consumer
● Pipelining
● Task-centric approach

17. HLS Development cycle
1. Get baseline version
2. Write simulation test
3. Run HLS synthesis
4. Simulate
5. Validate
6. Measure
7. Optimize
8. Goto 3

18. Sobel Edge Detection
Goal: Process image 1920x1080 @ 60HZ

19. Sobel Edge Detection
Baseline implementation
Iterate over image
● Convolve 3x3 window with Gx and Gy kernels
● Compute their absolute sum
● Write to corresponding output pixel
The FPGA frequency is this example is 150 Mhz
To meet 1920x1080@60Hz goal we must process data at rate 1 cycle/pixel or faster

20. Sobel Edge Detection
Baseline implementation

21. Sobel Edge Detection
Baseline implementation
40 cycles/pixel on FPGA
Timing violation

22. Sobel Edge Detection
Tuning FPGA implementation
Iterate over image
● Convolve 3x3 window with Gx and Gy kernels
Pipeline: Compute one field in the 3x3 filter window per clock cycle.
● Compute Gx and Gy absolute sum
● Write to corresponding output pixel

23. Sobel Edge Detection
Tuning FPGA implementation

24. Sobel Edge Detection
Tuning FPGA implementation
10 cycles/pixel on FPGA
Timing violation

25. Sobel Edge Detection
Tuning FPGA implementation
Iterate over image
● Pipeline: Apply pipeline to the inner loop (columns)
● Convolve 3x3 window with Gx and Gy kernels
○ Loop gets totally unrolled and computed at 1 cycle
● Compute Gx and Gy absolute sum
○ Also computed in parallel
● Write to corresponding output pixel

26. Sobel Edge Detection
Tuning FPGA implementation

27. Sobel Edge Detection
Tuning FPGA implementation
1 cycle/pixel on FPGA
Memory-access violation

28. Sobel Edge Detection
Tuning FPGA implementation
Issues
● Nine concurrent memory accesses
● More hardware blocks required
● HLS module can only connect a single port capable of one transaction/clock

29. Sobel Edge Detection
Tuning FPGA implementation
● Use BRAM to store intermediate line buffer
● Read data from external memory to line buffer
● Fill memory window (Flip-flop elements)
● Convolve 3x3 window with Gx and Gy kernels
○ Loop gets totally unrolled and computed at 1 cycle
● Compute their absolute sum
○ Also computed in parallel
● Write to corresponding output pixel

30. Sobel Edge Detection
Tuning FPGA implementation
1 cycle/pixel on FPGA
Achievement unlocked

31. The dark side
Of the FPGA development
● The tools aren’t great
● It works in simulator!
● Learning curve
● Debugging timing violations

32. Quick start
● FPGA Development board: Altera, Xilinx
● IDE & Samples: Vivado
● OpenCV support
● HLS for OpenCL

33. Image processing on FPGA
Eugene Khvedchenya
Questions?
https://ua.linkedin.com/in/cvtalks
ekhvedchenya@gmail.com
@cvtalks