Macromolecular crystallography is an experimental technique allowing to explore 3D atomic structure of proteins, used by academics for research in biology and by pharmaceutical companies in rational drug design. While up to now development of the technique was limited by scientific instruments performance, recently computing performance becomes a key limitation. In my presentation I will present a computing challenge to handle 18 GB/s data stream coming from the new X-ray detector. I will show PSI experiences in applying conventional hardware for the task and why this attempt failed. I will then present how IC 922 server with OpenCAPI enabled FPGA boards allowed to build a sustainable and scalable solution for high speed data acquisition. Finally, I will give a perspective, how the advancement in hardware development will enable better science by users of the Swiss Light Source.
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
OpenCAPI-based Image Analysis Pipeline for 18 GB/s kilohertz-framerate X-ray Camera at the Swiss Light Source synchrotron
1. WIR SCHAFFEN WISSEN – HEUTE FÜR MORGEN
OpenCAPI-based image analysis
pipeline for 18 GB/s kHz-framerate X-
ray camera at the SLS synchrotron
Filip Leonarski :: Beamline Data Scientist :: Macromolecular Crystallography
Page 1
2. • Introduction: Macromolecular crystallography at synchrotrons and X-ray
detectors
• Technology: POWER + OpenCAPI
• Solution: Jungfraujoch
Plan
Page 2
4. X-ray macromolecular crystallography (MX)
Page 4
1901 Nobel Prize
W. Röentgen
Discovery of X-rays
(Photo 51 by R.
Gosling and R.
Franklin)
1962 Nobel Prize
F. Crick, J. Watson and
M. Wilkins
Structure of DNA
double helix solved
with X-rays
5. X-ray macromolecular crystallography (MX)
Page 5
1901 Nobel Prize
W. Röentgen
Discovery of X-rays
(Photo 51 by R.
Gosling and R.
Franklin)
1962 Nobel Prize
F. Crick, J. Watson and
M. Wilkins
Structure of DNA
double helix solved
with X-rays
2009 Nobel Prize
V. Ramakrishnan*, T.
Steiz, A. Yonath*
Structure of ribosome
(*) some of their structures
were solved at PSI
6. Wikipedia:
X-ray crystallography is the experimental science determining the atomic and
molecular structure of a crystal, in which the crystalline structure causes a beam of
incident X-rays to diffract into many specific directions. By measuring the angles
and intensities of these diffracted beams, a crystallographer can produce a three-
dimensional picture of the density of electrons within the crystal.
X-ray macromolecular crystallography (MX)
Page 6
7. • Particle accelerators are source of the
brightest X-ray beam (multiple orders of
magnitudes as compared to conventional X-
ray tubes), when charged particles travel
through magnetic field
- Effect is nuisance for high energy physics
(undesirable energy loss),
- but it is a blessing for structural science =>
modern storage rings are build exclusively
as light sources.
• Synchrotrons provide continuous X-ray
beam, while X-ray free electron lasers
produce femtosecond long bright pulses
MX at synchrotron
Page 7
9. • 3 experimental stations at the synchrotron
• 1 experimental station at the SwissFEL
• Beamtime is shared between academic and
industrial users
- Industrial customers are mostly pharmaceutical
companies looking for drug binding to potential
drug targets
- Academic users are universities and scientific
institutes worldwide doing basic research in
structural biology
MX at Swiss Light Source and SwissFEL
Page 9
10. • New storage ring to be installed in 2024-2025
• Flux (photons/second) will increase by order of magnitude
• Measurements can be done 10x faster
• Enabling fragment screening method – i.e. single protein target is
crystallized with hundredths or thousands of molecular fragments to
find best drug
- This is like molecular docking, but fully experimentally
Major upgrade in 2024/2025 for SLS 2.0
Page 10
11. • PSI is major detector developer
- Hybrid pixel detectors developed for
CERN high energy physics
experiments
- Design could be used for X-ray
cameras – first PILATUS in 2000s
- PSI start-up Dectris, commercialized
PILATUS and EIGER detectors, most
synchrotrons are equipped with
their detectors
• Currently PSI is rolling out new
generation: JUNGFRAU
Page 11
New detector for SwissFEL and SLS 2.0
12. • Silicon sensor converts X-ray to
electric charge
• Bump bonded to sensor is ASIC, with
dedicated electronics for each pixel
• Pixel has three capacitors allowing
different amplification
• They are dynamically switched during
exposure to adjust for incoming
charge
Page 12
Adaptive gain detector to increase dynamic
range
Aim: measure reliably from 1 to 20,000,000 photons per second
13. Page 13
Adaptive gain detector to increase dynamic
range
0001010111110011
Pixel output in JF:
0001010111110011
Gain: 00:G0 01:G1 11:G2
ADC value: 0001010111110011
Photon number: =
!"# $ %&'&()*+
,*-.∗%01)1. &.&2,3 Gain and pedestal factors are
specific for pixel and gain setting
Prior calibration
Dedicated dark run
14. • Detector is modular
• 524,288 pixels per module
• 2.2 kHz * 524,288 pixels * 16 bit = 2.3 GB/s
- 2 x 10 Gbit/s links
• 4 Mpixel detector (2020)
- 16 x 10 Gbit/s
• 10 Mpixel (2022)
- 40 x 10 Gbit/s
Page 14
Modular detector
4 Mpixel (2020)
10 Mpixel (2022)
16. • Detector is streaming frames over UDP
- Receiver using Linux Datagram Socket
• Conversion of pixel read-out
- CPU SIMD code
• Compression
- CPU compression
First approach: scale conventional architecture
Page 16
17. • Detector is streaming frames over UDP
- Receiver using Linux Datagram Socket
• Conversion of pixel read-out
- CPU SIMD code
• Compression
- CPU compression
First approach: scale conventional architecture
Page 17
Aim
20 GB/s
Reached
5 GB/s
18. WIR SCHAFFEN WISSEN – HEUTE FÜR MORGEN
POWER / OpenCAPI / FPGA architecture
Page 18
19. • Real-time performance
- FPGA design is cycle-accurate, with fixed latency and throughput
• Large memory throughput
- FPGAs with HBM2 have 460 GB/s bandwidth to 8 GB large memory
• Ethernet on-board
- FPGA are made to work with network, often having dedicated “hard” cores for
ethernet
• Development of FPGAs is difficult and time consuming
- Hardware description languages
- PCI Express
• Virtex Ultrascale+ HBM (XCVU33P and XCVU35P)
- Availble as low-profile half-length 75W cards
FPGA are perfect devices for data acquisition
Page 19
20. • C/C++ compiler to produce
hardware design language (Verilog
or VHDL)
• All code is valid C++ code, it can be
executed on CPU and functionally is
generally equivalent
• Dedicated pragma to guide FPGA
synthesis
• It is generally understandable for
software developers, but may
contain strange/inoptimal
constructs from software point of
view
High-level synthesis
Page 20
Bitshuffle for 16-bit numbers
21. • For VU33/35P:
- Size: 8 GB
- Bandwidth: up to 460 GB/s
- Latency: up to 120 cycles @ 200 MHz
• Complex architecture
- 32 x 256-bit AXI3 interfaces
- Either operating as 32 separate memories
- Or as single memory with crossbar (at the cost of up to 50% throughput)
• 256-bit is a problem, as data are 512-bit (PCIe Gen3 x16) or 1024-bit (OpenCAPI,
PCIe Gen4 x16)
• Simulation only with special tools (Cadence Xcelium), impossible with Xilinx tools
High-bandwidth memory
Page 21
22. • PCI Express is CPU-centric bus, as it is design to
support peripherals
• This is good model, when FPGA is a coprocessor
to CPU – which sends data, and waits for reply
=> but for data acquisition, it is FPGA that is
producing the data, CPU has no prior knowledge
which packet will be processed at the time
• DMA is operating on physical addresses: virtual
addresses need to be pinned by kernel (so are
not swapped and moved)
Þ need to maintain own driver
Þ address translation cache possible on FPGA,
but requires memory
PCI Express DMA
Page 22
Xilinx QDMA is a robust
but highly complex
solution for PCI Express –
used to interface FPGAs
with x86 AMD and Intel
CPUs
23. • IBM POWER9 showed great numbers for
I/O and memory throughput in Summit
and Sierra supercomputers
• IBM designed own memory coherent
interface for accelerators
(CAPI/OpenCAPI), which has advantages
over PCIe
POWER architecture
Page 23
Source: Wikipedia
25. OpenCAPI
Page 25
FPGA
board
POWER9
CPU
OpenCAPI
cable
• Predecessor CAPI => proprietary IBM
• Communication over PCIe physical lines
(but different protocol)
• OpenCAPI => consortium model
• Dedicated cabling (8 x 25 Gbit/s lines)
• For POWER10 – this will be default memory interface,
(allowing to have any type of memory attached to CPU + to
“share” memory over network)
26. • Similar difference what 80286/80386 virtual
mode brought to software development
• In OpenCAPI one needs single kernel operation
=> Attach accelerator to running process
• Then, accelerator has access to virtual address
space of running process – it is FPGA that is
initiating the communication
=> Address translation is handled by TLB and OS
=> FPGA sees memory in a fully cache-coherent
way
• All security/reliability/efficiency mechanisms in
CPU and kernel are also present in OpenCAPI
Page 26
What difference brings OpenCAPI?
Source: Wikipedia
27. • Main function for the action contains a pointer to virutal address space
- On device the pointer will be synthesized as 1024-bit master memory-mapped
AXI interface
- On CPU this pointer has to be just set to zero (which is first address of virtual
address space)
• Any cell in virtual memory is just accessed as offset from this pointer
• Only requirement is that memory is aligned to 128-bytes
- No special memory allocator, malloc or mmap is fine
- No pinning/registering
• The same memory buffer class for both simulation and working with device
• For configuration, there is also 4 MiB memory-maped I/O space (like BAR in PCIe)
- On device implemented as slave AXI-lite (32-bit)
How to develop with OpenCAPI?
Page 27
28. • Open source “shell” mantained by IBM
• http://github.com/OpenCAPI/oc-accel
• Provides ready made tool to work with OpenCAPI (from transceiver setup to
AXImm bridge)
• Provides preconfigured interfaces for I/O peripherals (HBM, 100G, NVMe)
• Provides simulation environment
- One can simulate both SW and HW in a single simulation (both user FPGA
design and software are not modified from their “real” implementation)
OC-Accel
Page 28
29. WIR SCHAFFEN WISSEN – HEUTE FÜR MORGEN
Jungfraujoch – FPGA implementation
Page 29
30. Page 30
Jungfraujoch server
Ethernet
UDP/IP
Dark
current
Conversion
Strong
pixel finder
Bitshuffle
Memory
writer
FPGA board with OpenCAPI interface
- Data acquisition
- Initial data analysis
- Pre-compression
(2.5 Mpixel/board for JF)
Up to 50 GB/s acquisition and
data analysis in a single 2U
IBM POWER9 server with 1-4 FPGA
boards
Frame
summation
31. Page 31
Jungfraujoch FGPA streaming design
Modular design
• Stream of data handled by successive cores doing work in parallel
à throughput and latency of each core is determined by the hardware design
• Extra stages can be relatively simply added, option to bypass cores
• All cores are C++ functions, connected with AXI-Stream FIFOs
• As buffering is expensive on FPGA, it is best suited for algorithm that have limited
dependencies between frames
Ethernet
UDP/IP
Dark
current
Conversion
Strong
pixel finder
Bitshuffle
Memory
writer
Frame
summation
32. Page 32
Jungfraujoch
Ethernet UDP/IP core
Processes ethernet packets from network, ignores unnecessary packets, reads
frame header to get frame number, module number, etc.
Ethernet
UDP/IP
Dark
current
Conversion
Strong
pixel finder
Bitshuffle
Memory
writer
Frame
summation
33. Page 33
Jungfraujoch
Dark current core
This cores is responsible for calculating moving average of detector frames.
Calculated value is used as dark current (pedestal) for subsequent frames.
Ethernet
UDP/IP
Dark
current
Conversion
Strong
pixel finder
Bitshuffle
Memory
writer
Frame
summation
34. Page 34
Jungfraujoch
Conversion core
This cores translates JUNGFRAU read-out into units of energy or photon counts.
It benefits from very fast HBM2 memory within the FPGA (460 GB/s). Data
leaving this core can be used for processing by data analysis software.
Ethernet
UDP/IP
Dark
current
Conversion
Strong
pixel finder
Bitshuffle
Memory
writer
Frame
summation
35. Page 35
Jungfraujoch
Frame summation core (work in progress)
As data that left gain correction core are on linear scale, they can be summed to
reduce downstream data rate, if lower frame rate is needed, as compared to
detector.
Ethernet
UDP/IP
Dark
current
Conversion
Strong
pixel finder
Bitshuffle
Memory
writer
Frame
summation
36. Page 36
Jungfraujoch
Strong pixel finder core
This is first step of spot finding algorithm (for example COLSPOT). It identifies
pixels that are stronger than given number of standard deviations of their
neighborhood.
Ethernet
UDP/IP
Dark
current
Conversion
Strong
pixel finder
Bitshuffle
Memory
writer
Frame
summation
37. Page 37
Jungfraujoch
Bitshuffle
FPGAs are bit order agnostic. Therefore exchanging bit order in popular
compression prefilter is pretty much for free on FPGA.
Ethernet
UDP/IP
Dark
current
Conversion
Strong
pixel finder
Bitshuffle
Memory
writer
Frame
summation
38. Page 38
Jungfraujoch
Host memory write
Address in host memory buffer is calculated and data forwarded to host memory
via OpenCAPI. Additional image statistics are saved as well.
Ethernet
UDP/IP
Dark
current
Conversion
Strong
pixel finder
Bitshuffle
Memory
writer
Frame
summation
40. Jungfraujoch FPGA power usage is 18 W/board
for the whole streaming functionality
Page 40
Xilinx Vivado Power Report
2 boards for 4 Mpixel JUNGFRAU and 4 boards for 10 Mpixel JUNGFRAU
41. • VU33P or VU35P with 8 GB of HBM2
• OpenCAPI link and PCIe Gen3 x16 (or two
PCIe Gen4 x8)
• Small flash (2 kb) to store MAC address,
board IR
• QSFP-DD optical socket (same as QSFP28,
but with 8 lanes for 2x100G) =>
compatible with QSFP28 transceivers
• Up to 75W
Alpha Data 9H3 board
Page 41
42. • Software tests – Catch2
- 8 min
- Among other software tests includes 13
FPGA action tests (whole SLS code)
- Automated tests cover 95% lines of high-
level synthesis code
- Covers most of the functionality
correctness – including address calculation
- Main limitation is debugging of FIFOs
parallel behavior (deadlocks, etc.)
• Hardware simulation – Cadence Xcelium
- 4 hours
- Collection of 8 frames from single module
- Checks if hardware description is correct,
can find problems with synchronization,
and other, very rare, issues
- Too slow to verify functionality
OpenCAPI programming - testing
Page 42
43. • Detector and data acquisition system was sent in
November for an experiment in Photon Factory, KEK
• More than 2,000 datasets collected for protein
targets, few real-life native-SAD structures solved
• Due to pandemic, detector support and
development (including deployment of new FPGA
design) was done fully remotely from Switzerland
Commissioning in KEK (Jan – May 2021)
Page 43
BL-1A Photon Factory
JUNGFRAU detector (up)
tested in helium chamber
for native-SAD
measurements with 3.75
keV X-rays
44. Page 44
Structure of Nucleocapsid Phosphoprotein from
SARS-CoV-2 solved in 1 second
• Crystal was previously measured with
conventional setup at our beamline –
with measurement taking longer than
one minute
• With JUNGFRAU detector and
OpenCAPI readout, 2000 images
collected in one second allowed to
solve structure of this protein
• Experimental team: Filip Leonarski, Sylvain
Engilberge, Vincent Olieric, Meitian Wang (MX
Group), Aldo Mozzanica (PSI Detector Group)
• SARS-CoV-2 protein was produced by Zinzula, L.,
Basquin, J., Bracher, A., Baumeister, W. (MPI,
Martinsried)
45. Possible gain from using FPGA based system
Page 45
Courtesy: B. Mesnet (IBM)
46. Possible gain from using FPGA based system
Page 46
Courtesy: B. Mesnet (IBM)
47. MX Group (PSI)
• Vincent Olieric
• Takashi Tomizaki
• Chia-Ying Huang
• Sylvain Engilberg
• Justyna Wojdyła
• Meitian Wang
Detector Group (PSI)
• Aldo Mozzanica
• Martin Brückner
• Carlos Lopez-Cuenca
• Bernd Schmitt
Science IT (PSI)
• Leonardo Sala
Controls (PSI)
• Andrej Babic
• Leonardo Hax-Damiani
SLS management (PSI)
• Oliver Bunk
Photon Factory, KEK
• Naohiro Matsugaki
• Yusuke Yamada
• Masahide Hikita
MAX IV
• Jie Nan
• Zdenek Matej
Uni Konstanz
• Kay Diederichs
LBL
• Aaron Brewster
DLS
• Graeme Winter
• DIALS Team
ESRF
• Jerome Kieffer
IBM Systems (France)
• Alexandre Castellane
• Bruno Mesnet
InnoBoost SA
• Lionel Clavien
Acknowledgements
Page 47