Jungfraujoch: 38 GB/s Real-Time Serial Crystallography Data Acquisition, Processing and Smart Reduction

The exponential growth in data generation from advanced X-ray detectors presents a critical challenge in macromolecular crystallography. We introduce the Jungfraujoch server, a revolutionary heterogeneous computing system that achieves unprecedented real-time data processing capabilities of 38 GB/s from a 9 Mpixel JUNGFRAU detector operating at 2 kHz. By synergizing FPGA-based data acquisition, GPU-optimized indexing, and CPU-driven compression, our system enables intelligent data reduction that maintains 100% of indexed frames while selectively preserving only 5% of non-indexed frames. This approach achieves a 96% true positive rate and 4% false negative rate in identifying non-indexed data, reducing data throughput from 38 GB/s to approximately 2 GB/s without compromising data quality. Field testing at the MicroMAX beamline at MAX IV demonstrates sustained performance at full frame rates, while successful deployments at multiple facilities showcase the versatility of the system beyond macromolecular crystallography and JUNGFRAU detectors. The Jungfraujoch server represents a significant advancement in scientific data management, offering a scalable solution for next-generation light sources and establishing a new paradigm for real-time data processing in experimental physics. Data collected with DCM: /data/visitors/micromax/20230350/20240521/raw/LysozymeJet8/LysozymeJet8-lysozyme1/LysozymeJet8-lysozyme1_1_master.h5 - Flux: ~10e12 ph/s Data collected with MLM: /data/visitors/micromax/20230350/20240521/raw/run2/LysozymeJet5/LysozymeJet5-lysozyme1/LysozymeJet5-lysozyme1_2_master.h5 - Flux: ~10e14 ph/s