Data Acquisition System Based on the Bluesky Suite in the Shanghai Synchrotron Radiation Facility

Featured Application

This system is characterized by multi-channel simultaneous acquisition capability, high signal-to-noise ratio and high universality. On one hand, the system is capable of being extended to other beamlines efficiently and easily. On the other hand, it improves the experimental efficiency and reduces experimental error.

Abstract

The Shanghai Synchrotron Radiation Facility Phase-II Project is now under construction and it consists of 16 newly built advanced beamlines. The experiments carried out at these new beamlines tend to be characterized by high flux, multi-mode, ultra-fast frequency, in situ and dynamic loading. To meet these emerging experimental demands of SSRF Phase-II beamlines, the authors have developed a unified and extensible data acquisition system based on Bluesky suite. It has been applied at the BL07U Spatial-resolved and Spin-resolved ARPES and magnetism beamline. The experiment results show that with up to 5 channel signals, the system is capable of collecting total electron yield signals of experimental sample and standard sample, and incident photon flux signals simultaneously. It improves the user experimental efficiency and quality. In addition, thanks to Bluesky suite, all devices can be abstracted as motors or detectors and then represented by a Python object. Therefore, with the architecture, modules and interface developed for BL07U, the system can be extended to more beamlines efficiently and easily.

1. Introduction

The Shanghai Synchrotron Radiation Facility (SSRF) is a third-generation synchrotron light source equipped with a 150 MeV linear accelerator, a 3.5 GeV booster, a 3.5 GeV storage ring, beamlines, experimental endstations, and supporting facilities. SSRF opened to users in 2009 with 7 Phase-I beamlines. Another 6 beamlines were built as part of the Follow-up Beamline Program (FBP) in the next few years. In 2016, SSRF launched the largest follow-up beamline construction project—SSRF Phase-II Beamline Project.

Table 1. Detailed overview of the 16 state-of-the art beamlines of the SSRF Phase-II Beamline Project.

ID	Name	Source	Energy Range
BL20U1 BL20U2	Energy Material Beamline (E-line)	IVU+EPU	130 eV~18 keV
BL11B	Hard X-ray Spectroscopy Beamline	BM	5~30 keV
BL16U1	Medium-energy Spectroscopy Beamline	U	2.1~16 keV
BL07U	Spatial-resolved and Spin-resolved ARPES and Magnetism Beamline (S2-line)	Twin EPU	50~2000 eV
BL02U1	Membrane Protein Crystallography Beamline	IVU	7~15 keV
BL02U1	Surface Diffraction Beamline	CPMU	4.8~28 keV
BL03SB	Laue Micro-diffraction Beamline	SB	7~30 keV
BL13U	Hard X-ray Nanoprobe Beamline	IVU	5~25 keV
BL18B	3D Nano Imaging Beamline	BM	5~14 keV
BL05U& BL06B	Dynamics Beamline (D-line)	IVU+BM	5~25 keV 10~10,000 cm−1
BL10U1	Time-resolved USAXS Beamline	IVU	8~15 keV
BL16U2	Fast X-ray Imaging Beamline	CPMU	8.7~30 keV
BL10U2	Biosafety P2 Protein Crystallography Beamline	IVU	7~18 keV
BL13SSW	Radioactive Materials Beamline	W	5~50 keV
BL12SW	Ultra-hard X-ray Applications Beamline	SCW	30~150 keV
BL03SS (ID)	Laser Electron Gama Source Beamline (SLEGS)	ID	0.4~20 MeV
BL09B	X-ray Test Beamline	BM	4~30 keV

shows the detailed overview of the 16 state-of-the art beamlines of the SSRF Phase-II Beamline Project. When the Phase-II project completes by the end of 2023, Shanghai Light Source will have nearly 40 beamlines in operation supporting users all over the world [1,2].

The SSRF Phase-I and the FBP beamline control system are normally based on the Experimental Physics and Industrial Control System (EPICS) [3,4,5], while there is no unified, extensible and interface-standardized framework for the data acquisition system for the experimental endstations.

Table 2. The development environment of the endstation data acquisition system at the SSRF Phase-I Project and the FBP beamlines.

	ID	Software Development Environment
SSRF Phase-I Project	BL08U1-A, BL13W1	C Language
	BL08U1-B, BL15U1	EPICS
	BL14W1, BL16B1	Labview
	BL14B1	SPEC
	BL17U1	Blu-Ice/DCS
FBP	BL01B1, BL09U	Niche business software
	BL17B, BL18U, BL19U1	Blu-Ice/DCS
	BL19U2	EPICS
	BL02B, BL03U	Niche business software

shows the development environment of the endstation data acquisition systems at the SSRF Phase-I Project and the FBP beamlines [6,7,8,9,10,11,12]. As can be seen from the table, the development environment is complex and diverse, from c language to Blu-Ice/DCS, which makes maintenance and upgrade of these data acquisition systems extremely difficult and labor-intensive.

Compared with the SSRF Phase-I and the FBP, the SSRF Phase-II Project aims to greatly expand the experimental capacity of SSRF, making the resolution of time, space, energy and momentum in SSRF almost reach the limit of the third-generation synchrotron radiation light source. The state-of-the art Phase-II beamlines bring great changes in many aspects, such as sample scan, hardware equipment and data amount. For example, the sample scan expands from one-dimensional and two-dimensional to a multi-dimensional continuous scan. The new generation hardware equipment includes a huge number of area detectors with a larger imaging area and smaller pixels. Additionally, high time resolution accompanied with fast sampling frequency results in a high data-generation rate and high requirements for data transmission peak value. More specifically, beamlines in the SSRF Phase-II Project will have ultra-high spatial resolution of 10 nanometer and ultra-fast time resolution of 100 picosecond. The existing data acquisition system can hardly adapt to the changes mentioned above and meet the requirements of these new beamlines. It is necessary and reasonable to develop a uniform and universal data acquisition system based on the common requirements of the SSRF Phase-II beamlines. With overall consideration and planning, and within a unified software environment and architecture, it can significantly avoid repeated development and reduce the workload for long-term operation, maintenance, and upgrades.

At present, the international mainstream data acquisition architectures for synchrotron radiation light sources and other large facilities are Bluesky suite [13], Sardana [14] and Generic Data Acquisition (GDA) [15]. Bluesky suite is a set of tools that can assist beamline scientists to realize complex experimental procedures with little understanding of low-level control or data. It is developed by the American National Synchrotron Light Source-II (NSLS-II). Additionally, NSLS-II has developed and fielded a data acquisition system based on Bluesky Suite at 12 NSLS-II beamlines. Bluesky suite currently only interface with EPICS. Sardana development was started at the ALBA synchrotron, which is a third-generation synchrotron light facility located in Barcelona that is supported today by a larger community, which includes several other laboratories and individuals (ALBA, Deutsches Elektronen Synchrotron (DESY), Swedish national synchrotron laboratory MaxIV, Solaris National Synchrotron Radiation Center, European Synchrotron Radiation Facility (ESRF)). Sardana is based on Tango. The GDA project is an open-source framework for creating customized data acquisition software for science facilities, such as neutron and X-ray sources. The software is Java/Eclipse-based, free and released under the GPLv3. The development team at Diamond light source have developed the GDA for the majority of beamlines across all the scientific techniques at Diamond.

SSRF beamline control systems are mainly based on EPICS. In addition, Bluesky suite is designed to interface with EPICS and possess experiment generality. Therefore, we adopt Bluesky suite to develop a uniform and universal data acquisition system for SSRF Phase-II beamlines. Bluesky suite contains three main parts: Bluesky, Python abstractions of hardware (ophyd) and databroker [6]. Bluesky and ophyd are the co-developed acquisition software. Bluesky orchestrates the execution of an experiment and communicates with low-level hardware via ophyd. Ophyd is a hardware abstraction library and it represents hardware in Python frame, as hierarchical objects grouping together related values form the underlying control system; it interfaces with EPICS via PyEpics. Many facilities use ophyd to integrate with control systems that use EPICS. Ophyd is typically used with the Bluesky Run Engine for experiment orchestration and data acquisition. Bluesky collects measurements and metadata into documents which are stored in databases and can be used in any live visualization or processing. Databroker can access these documents and export them into any desired format.

So far, data acquisition systems based on SSRF Bluesky suite have been developed and applied at the BL07U beamline and the BL16U1 beamline of SSRF. In particular, the BL07U data acquisition system has been highly evaluated after it opened to users [16,17].

2. Materials and Methods

2.1. Sub-Assemblies

X-ray absorption spectroscopy (XAS) is a technique that probes the local atomic structure and chemical environment of elements in a material by measuring the attenuation of X-rays as they pass through a sample [18,19,20]. The X-ray absorption spectrum obtained from XAS is characterized by a series of peaks and edges that correspond to the absorption of X-rays by the different electron shells of the absorbing atoms. XAS comprises two regions: X-ray absorption near edge structure (XANES/NEXAFS), which provides information about the unoccupied electronic states of the absorbing atoms, and extended fine structure (EXAFS), which provides information about the local atomic environment of the absorbing atoms. As a scientific experimental method, XAS plays an important role in material, physics, chemistry, and life sciences research [21,22,23,24,25]. XAS can be collected in multiple ways, such as the total electron yield (TEY) method, the total fluorescence yield (TFY) method, the constant final state (CFS) method, and the constant initial state (CIS) method [26]. The TEY method with the sample current mode is widely used because it can be obtained using a relatively simple experimental setup. However, because the escape depth of an electron in solids is from a few to a few-to-ten angstrom, the TEY method is employed to obtain sample surface information [27,28]. The TFY method measures the fluorescent X-ray that can arise from a deeper region of the sample.

The BL07U beamline adopts the TEY method and the TFY method for XAS. The BL07U beamline is a new extreme ultraviolet and soft X-ray beamline in the SSRF Phase-II Project. BL07U is the combination of XAS, X-ray magnetic circular dichroism (XMCD), X-ray magnetic linear dichroism (XMLD), Nano resolved-Angle resolved photoemission spectroscopy (Nano-ARPES) and Spin resolved-Angle resolved photoemission spectroscopy (Spin-ARPES) beamline. The beamline is equipped with a pair of elliptical polarized undulators (EPUs) and a variable-included-angle plane-grating monochromator, and delivers circularly polarized or linear-polarized X-rays in the energy range from 50 to 2000 eV. The beamline has two branches: one dedicated to Nano-ARPES, which has a minimum spot size of only 200 nm, and another branch dedicated to Spin-ARPES, vector magnetic field and high magnetic field endstations. The energy-resolving power of BL07U is over 9000 at 867 eV and 15,000 at 91 eV. The photo flux of BL07U is over 2.5 × 10¹¹ photons/s/0.01%BW at 867 eV and 2.6 × 10⁹ photons/s/0.01%BW at 91 eV. The spot size is 150 × 50 ${\mathsf{\mu }\mathrm{m}}^2$ at 91 eV.

The TEY and TFY experiments can be carried out simultaneously at the BL07U beamline. Figure 1 shows the hardware of the BL07U XAS measurement system. It consists of 5 channels for measurement. The incident X-ray photon flux I₀ can be measured in two channels. One channel detects I₀ by a toroidal mirror (TM) with a gilded surface in the X-ray light path. Another channel detects I₀ by a clean gold mesh in the X-ray light path. The sample signal can be obtained through three other channels. The standard sample current is connected to the input of the current amplifier by using a shield cable. The standard sample has a known high-purity composition; hence, it is commonly used to calibrate and test the performance of instruments at the beamline. It can also be used to infer the composition of the sample to be analyzed by comparing the spectra, as well as to reduce the error on the beam energy calibration if it is acquired simultaneously. The TFY method measures the fluorescent X-ray that can arise from a deeper region of the sample. For the TEY experiment, the sample current is also connected to the input of the current amplifier via a shield cable. For the TFY experiment, the fluorescence signal of the sample is detected by a photodiode (PD) that was placed in front of the sample and outside the X-ray light path. The PD signal is connected to the input of the current amplifier. All of the above 5 channel measurements can be removed, placed into the X-ray path, or half-cut by the X-ray by controlling the motor of the corresponding bracket.

Five Stanford SR570 amplifiers were used to measure the TM current, the gold mesh current, the standard sample current, and the TEY and the TFY signals, respectively. The SR570 has a sensitivity setting from 1 $\mathrm{pA}/\mathrm{V}$ to 1 $\mathrm{mA}/\mathrm{V}$ [29]. In the experiment, the sensitivity was set from 20 $\mathrm{pA}/\mathrm{V}$ to 100 $\mathrm{nA}/\mathrm{V}$, which could cover the current range. The SR570 contains two first-order RC filters and users can configure cutoff frequency and type from the front panel. Together, the filters can be configured as a 6 or 12 dB/oct rolloff low-pass or high-pass filter, or as a 6 dB/oct rolloff band-pass filter. Cutoff frequencies are adjustable from 0.03 Hz to 1 MHz [29]. In the experiment, the SR570 was configured as a 12 dB/oct rolloff low-pass filter and the cutoff frequency was set at 30 Hz. This parameter setting could effectively suppress the noise caused by the low temperature, magnetic field and vibration of vacuum equipment. The SR570 provides an input offset-current adjustment to suppress any undesired DC background currents. Offset currents can be specified from ±1 pA to ±1 mA in roughly 0.1% increments [29]. In the experiment, the input offset current was set from 1 $\mathrm{pA}$ to 1 $\mathrm{nA}$, which could suppress any unwanted DC background currents.

The SR570 converted current signal into direct voltage. Subsequently, the Volt-Frequency converter received the voltage and converted it to frequency to improve linearity and enhance anti-interference capacity. Each Volt-Frequency converter has two inputs, and three Volt-Frequency converters were used. Finally, the SIS3820 board received the frequency signals.

The SIS3820 combines the functionality of multi-scalers and standard counters and without compromising performance [30]. One key feature of the SIS3820 is that it has 32 channels, which ensures that 5 channels of data can be collected simultaneously during the experiment. The SIS3820 accepts inputs with a frequency of up to 250 MHz. In standard counter/scaler mode, it can read data on the fly with an accuracy to the least signal. No count is lost in read on the fly mode. Compared with the picoammeter, the SIS3820 can realize the fly scan function.

The SIS3820 counted the frequency signal according to the trigger signal from the embedded CPU MVME5500 board and sent the counting to the MVME5500 board. The MVME5500 and SIS3820 board ran in a Virtual Machine Environment (VME) machine and communicated via VME bus. The MVME5500 board transmitted these counting data to the computer via Ethernet according to the commands from the experimental computer.

2.2. Control and Data Acquisition Architecture

The architecture, shown in Figure 2, of the control and data acquisition software developed for the BL07U line is based on Bluesky suite and EPICS. The design concepts of these packages allow for building a versatile data acquisition system. From the bottom to the top are the hardware layer, the control layer, the data acquisition layer and the operator interface layer.

Specifically, the hardware layer consists of all the setups, including one toroidal mirror, one double crystal monochromator, five SR570 amplifiers, three Volt-Frequency converters, one scaler SIS3820, one embedded CPU MVME5500 and one Virtual Machine Environment (VME) machine.

The control layer is developed based on EPICS. EPICS consists of two major components: Input/Output controller (IOC) and Operator Interface (OPI) [5,31]. The IOC uses Process Variables (PVs) to control spectroscopy setups. For example, via corresponding PVs, the monochromator can change its energy, angle and status; and the scaler SIS3820 can set its dwell time, status and 32-channel scaler value. IOC runs on the embedded CPU MVME5500 in the vxWorks operating system. OPI runs on the experimental computer in the CentOS operating system. OPI and IOC generally exchange PVs data by EPICS Channel Access (CA) protocol via Ethernet.

The data acquisition layer is developed based on Bluesky suite, which is a suite of Python code. EPICS provides CA application interface libraries with various high-level languages, and the interface with Python is PyEpics [5]. The control layer and the data acquisition layer communicate through the PyEpics mechanism.

The operator interface layer offers a graphical user interface (GUI), and the GUI is developed based on Control System Studio (CSS) [32]. CSS is an OPI application software supported by EPICS. CSS communicates with the control layer in real time through the CA mechanism and the data acquisition layer in real time through PyEpics, respectively.

2.3. Control and Data Acquisition Software

The control and data acquisition software of BL07U is developed based on the above architecture. Because of the design concepts of Bluesky suite and EPICS, using the above architecture as a base allows for building a versatile data acquisition system. The data acquisition system in BL07U consists of six types of working modules, which are the time scan module, the normal energy scan module, the segmented energy scan module, the monochromator and Elliptically Polarizing Undulator (EPU) 58 linkage module, the monochromator and EPU90 linkage module, and a plotting module. The data acquisition system has the capability of running different irradiation experiments simultaneously. Figure 3 shows the software flow of all scan modules. Figure 4 shows the GUI software and a live visualization result.

For the time scan module, users set the energy value and monitor the X-ray photon flux in real time. For the normal scan module, users set the starting and ending energy points and the dwell time, and run the function scripts by clicking the START button. For the segmented scan module, users can use different scan precision in different energy ranges. For the monochromator and EPU linkage module, users set the energy parameters, and the function scripts automatically calculate the corresponding gap value. By combining energy and gap, the quality of absorption spectrum measurement can be improved. The linkage module consists of linkage with EPU58 and EPU90. The linkage with EPU58 is suitable for the measuring of elements with large energy absorption edge, while the linage with EPU90 is suitable for the measuring of elements with small energy absorption edge.

For the plot module, users can use the saved data to reproduce spectra in one experiment or compare spectra in multiple experiments. Figure 5 shows the plot module interface. With the plot module, users can select XAS results in different experiments, and click the Plot button. The Python program can automatically read and compare data, so as to quickly and intuitively determine whether the samples have XMCD or XMLD signals.

For all scan modules, experiments can be run unattended, automatically suspending and resuming if needed. The data (live) are exported in .cvs format and saved on the experimental computer. During all the experiments, the data are available for online visualization and processing.

The development of BL07U software focuses on the use of the SIS3820 and the monochromator. EPICS offers two modules for SIS3820 control: mca [33] and std [34]. Drivers of the mca and std modules are both written to support the use of the SIS3820 as a multichannel scaler. Both modules offer 32 records corresponding to 32 channels of SIS3820. There are additional records to control the start-and-stop acquisition of the device, and control the dwell time and channel-advance source. The channel-advance source is used to tell the hardware whether to advance the current channel according to its internal clock, or in response to an external channel-advance signal. The main difference between the two modules is the data storage method during the experiment. The std module provides a variable for each channel that is used to update the real-time data. Meanwhile, the mca module provides an array for each channel that is updated to store real-time data.

After comprehensive consideration of live visualization and the ability given by mca and std, we decided whether to use std or mca module at control layer.

Table 3. The configurations of five scan modules.

Scan Module	Live Visualization	Hardware Layer	Control layer (EPICS Module: std or mca)	Data Acquisition Layer (Bluesky Suite Pre-Built Device)	Customized Class
Time scan	CSS	SIS3820	mca	/	/
Normal energy scan	LivePlot	Monochromator, SIS3820	std, motor	EpicsScaler, EpicsMotor	blcgscaler
Segmented energy scan	LivePlot	Monochromator, SIS3820	std, motor	EpicsScaler, EpicsMotor	blcgscaler
EPU58 linkage	LivePlot	Monochromator, SIS3820, EPU58	std, motor	EpicsScaler, EpicsMotor	blcgscaler, gap58Motor
EPU90 linkage	LivePlot	Monochromator, SIS3820, EPU58	std, motor	EpicsScaler, EpicsMotor	blcgscaler, gap90Motor

shows the configurations for these five scan modules. The time scan module adopted the mca module at control layer and adopted the array-friendly CSS to realize live visualization. In the experiment of the normal energy scan module and the segmented energy scan module, the SIS3820 should collect data once after each step of the monochromator. These two modules selected the std module at control layer and called EpicsMotor and blcgscaler class in the data acquisition layer. The EpicsMotor class is offered by ophyd. By calling EpicsMotor, data acquisition layer invocation of the monochromator can be achieved. The blcgscaler is a custom class. The blcgscaler class is inherited from the Device class in ophyd. The blcgscaler class added a parameter called spectrum and defined the spectrum value in the trigger function. This spectrum was used to display the absorption spectrum in real time. By calling the customized blcgscaler class, the system can collect the multi-channel data simultaneously and realize the live visualization of the multi-channel data.

Compared with the normal energy scan module and the segmented energy scan module, the EPU58 and EPU90 linkage module called two customized class gap58Motor and gap90Motor. The gap58Motor and gap90Motor class added a parameter called gap_value and defined the gap_value value in the trigger function. The gap_value value corresponds to the monochromator value. The gap_value value is assigned to the EPU when the monochromator reaches the target value, and the EPU then moves to the gap_value value. By calling gap58Motor and gap90Motor, the monochromator and EPU can be interlinked.

Moreover, Bluesky suite contains a number of pre-built devices for common hardware as well as the tools to build customized devices. Bluesky suite encapsulates many common functions such as scan and count. These ensure that the BL07U data acquisition system is very versatile.

3. Results

The experiment regarding the Ti L-edge absorption spectra of the standard sample TiO₂ and the sample containing Ti element was carried out in BL07U using TEY method. This experiment adopted the normal energy scan module. The range of the scan energy was from 452 to 470 $\mathrm{eV}$. The scan step was 0.1 $\mathrm{eV}$. The dwell time at each energy point was 0.2 s. These preceding parameters were set through the GUI. The normal energy scan module can measure 32 channels of data simultaneously. In this experiment, the channel-2 was selected to measure the photon flux of standard sample TiO₂ in semi-cut light state. The channel-4 was selected to measure the incident photon flux detected by a clean gold mesh. The channel-5 was selected to measure the photon flux of the sample. The selection of channels was achieved through the interface. After the samples were placed and the above parameters were set through the interface, the energy scan was carried out automatically by clicking the Start button. In the process of energy scan, the Python script realized live visualization by calling the LivePlot function. When the energy scan was completed, the experimental data was automatically saved to a .cvs file.

The above experimental data were processed, and the spectra of channel-2 and channel-5 were normalized to the incident photon flux of channel 4. The TEY spectrum of standard sample TiO₂ is shown in Figure 6. The TEY spectrum of sample is shown in Figure 7. The incident photon flux is detected by a clean gold mesh and shown in Figure 8.

4. Discussion

Using the BL07U data acquisition system, users obtained the Ti L-edge absorption spectra of TiO₂ standard sample and users’ samples at the same time. The peaks of absorption spectra in Figure 6 and Figure 7 were roughly the same, and Figure 6 is the standard sample of TiO₂, so it indicated that the user’s sample was a Ti compound. The difference in the peaks’ shapes between Figure 6 and Figure 7 indicated that the coordination and valence states of Ti were different. The data in Figure 6 and Figure 7 were obtained simultaneously, so the difference in the peaks’ shapes was not caused by the difference in incident photon flux.

The simultaneous measurement of multiple channels has three advantages: First, it shortens the experiment time. Second, the experimental results suffer from less systematic uncertainties as the effect induced by the non-uniformity of the incident photon flux is ruled out. Third, the error on the beam energy calibration can be reduced by inquiring the theoretical value of the peak position of the standard sample.

5. Conclusions

A unified and extensible data acquisition system based on Bluesky suite has been developed. The data acquisition system can be invoked and executed through input instructions or through GUI. SSRF users prefer and are accustomed to using the GUI interface, so all user-defined parameters, initialization, composition, execution procedures, etc., are implemented through the GUI interface. In SSRF, massive data storage and full life-cycle management, real-time data analysis, data mining and artificial intelligence and automation are possible, and those functions are supported by the Big Data Science Center system [35,36].

The experiment result at BL07U shows that with up to five channel signals, the system is capable to collect signals of experimental sample, standard sample and incident photon flux simultaneously. For users, when conducting experiments at the BL07U, TEY and TFY data acquisition can be implemented at the same time thanks to the hardware layout of the beamline. In addition, the SR578 in the hardware layer filters out the medium and high frequency noise and suppresses the DC background current, giving the system the characteristics of low noise and high signal-to-noise ratio. User experimental efficiency and quality are highly improved.

In addition, for the Bluesky suite, all devices are abstracted into only two categories: motors and detectors. Whether they are motors or detectors, they can be represented by a Python object, in that they can be read or set. The interface is general enough to address every kind of hardware SSRF has encountered. Therefore, with the architecture, modules and interface developed for BL07U, the system can be extended to more beamlines efficiently and easily.