Drivers for DAQ board operation

For high-speed read-out of a Timepix4 device its high-speed links are converted from electrical to optical using Samtec FireFly:sup:TM devices located on the SPIDR4 Control board. The optical fibers from the FireFly devices are connected to a PCIe data-acquisition (DAQ) FPGA board running dedicated firmware, plugged into a PCIe slot of a PC running a LinuxOS. The board running the firmware receives and decodes the data (or whatever data processing the firmware is programmed to do) and the data is transferred by means of DMA (Direct Memory Access; CPU-independent data transfer) operations into the PC host memory for further processing and/or transport.

To operate the PCIe DAQ board two drivers are required; one driver provides access to the board through its registers accessible through the PCIe interface, for configuration and monitoring of the board and its firmware; the second driver allocates a large chunk of contiguous PC host memory and then manages it, serving multiple clients requesting a chunk or multiple chunks of contiguous memory of the desired size, i.e. as the destination for a DMA operation transferring Timepix4 data from the board to PC memory.

The drivers are called wupper and cmem_rcc respectively, and have been obtained from the CERN ATLAS FELIX project (https://atlas-project-felix.web.cern.ch), with small adaptations to remove dependencies on a few ATLAS libraries. The FELIX project develops a PCIe data-acquisition board and system using the PCIe DMA engine, which is used in the SPIDR4 DAQ board firmware too.

Updates of both drivers are still occurring now and then, so we must benefit from keeping track of changes in these drivers. The code for the drivers (both wupper and cmem_rcc), wupper API library, wupper register map, wupper register map code generation and tools for wupper can all be found in the SPIDR4 project wupper git repository.

wupper driver

Both the PCIe DMA engine FPGA design, incorporated in the SPIDR4 DAQ board firmware, and the driver developed alongside it are called wupper.

The main purpose of the wupper driver is to provide access to the so-called PCIe BAR0, BAR1 and BAR2 (BARD = Base Address Register) spaces, where the firmware maps the registers which are used to configure the DMA controller(s), as well as a range of registers provided by the specific firmware version for configuration and monitoring of the DAQ board plus firmware.

Access to the driver functions is by means of an API library (located in folder wupperapi), which provides functions for opening and closing the device, configuring DMA controllers, configuration and control of baord interrupts and read and write access to onboard firmware registers and register bitfields on the basis of symbolic names. The latter is made available by the register definitions in the register map library, located in folder wupperregmap. The register map code is generated from a firmware description yaml file, the tools for which are located in wuppercodegen, including documentation.

To see information about the board(s) and firmware check the output of:

cat /proc/wupper

For more background information on the wupper PCIe DMA engine and associated driver see this document:

https://www.nikhef.nl/~n48/wupper.pdf

Be aware that the software section of this document is not up-to-date with the software provided for SPIDR4 (preferably this should be added to this document). The software tools for the board may be subject to future changes and additions since the SPIDR4 DAQ board gateware is under development.

cmem_rcc driver

The so-called cmem_rcc driver starts up preferably early in the booting process of the PC Linux OS and allocates a configurable amount of host main memory (usually multiple GBytes), ensuring it is a contiguous range of physical memory, which the DMA operations by the PCIe board require.

The driver manages the memory buffer by providing applications the ability to request a (contiguous) memory segment of any desired size (as long as the driver has a contiguous segment of that size available), which the driver allocates from its memory buffer and assigns to the application, that receives a pointer to the allocated memory segment. The allocated buffer is returned to the driver when the application exits, whether cleanly or not, and the memory is returned to the available (free) memory managed by the driver and available for the next ‘client’.

To see the current status of the memory buffer and currently allocated segments have a look at the output of:

cat /proc/cmem_rcc

For more background information on this driver see this document:

https://www.nikhef.nl/~n48/cmem_rcc.pdf.

The document includes a description of an API library for the driver, but since only a small number of ioctl() calls suffice for our purposes, it was decided not to include the API for the SPIDR4 applications (also because, if unchanged, the API code brings in a few more dependencies on libraries from the ATLAS project, which we have tried to avoid).