Written by Ariana Eisenstein, Member of Technical Staff, LeafLabs
Lotus is a LeafLabs project supported by a Small Business Innovation Research grant from the National Institute of Mental Health of the National Institutes of Health, under Award Number R43MH109332, with the goal of using light-field microscopy to visualize neural activity in the zebrafish brain at single neuron resolution. In order to perform high-speed volumetric calcium imaging, used to visualize neural activity, we have proposed a microscope design with a frame rate of 300FPS and resolution of 4096 by 3072 pixels, to capture the entire neonatal zebrafish brain.
The CMV12000 is a high-speed CMOS image sensor. It has a resolution of 4096 by 3072 pixels (22.5mm x 16.9mm) and can operate at a maximum rate of 300FPS. At highest resolution and frame rate, the sensor outputs data at over 3.77 GBPS. The data, synchronized clock, and a control signal are output over LVDS high-speed data lines. The sensor is controlled through a SPI bus and several control lines (clock, reset, capture).
The basic architecture of the sensor interface is as follows: the sensor is mounted on a custom breakout board and all digital lines are broken out to a FMC connector. The custom board is mounted on a KC705 FPGA development board via the FMC. The KC705 is mounted in a host PC’s PCIe slot. On the host PC, data is stored in NVMe drives. Custom visualization tools pull data from the NVMe drives and convert it into images.
The first tasks for sensor bring-up were HDL development and breakout board design. HDL could be tested in simulation and with an internal sensor model, prior to receiving physical hardware. Using the CMV12000 data sheet, a model of sensor behavior under specific control patterns was developed, to verify the correct operation of the HDL. This model was checked in simulation and compiled to a binary for the KC705. With this model, a preliminary PCIe interface, that could control the SPI and control lines, and collect streamed data and store it in an NVMe drive, was developed.
Concurrently, while the HDL was in development, a custom breakout board was designed and sent out for manufacturing. The board design had to connect the high-speed outputs from the sensor to the FMC connector, connect the control lines to the sensor inputs from the FMC connector, and convert the power lines from the FMC connector to voltages usable by the sensor.
The first revision of the board was a learning experience. After initially inserting the breakout board in the FMC connector and powering the FPGA development board, one of the power regulators on the breakout board was discovered to be current limited to below current draw at maximum frame rate. The connection to this regulator was removed and the line was powered with an external power supply.
The first tests that were run on the board were programming the SPI register and attempting to capture a single frame. SPI line transactions worked correctly; when checking the frame data, the most significant 2 bits were dropped. Additionally, the high-speed clock could not be locked in a PLL. Hmmm.
Using a high-speed digital scope, the team looked at the clock line and saw Figure 1. This signal cannot be as a clock because one of the cycles is dropped, but this signal explained the missing data bits.
Figure 1: Not a usable clock.
After discovering this, focus shifted to fixing this signal. The first task was to recheck all connections and wiring on the breakout board and to confirm the routing and components were correct. In this check, we discovered that all the power regulators were the wrong packages, and therefore all power supplies were wrong. We removed the connections to these regulators and used external power supplies for all power lines (Figure 2).
We manually initiated the sensor start-up sequence of enabling power supplies, then enabling the clock signals to the sensor, and finally flipping the sensor reset signal. No luck, clock was still bad.
Figure 2: CMV12000 Breakout Board with wires for external power supplies
Next, we tried to fine tune the settings on the sensor. Using product and application guides, we attempted to mirror the manufacturer’s guidelines completely, including adding a physical differential resistor to pins coming from the board (Figure 3). We checked the internal temperature, more conservative usages patterns, and different startup sequences. When none of these methods yielded results, it was decided the earlier bad usage could have broken the sensor, and we should buy a new one.
Figure 3: Adding a 100Ohm Resistor
We carefully checked all connections and added in the safety measures, based of off lessons learned from the damaged caused to the first sensor. Again, SPI transactions were correct. We checked the clock and saw a good clock that can lock in a PLL. We then attempted to capture a single frame, using the sensor supplied test pattern. Examining the raw hex data output showed correct values. We used this data to test the reordering script and generated Figure 4, which looks reasonable with some noise.
Figure 4: Test pattern
Rather than clean up the collection, we were going to push for first light. We decided to block the sensor except for a sliver to see if the different levels of light were visible. We captured the image shown in Figure 5.
Figure 5: First Light
In the current setup, the sensor doesn’t have a lens. We can only see changes in brightness, not actual focused images. The final test was to put a lens in front of the sensor and capture a focused image (Figure 6).
Figure 6: Lens for the sensor
Can we get an image? Yes! Figure 7. Sensor was online. Success!
Figure 7: Captured focused image.
The bring-up of the cmv12000 had several iterations of design, debug, and redesign. Through this effort, we have been able to develop a new hardware integration process that will allow us to more quickly get new hardware online and integrated into our existing systems.
