This version is outdated by a newer approved version.
This version (15 Jun 2021 12:50) was approved by Michael Hennerich.The Previously approved version (06 May 2021 11:01) is available.
This version (15 Jun 2021 12:50) was approved by Michael Hennerich.The Previously approved version (06 May 2021 11:01) is available.This is an old revision of the document!
Table of Contents
ADRV9009-ZU11EG Multi-SOM Synchronization
….
Hardware setup
Required equipment
- EVAL-HMC7044 circuit evaluation board
EVAL-HMC7044 modifications
There is no over-voltage or reverse polarity protection
Connect a 5V 1A power supply to TP17(GND) and TP14(VCC)
The following changes need to be made on the evaluation board:
- Populate J4 and J20 SMA (in case they aren't)
- Replace C28 and C59 with 0 ohm resistors
- Replace R159,160,276,180,181,365 with 50 ohm resistors
SPI connection to the carrier board
The following connections between connectors P25(carrier) and J1(clk) need to be done.
| P25 | J1 | Function |
|---|---|---|
| 2 | 12 | GND |
| 3 | 20 | CS |
| 4 | 16 | MOSI |
| 5 | 18 | MISO |
| 6 | 14 | CLK |
RefCLK and Sync connections to the carrier boards
- CLKOUT5_P and CLKOUT6_P connect to the SYNC SMAs on the carrier
- CLKOUT0 and CLKOUT2 connect to the REFCLK SMAs on the carrier
Only the following outputs work as SYNC in CMOS mode: CLKOUT0,3,5,6,9,10 and 13. The other outputs are 180deg out of phase in CMOS mode and should be used as differential REFCLOCK.
Software
- 11/09/2020
- 02/04/2021
The board which connects the external HMC7044 clockchip is refered as primary and requires the devicetree (system.dtb) from the primary folder in the archive. Likewise the board without the external clockchip connected is called secondary, and requires the devicetree from the secondary folder.
Theory of Operation
There are two domains of synchronization that are considered in this configuration, the ADRV9009 transceivers and the FPGAs. Synchronization for the transceivers is provided by the clocking tree of HMC7044s and the JESD protocol. In the diagram several HMC7044s are cascade from a parent, or what we call external HMC7044, who is responsible for general system reference (sysref) control. These reference signals feed the clock chips on the individual SOMs and FPGAs.
During multi-chip synchronization (MCS), which is a feature of the ADRV9009s, all baseband data from the converters is synchronized across transceiver chips. This requires specific sysrefs to be captured at each of the transceiver simultaneously. This will also create deterministic phase differences between transceivers, when RFPLL sync is enabled, as well. The individual API rights to the transceivers, clock chips, and their sequences are detailed in the rx method of the python class adrv9009_zu11eg_multi.py.
For more information on:
Please visit the JESD204 (FSM) Interface Linux Kernel Framework page
It's important to know, while the devices are not yet fully initialized, they must not be accessed via their regular API, since they are not yet fully initialized.
You need to make sure:
- OSC doesn't start automatically remove it from /home/analog/.config/autostart/ADI IIO Oscilloscope.
- No other script starts automatically which tries to access the adrv9009-phy IIO devices.
- Please also take into consideration that both boards need to have unique HW-MAC addresses, otherwise strange network issues will happen.
Synchronization at the application layer
Synchronization at the application layer across multiple FPGAs is achieved using the external synchronization feature of the transport layer cores and using the SYSREF signal as the external synchronization signal.
Once the JESD links are up the SYSREF pulses are no longer required from the JESD link perspective. However later assertions of the SYSREF pulses must respect the timing of the initial pulses in terms of phase and frequency to match the LMCF/LEMC of the link layer. These later SYSREF pulses can be used as references for simultaneous data capture/drive on multiple FPGAs.
The synchronization mechanism must be orchestrated by software, software must disable the SYSREF generation to FPGAs before arming the external synchronization mechanisms from all the transport layer cores from all FPGAs, it must program all DMA cores to prepare moving data to or from system memory depending on direction, then software must program the clock chips for a single SYSREF pulse that will reach the transport layer cores simultaneously.
Running the software
examples/adrv9009_som_multi.py
michael@mhenneri-D06:~/devel/git/pyadi-iio$ python3 examples/adrv9009_som_multi.py --Connecting to devices Iteration# 0 ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage1> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage1> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage2> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage2> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage3> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage3> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <link_setup> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <link_setup> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage1> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage1> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage2> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage2> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage3> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage3> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage4> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage4> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage5> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage5> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clocks_enable> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clocks_enable> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <link_enable> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <link_enable> with status <0> HMC7044s CAP bank select: [14, 14, 14, 13, 13, 14, 13] JESD Link status: DATA (84a50000.axi-jesd204-rx) JESD Link status: DATA (84a30000.axi-jesd204-tx) JESD Link status: DATA (84a70000.axi-jesd204-rx) JESD Link status: DATA (84a50000.axi-jesd204-rx) JESD Link status: DATA (84a30000.axi-jesd204-tx) JESD Link status: DATA (84a70000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a50000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a30000.axi-jesd204-tx) JESD SYSREF captured: Yes (84a70000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a50000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a30000.axi-jesd204-tx) JESD SYSREF captured: Yes (84a70000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a50000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a30000.axi-jesd204-tx) JESD SYSREF alignment error: No (84a70000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a50000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a30000.axi-jesd204-tx) JESD SYSREF alignment error: No (84a70000.axi-jesd204-rx) JESD 84a50000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a70000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a50000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a70000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes ########### Across Chip (A): 16.733179 Across FMC8 (A): 174.713495 Across Chip (B): 8.009678 Across FMC8 (B): -162.783896 Across SoM (AB): 40.307809 ########### Iteration# 1 ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage1> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage1> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage2> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage2> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage3> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage3> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <link_setup> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <link_setup> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage1> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage1> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage2> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage2> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage3> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage3> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage4> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage4> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage5> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage5> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clocks_enable> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clocks_enable> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <link_enable> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <link_enable> with status <0> HMC7044s CAP bank select: [14, 14, 14, 13, 13, 14, 13] JESD Link status: DATA (84a50000.axi-jesd204-rx) JESD Link status: DATA (84a30000.axi-jesd204-tx) JESD Link status: DATA (84a70000.axi-jesd204-rx) JESD Link status: DATA (84a50000.axi-jesd204-rx) JESD Link status: DATA (84a30000.axi-jesd204-tx) JESD Link status: DATA (84a70000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a50000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a30000.axi-jesd204-tx) JESD SYSREF captured: Yes (84a70000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a50000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a30000.axi-jesd204-tx) JESD SYSREF captured: Yes (84a70000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a50000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a30000.axi-jesd204-tx) JESD SYSREF alignment error: No (84a70000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a50000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a30000.axi-jesd204-tx) JESD SYSREF alignment error: No (84a70000.axi-jesd204-rx) JESD 84a50000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a70000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a50000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a70000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes ########### Across Chip (A): 16.754098 Across FMC8 (A): 174.305087 Across Chip (B): 7.621835 Across FMC8 (B): -163.603793 Across SoM (AB): 40.224082 ########### Iteration# 2 ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage1> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage1> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage2> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage2> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage3> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage3> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <link_setup> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <link_setup> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage1> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage1> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage2> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage2> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage3> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage3> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage4> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage4> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage5> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage5> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clocks_enable> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clocks_enable> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <link_enable> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <link_enable> with status <0> HMC7044s CAP bank select: [14, 14, 14, 13, 13, 14, 13] JESD Link status: DATA (84a50000.axi-jesd204-rx) JESD Link status: DATA (84a30000.axi-jesd204-tx) JESD Link status: DATA (84a70000.axi-jesd204-rx) JESD Link status: DATA (84a50000.axi-jesd204-rx) JESD Link status: DATA (84a30000.axi-jesd204-tx) JESD Link status: DATA (84a70000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a50000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a30000.axi-jesd204-tx) JESD SYSREF captured: Yes (84a70000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a50000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a30000.axi-jesd204-tx) JESD SYSREF captured: Yes (84a70000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a50000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a30000.axi-jesd204-tx) JESD SYSREF alignment error: No (84a70000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a50000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a30000.axi-jesd204-tx) JESD SYSREF alignment error: No (84a70000.axi-jesd204-rx) JESD 84a50000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a70000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a50000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a70000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes ########### Across Chip (A): 16.590096 Across FMC8 (A): 173.989377 Across Chip (B): 7.867138 Across FMC8 (B): -164.080114 Across SoM (AB): 39.479836 ###########
Troubleshooting
On the primary setup check the status of the external clockchip.
This chip reporting status Unsynchronized is expected, since this chip is only frequency locked against
a 30.720MHz reference clock.
This specifies any shell prompt running on the target
root@analog:~# iio_attr -D hmc7044-ext status
--- PLL1 ---
Status: Locked
Using: CLKIN1 @ 30720000 Hz
PFD: 3840 kHz
--- PLL2 ---
Status: Locked (Unsynchronized)
Frequency: 2949120000 Hz (Autocal cap bank value: 13)
SYSREF Status: Valid & Locked
SYNC Status: Unsynchronized
Lock Status: PLL1 & PLL2 Locked
root@analog:~#
On the primary and secondary setup check the status of all clockchips in the topology.
This specifies any shell prompt running on the target
root@analog:~# iio_attr -D hmc7044-car status --- PLL1 --- Status: Locked Using: CLKIN1 @ 30720000 Hz PFD: 7680 kHz --- PLL2 --- Status: Locked (Synchronized) Frequency: 2949120000 Hz (Autocal cap bank value: 14) SYSREF Status: Valid & Locked SYNC Status: Synchronized Lock Status: PLL1 & PLL2 Locked root@analog:~# iio_attr -D hmc7044 status --- PLL1 --- Status: Locked Using: CLKIN1 @ 30720000 Hz PFD: 30720 kHz --- PLL2 --- Status: Locked (Synchronized) Frequency: 2949120000 Hz (Autocal cap bank value: 14) SYSREF Status: Valid & Locked SYNC Status: Synchronized Lock Status: PLL1 & PLL2 Locked root@analog:~# iio_attr -D hmc7044-fmc status --- PLL1 --- Status: Locked Using: CLKIN1 @ 30720000 Hz PFD: 30720 kHz --- PLL2 --- Status: Locked (Synchronized) Frequency: 2949120000 Hz (Autocal cap bank value: 14) SYSREF Status: Valid & Locked SYNC Status: Synchronized Lock Status: PLL1 & PLL2 Locked root@analog:~#
resources/eval/user-guides/adrv9009-zu11eg/syncronization.1622123573.txt.gz · Last modified: 27 May 2021 15:52 by Laszlo Nagy