This version (30 Jan 2023 01:59) was approved by Joyce Velasco.The Previously approved version (27 Jan 2023 08:41) is available.
Table of Contents
ADRV9009-ZU11EG Multi-SOM Synchronization
Clock tree synchronization considerations
The HMC7044 used throughout the entire clock-tree in this design supports two alternative synchronizations modes and methods. Both modes may have their own benefits and tradeoffs, such as:
- Jitter
- Correlated Close in Phase Noise
- Timing Requirements
- Phase Synchronization reliability over PVT
- Unwanted Signal Coupling
- Thermal Drift
- Power Consumption
- etc.
We recommend planning for and evaluating both options.
We provide device-trees for both methods.
Reference distribution
A lower frequency reference is used between different levels in the clock tree (Inter-stage Frequency). All clock-chips in the hierarchy require its own local VCXO and this reference is used to lock the VCXO using PLL1 to the external reference. Any of the four available reference inputs CLKINx can be used in this mode.
A additional SYNC signals is used to generate the synchronization event. If the SYNC pin transitions from 0 to 1 with sufficient setup/hold margin with respect to the VCXO, this synchronization event is deterministically carried through PLL2, up the timing chain through the N2 divider, and then to the master SYSREF timer. This mechanism of deterministic phase adjustment allows synchronization of the SYSREF timer and output phases of multiple HMC7044 devices. Please see the datasheet chapter “Multichip Synchronization via PLL2“ For further details.
Clock distribution
The maximum frequency used in the system is generated by the topmost HMC7044 and then distributed throughout the entire clock tree (Inter-stage Frequency).
This method bypasses the PLL1 and PLL2 of all clock chips below the TOP chip. All lower level clock-chips act as clock fanout buffers, where only the clock distribution network output dividers can be used.
This method is referred as clock distribution. All lower level clock-chips receive their input clock via FIN/CLKIN1 and are synced via RFSYNC/CLKIN0.
This mode also allows for TRX baseband rates that would be otherwise not possible with the default installed VCXO of 122.880MHz. Let's say someone needs exactly 250.000MSPS. This becomes possible by providing a 500.000MHz or 1000.000MHz external clock.
Depending on your ADRV2CRR-FMC Carrier Board Hardware revision following stuffing options need to be checked. These are required to route RFSYNC DC coupled to the from SMA connectors J5 RFSYNC_P, J6 RFSYNC_N to the HMC7044 RFSYNC input.
Rev C:
- Replace C18, C19, C236, C240 with 0 Ohm resistors
- Replace C289, C290 with 0 Ohm resistors
- Unload 0 Ohm resistors from location R77, R112 and insert to R110, R111
Rev C.1:
- Replace C289, C290 with 0 Ohm resistors
- Unload 0 Ohm resistors from location R77, R112 and insert to R110, R111
Hardware setup
Required equipment
- EVAL-HMC7044 circuit evaluation board
EVAL-HMC7044 modifications
There is no over-voltage or reverse polarity protection
Connect a 5V 1A power supply to TP17(GND) and TP14(VCC)
The following changes need to be made on the evaluation board:
- Populate J4 and J20 SMA (in case they aren't)
- Replace C28 and C59 with 0 ohm resistors
- Replace R159,160,276,180,181,365 with 50 ohm resistors
SPI connection to the carrier board
The following connections between connectors P25(carrier) and J1(clk) need to be done.
| P25 | J1 | Function |
|---|---|---|
| 2 | 12 | GND |
| 3 | 20 | CS |
| 4 | 16 | MOSI |
| 5 | 18 | MISO |
| 6 | 14 | CLK |
RefCLK and Sync connections to the carrier boards
- CLKOUT5_P and CLKOUT6_P connect to the SYNC SMAs on the carrier
- CLKOUT0 and CLKOUT2 connect to the REFCLK SMAs on the carrier
Only the following outputs work as SYNC in CMOS mode: CLKOUT0,3,5,6,9,10 and 13. The other outputs are 180deg out of phase in CMOS mode and should be used as differential REFCLOCK.
Software
The board which connects the external HMC7044 clockchip is refered as primary and requires the devicetree (system.dtb) from the primary folder in the archive. Likewise the board without the external clockchip connected is called secondary, and requires the devicetree from the secondary folder.
Theory of Operation
There are two domains of synchronization that are considered in this configuration, the ADRV9009 transceivers and the FPGAs. Synchronization for the transceivers is provided by the clocking tree of HMC7044s and the JESD protocol. In the diagram several HMC7044s are cascade from a parent, or what we call external HMC7044, who is responsible for general system reference (sysref) control. These reference signals feed the clock chips on the individual SOMs and FPGAs.
The System Clocking Tree Diagram is located here:
During multi-chip synchronization (MCS), which is a feature of the ADRV9009s, all baseband data from the converters is synchronized across transceiver chips. This requires specific sysrefs to be captured at each of the transceiver simultaneously. This will also create deterministic phase differences between transceivers, when RFPLL sync is enabled, as well. The individual API rights to the transceivers, clock chips, and their sequences are detailed in the rx method of the python class adrv9009_zu11eg_multi.py.
For more information on:
Please visit the JESD204 (FSM) Interface Linux Kernel Framework page
It's important to know, while the devices are not yet fully initialized, they must not be accessed via their regular API, since they are not yet fully initialized.
You need to make sure:
- OSC doesn't start automatically remove it from /home/analog/.config/autostart/ADI IIO Oscilloscope.
- No other script starts automatically which tries to access the adrv9009-phy IIO devices.
- Please also take into consideration that both boards need to have unique HW-MAC addresses, otherwise strange network issues will happen.
Synchronization at the application layer
Synchronization at the application layer across multiple FPGAs is achieved using the external synchronization feature of the transport layer cores and using the SYSREF signal as the external synchronization signal.
Once the JESD links are up the SYSREF pulses are no longer required from the JESD link perspective. However later assertions of the SYSREF pulses must respect the timing of the initial pulses in terms of phase and frequency to match the LMCF/LEMC of the link layer. These later SYSREF pulses can be used as references for simultaneous data capture/drive on multiple FPGAs.
The synchronization mechanism must be orchestrated by software, software must disable the SYSREF generation to FPGAs before arming the external synchronization mechanisms from all the transport layer cores from all FPGAs, it must program all DMA cores to prepare moving data to or from system memory depending on direction, then software must program the clock chips for a single SYSREF pulse that will reach the transport layer cores simultaneously.
Running the software
examples/adrv9009_som_multi.py
michael@mhenneri-D06:~/devel/git/pyadi-iio$ python3 examples/adrv9009_som_multi.py --Connecting to devices Iteration# 0 ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage1> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage1> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage2> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage2> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage3> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage3> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <link_setup> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <link_setup> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage1> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage1> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage2> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage2> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage3> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage3> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage4> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage4> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage5> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage5> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clocks_enable> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clocks_enable> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <link_enable> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <link_enable> with status <0> HMC7044s CAP bank select: [14, 14, 14, 13, 13, 14, 13] JESD Link status: DATA (84a50000.axi-jesd204-rx) JESD Link status: DATA (84a30000.axi-jesd204-tx) JESD Link status: DATA (84a70000.axi-jesd204-rx) JESD Link status: DATA (84a50000.axi-jesd204-rx) JESD Link status: DATA (84a30000.axi-jesd204-tx) JESD Link status: DATA (84a70000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a50000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a30000.axi-jesd204-tx) JESD SYSREF captured: Yes (84a70000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a50000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a30000.axi-jesd204-tx) JESD SYSREF captured: Yes (84a70000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a50000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a30000.axi-jesd204-tx) JESD SYSREF alignment error: No (84a70000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a50000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a30000.axi-jesd204-tx) JESD SYSREF alignment error: No (84a70000.axi-jesd204-rx) JESD 84a50000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a70000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a50000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a70000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes ########### Across Chip (A): 16.733179 Across FMC8 (A): 174.713495 Across Chip (B): 8.009678 Across FMC8 (B): -162.783896 Across SoM (AB): 40.307809 ########### Iteration# 1 ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage1> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage1> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage2> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage2> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage3> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage3> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <link_setup> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <link_setup> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage1> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage1> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage2> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage2> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage3> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage3> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage4> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage4> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage5> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage5> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clocks_enable> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clocks_enable> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <link_enable> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <link_enable> with status <0> HMC7044s CAP bank select: [14, 14, 14, 13, 13, 14, 13] JESD Link status: DATA (84a50000.axi-jesd204-rx) JESD Link status: DATA (84a30000.axi-jesd204-tx) JESD Link status: DATA (84a70000.axi-jesd204-rx) JESD Link status: DATA (84a50000.axi-jesd204-rx) JESD Link status: DATA (84a30000.axi-jesd204-tx) JESD Link status: DATA (84a70000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a50000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a30000.axi-jesd204-tx) JESD SYSREF captured: Yes (84a70000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a50000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a30000.axi-jesd204-tx) JESD SYSREF captured: Yes (84a70000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a50000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a30000.axi-jesd204-tx) JESD SYSREF alignment error: No (84a70000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a50000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a30000.axi-jesd204-tx) JESD SYSREF alignment error: No (84a70000.axi-jesd204-rx) JESD 84a50000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a70000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a50000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a70000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes ########### Across Chip (A): 16.754098 Across FMC8 (A): 174.305087 Across Chip (B): 7.621835 Across FMC8 (B): -163.603793 Across SoM (AB): 40.224082 ########### Iteration# 2 ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage1> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage1> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage2> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage2> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clk_sync_stage3> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clk_sync_stage3> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <link_setup> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <link_setup> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage1> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage1> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage2> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage2> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage3> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage3> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage4> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage4> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <opt_setup_stage5> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <opt_setup_stage5> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <clocks_enable> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <clocks_enable> with status <0> ip:10.44.3.53: DEVICE0: Is <Paused> in state <link_enable> with status <0> ip:10.44.3.39: DEVICE1: Is <Paused> in state <link_enable> with status <0> HMC7044s CAP bank select: [14, 14, 14, 13, 13, 14, 13] JESD Link status: DATA (84a50000.axi-jesd204-rx) JESD Link status: DATA (84a30000.axi-jesd204-tx) JESD Link status: DATA (84a70000.axi-jesd204-rx) JESD Link status: DATA (84a50000.axi-jesd204-rx) JESD Link status: DATA (84a30000.axi-jesd204-tx) JESD Link status: DATA (84a70000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a50000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a30000.axi-jesd204-tx) JESD SYSREF captured: Yes (84a70000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a50000.axi-jesd204-rx) JESD SYSREF captured: Yes (84a30000.axi-jesd204-tx) JESD SYSREF captured: Yes (84a70000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a50000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a30000.axi-jesd204-tx) JESD SYSREF alignment error: No (84a70000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a50000.axi-jesd204-rx) JESD SYSREF alignment error: No (84a30000.axi-jesd204-tx) JESD SYSREF alignment error: No (84a70000.axi-jesd204-rx) JESD 84a50000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a70000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a50000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a70000.axi-jesd204-rx: 0 0 0 0 0 0 0 0 JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a50000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes JESD 84a70000.axi-jesd204-rx: Yes Yes Yes Yes Yes Yes Yes Yes ########### Across Chip (A): 16.590096 Across FMC8 (A): 173.989377 Across Chip (B): 7.867138 Across FMC8 (B): -164.080114 Across SoM (AB): 39.479836 ###########
Troubleshooting
On the primary setup check the status of the external clockchip.
This chip reporting status Unsynchronized is expected, since this chip is only frequency locked against
a 30.720MHz reference clock.
This specifies any shell prompt running on the target
root@analog:~# iio_attr -D hmc7044-ext status
--- PLL1 ---
Status: Locked
Using: CLKIN1 @ 30720000 Hz
PFD: 3840 kHz
--- PLL2 ---
Status: Locked (Unsynchronized)
Frequency: 2949120000 Hz (Autocal cap bank value: 13)
SYSREF Status: Valid & Locked
SYNC Status: Unsynchronized
Lock Status: PLL1 & PLL2 Locked
root@analog:~#
On the primary and secondary setup check the status of all clockchips in the topology.
This specifies any shell prompt running on the target
root@analog:~# iio_attr -D hmc7044-car status --- PLL1 --- Status: Locked Using: CLKIN1 @ 30720000 Hz PFD: 7680 kHz --- PLL2 --- Status: Locked (Synchronized) Frequency: 2949120000 Hz (Autocal cap bank value: 14) SYSREF Status: Valid & Locked SYNC Status: Synchronized Lock Status: PLL1 & PLL2 Locked root@analog:~# iio_attr -D hmc7044 status --- PLL1 --- Status: Locked Using: CLKIN1 @ 30720000 Hz PFD: 30720 kHz --- PLL2 --- Status: Locked (Synchronized) Frequency: 2949120000 Hz (Autocal cap bank value: 14) SYSREF Status: Valid & Locked SYNC Status: Synchronized Lock Status: PLL1 & PLL2 Locked root@analog:~# iio_attr -D hmc7044-fmc status --- PLL1 --- Status: Locked Using: CLKIN1 @ 30720000 Hz PFD: 30720 kHz --- PLL2 --- Status: Locked (Synchronized) Frequency: 2949120000 Hz (Autocal cap bank value: 14) SYSREF Status: Valid & Locked SYNC Status: Synchronized Lock Status: PLL1 & PLL2 Locked root@analog:~#
resources/eval/user-guides/adrv9009-zu11eg/syncronization.txt · Last modified: 30 Jan 2023 01:59 by Joyce Velasco