How Do I Run SyncE Ethernet 10G and 25G Demo Designs on a Speedster7t AC7t1500 VectorPath Card?

Description

This design demonstrates synchronous Ethernet (SyncE) locking between a received Ethernet stream and the related transmitted Ethernet stream. The design has two Ethernet channels:

• A 10G channel on Ethernet 0
• A 25G channel on Ethernet 1

When SyncE is enabled for either of the channels, the Ethernet receive clock for that channel is used as the reference for the on-board clock generator chip providing the reference clocks for the Ethernet transmission and ensuring that the Ethernet transmit rate is locked to the received rate.

Prerequisites

The design requires that the VectorPath card supports connections from GPIO pins BL14 and BL50 to the clock generator. This is implemented either as board modifications on a rev3 VectorPath card, or natively with a rev4 card. If using a modified rev3 VectorPath card, and updating an existing project, it is also necessary to replace one of the VectorPath GPIO configuration files:

Replace /src/acxip/vp_clkio_nw.acxip with /src/acxip/vp_clkio_nw_no_msio.acxip. The no_msio file is required for a modified board because, without it, the FPGA outputs the clocks to both the MSIO pin and the test header pins, and both are connected to the clock generator causing a signal clash and resulting corruption of the signal. VectorPath rev4 boards do not have the MSIO signals connected, so the original vp_clkio_nw.acxip can be used.

It is also necessary to program the on-board clock generator with a suitable configuration file. This file specifies the Ethernet recovered clock frequencies output from the FPGA to the clock generator. For this design, a suitable file can be found in the /clock_generator directory.

To download the file to the clock generator:

1. With a USB cable connected between the VectorPath card and the host, power on the device.
2. Open a command prompt (i.e., Windows PowerShell) on the host.

3. Execute:

   PS U:\ bw_bmc_clock_programmer program -i USB -c 0 program --clock-name Si5395 <filename>

4. After programming the file, either power cycle the card or execute the following to load the new configuration:

   PS U:\ bw_bmc_clock_programmer reload

5. To confirm the new configuration, execute:

   PS U:\ bw_bmc_clock_programmer display

For full instructions, please see the BittWare VectorPath card User Guide.

Design

The recovered and divided-down receive clocks from each of the channels are output to the on-board clock generator, (Si5396C). The RX clock from the 10G channel is input to IN1, and the RX clock from the 25G channel is input to IN2. The clock generator automatically switches from its on-board oscillator whenever either IN1 or IN2 have a valid clock.

For this design, the recovered clock frequencies are derived as follows:

F_(recovered) = Serdes_rate / (PMA width × 16)

Therefore, the two frequencies are as follows:

F_(10g) = 10.3125G / (32 × 16) = 20.141MHz

F_(25g) = 25.78125G / (32 × 16) = 50.354MHz

Note: The two inputs have a priority order. If the 10G channel recovered clock is present on IN1, the IN2 signal is not used. The recovered clocks are output on both the test[2:1] pins (which connect to the clock generator) and also the GPIO header (to allow for debug). It is not necessary to enable the GPIO outputs, and it may be desirable to turn these off in a production system in order to reduce noise and crosstalk.

Control of the clock outputs is described as follows:

• All clocks are controlled by the synce_control_reg, register 90 at address 0x168.
• Bits [3:0] enable the recovered clocks, {eth_1_tx, eth_0_tx, eth_1_rx, eth_0_rx}. As a minimum, the RX clock for either eth_0 or eth_1 must be enabled. The TX clocks are only for debug purposes.
• Bits [5:4] are output enables for the RX recovered clocks to the clock generator. One of these two must be enabled for SyncE operation. If both are enabled, the clock generator selects the eth_0 (10G) input.
• Bits [11:7] are the output enables for the GPIO header. These are for debug purposes only and show that the clocks are locked.
• All of these signals are described and controlled by variables at the top of /demo/scripts/eth_10g_25g_synce_vp_demo.tcl.

When monitoring the GPIO pins for debug, connect scope probes to the following pins:

• 0, (RX), and 2, (TX), for the 10G channel
• 5, (TX), and 7, (RX), for the 25G channel

Downloading Bitstream

If two VectorPath cards are being used, and board-to-board tests are to be run, it is suggested that two separate ACE windows are opened, (if there are sufficient ACE licenses). This allows easier control of each board separately:

1. Navigate to /demo/scripts.
2. If only a single VectorPath card is present, execute $ set jtag_id [jtag::get_connected_devices]. If more than one card, set jtag_id to the appropriate card.
3. Download the bitstream: $ ac7t1500::program_hex_file ../bitstream/<design_name>.<version>.hex.

Runtime

The design can be controlled using the Tcl scripts, eth_10g_25g_synce_vp_demo.tcl.

Note: If running board-to-board, two QSFP cables are required. These should connect between each of the QSFP connectors on the VectorPath card. The QSFP-56 connector is used for the 10G channel, while the QSFP-DD connector is used for the 25G channel.

Initial setup

1. Set $BOARD_TO_BOARD depending on whether two or one boards are being used.

2. Set the SyncE configuration as required.

   NOTE: When changing between NES loopback and board-to-board, it is necessary to re-download the bitstream.

   IMPORTANT: When NES loopback is selected, SyncE is not operational because the board is not creating a link to another board, but only to itself.

3. Execute:

   $ source eth_10g_25g_synce_vp_demo.tcl

   This confirms the bitstream version, attempts to perform Ethernet link up (either using NES loopback, or board-to-board connections), and then runs 25M packets.

4. To enable and disable SyncE:

   1. To read the synce_control_reg, execute:

      $ ac7t1500::nap_axi_read NAP_SPACE 5 5

   2. Bits [5:4] are the output enables for the recovered clock to the clock generator. If a value of a515 is read, 10G clocks are being used. If bit 4 is cleared, the clock generator reverts to its on-board crystal, losing SyncE lock.

   3. To clear bit 4, execute:

      $ ac7t1500::nap_axi_write NAP_SPACE 5 5 a505

   4. To set bit 4, execute:

      $ ac7t1500::nap_axi_write NAP_SPACE 5 5 a515

   5. The process is similar for bit 5 which controls the 25G recovered clock (a52a and a50a).

5. In addition to the above procedure, the clock generator status can be monitored using the on-board BMC commands.

   To do this, execute:

   $ bw_bmc_clock_programmer -i USB -c 0 display

Setup Diagram

The following figure shows the setup diagram and how communication works between initiator and responder vector path cards in the 10.3125 Gbps case. QSFP cables are connected between the two cards.

Data clocks from the GPIO ports, Tx/Rx Dclk#, are connected to the scope input to display the signal.

Figure: Setup Diagram

Four Channel Divided Down Parallel Data Bus Clock Signals

For the 10.3125G case, the following figure shows that when SyncE is enabled on the responder card, all four channels of the divided-down parallel data bus clock signals are synchronous with no phase shift:

• Ch1 – initiator VectorPath data TX Clock
• Ch2 – initiator VectorPath data RX Clock
• Ch3 – responder VectorPath data TX Clock
• Ch4 – responder VectorPath data RX Clock

If there is no traffic resulting from a disconnect or disabling of the recovered clock, then neither side is synced.

Figure: Four Channel Divided Down Parallel Data Bus Clock Signals