Articles in this section

See more

What is the DBI Gateway and How Can I Use It?

  • Updated
Follow

Description

What is the DBI gateway and how can I use it in my design?

Answer

The PCIe controller's data bus interface (DBI) provides access to the PCIe control and status registers (CSRs). These registers are commonly accessed by host-side application software to configure and use various PCIe features such as the address translation unit (ATU) and direct memory access (DMA) controller.

The native DBI is not designed to handle multiple asynchronous requests. If multiple sources make simultaneous requests to the DBI then these requests could overlap, causing collisions and resulting in undefined behavior. Achronix provides the DBI gateway as a soft logic solution that synchronizes incoming requests to the DBI. The DBI gateway manages access to the native DBI such that multiple sources can independently access the PCIe registers without risk of a collision. The DBI gateway is implemented as part of the Speedster7t Device Manager, so no additional macros need be instantiated.

Due to its lack of asynchronous request support, Achronix does not recommend using the native DBI. However, there are certain applications where usage of the native DBI is required. One such example is bitstream programming via PCIe. In cases where the native DBI must be used, special care must be taken to guarantee that DBI requests do not overlap. For more information on using the native DBI see the section "Bitstream Programming via PCIe" in Speedster7t Configuration User Guide (UG094).

Configuration

The DBI gateway can be enabled in the Achronix Device Manager (ADM) by selecting the appropriate options in section,   "The PCI Express IP Configuration GUI interface in ACE" in the Speedster7t PCIe User Guide (UG098), and then "Generating I/O Ring Design Files" section in ACE help for the design. As of ACE 10.2, the DBI gateway can be enabled for PCIE_1 or PCIE_0 by selecting the Enable PCIE_1 Support option or the Enable PCIE_0 Support option respectively. The DBI gateway can be enabled for both PCIE_0 and PCIE_1 at the same time as well.

Note
In ACE releases previous to ACE 10.2, to enable the DBI gateway in the Device Manager, select the Enable PCIE_0 DBI Gateway option or E nable PCIE_1 DBI Gateway option in the ACE Speedster7t Device Manager Configuration Editor. Enable PCIE_0 DBI Gateway option must be selected for PCIE_0 (PCIe ×8) DBI access, while Enable PCIE_1 DBI Gateway option must be selected for PCIE_1 (PCIe ×16) DBI access.

ACE_10p2_ADM_config.PNG

Device Manager with DBI Gateway Enabled for PCIE_1

Addressing

The sections below describes the addressing scheme for the DBI gateway. All accesses should use 4-byte aligned addresses.

DBI Gateway NoC Address Space

The DBI gateway is addressable using the NoC address of the NAP that the Device Manager is placed at. The NoC address of a NAP can be determined according to table shown below. 

NAP Address Translation

Address Bit 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 … 0
NAP 0 0 0 1 0 0 0 NAP Column NAP Row Memory Address

Where:

  • Addr[41:35] = 7'b0001000 – Accesses any NAP endpoint in the device.
  • Addr[34:31] = NAP Column – For AC7t1500, valid values for this field are 0 to 9, west to east. Columns on the 2D NoC are numbered 1 to 10, west to east. In order to save bits in the address, the number for a column "N" becomes "N-1" for this value (column 3 uses the value of 2).

  • Addr[30:28] = NAP Row – For AC7t1500, valid values for this field are 0 to 3, south to north. Rows on the 2D NoC are numbered 1 to 8, south to north. In order to save bits in the address, the number for a row "N" becomes "N-1" for this value (row 5 uses the value of 4).

    Note
    Although there are 8 rows in the AC7t1500, only the southern 4 rows are valid for use by the Speedster7t Device Manager.

  • Addr[27:0] = Memory Address – Passed to the FPGA fabric logic. For details on the DBI gateway memory addresses, see Memory Map below.

    Note
    The row and column fields in the address for AXI transactions start numbering from 0; whereas, placement constraints in ACE use the actual row and column numbers, starting from 1.

To convert from a NAP location to a 2D NoC address, on the Speedster7t AC7t1500 FPGA, use the following equation:

Address = 0x40_0000_0000 + ((COLUMN – 1) << 31) + ((ROW–1) << 28)

DBI Gateway Memory Map

Memory address bits[27:0] of the DBI gateway's NAP address space are mapped into sections according to the table below.

DBI Gateway Address Mapping

DBI Gateway Address Space [27:23] 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
PCIe configuration header space and capability structures 0x0 PCIE_1=0, PCIE_0=1(1) 0 0 PF[1:0](2) 0x0 register offset(3)
MSI-X vector table 0x0 PCIE_1=0, PCIE_0=1(1) 0 0 PF[1:0](2) 0x0 0xD register offset
MSI-X pending bit array (PBA) 0x0 PCIE_1=0, PCIE_0=1(1) 0 0 PF[1:0](2) 0x0 0xE register offset
ATU registers 0x0 PCIE_1=0, PCIE_0=1(1) 1 1 0 0 0 0 register offset within the ATU space
DMA register 0x0 PCIE_1=0, PCIE_0=1(1) 1 1 0 0 0 1 register offset within the DMA space

Table Notes

  1. Determines the PCIE target. Value: 1'b0 selects PCIEx16 and value: 1'b1 selects PCIE ×8.
  2. Physical function number
  3. Register space for PCIe configuration header and capability structure is within 0x0 to 0x700

Accessing PCIe Registers

The PCIe control and status registers are accessible by issuing AXI read or write transactions to the address space of the DBI gateway's NAP. Read and write transactions to the DBI gateway can be initiated from PCIe, JTAG, and AXI initiator logic connected to a NAP. Accessing the DBI gateway from PCIe requires a PCIe BAR that is configured to map to the NAP address space being used by the DBI gateway.

For more information on configuring the PCIe IP to access the DBI gateway see PCIe Bar Mapping and Software Usage section below, and the knowledge base article VectorPath PCIe Design Configuration.

Read Transaction

A read from the PCIe registers can be completed by issuing an AXI read transaction to the DBI gateway address space. Supported AXI read transactions must have a burst size of 32 bits and a burst length of 1. The read transaction completes with an AXI read response that is received by the AXI initiator.

Examples

Consider a design with the DBI gateway enabled in the Device Manager and placed on the NAP located at row 4, column 6. The DBI gateway address space will use the NAP address space for row 4, column 6 which, according to the NAP address translation table, is 0x042b0000000. The PCIe configuration header space for the PCIE ×16 (PCIE_1) interface begins at address 0x0 of the DBI gateway address space indicated in the DBI Gateway Address Mapping table. Therefore, a read to the vendor ID and device ID register (offset 0x0 from the base address of the PCIe configuration header space) will be at NoC address 0x042b0000000.

The example below shows a 32-bit read to the vendor ID and device ID register using a Tcl read command to the DBI gateway.

ac7t1500:noc_read 0x042b0000000
00101b59]]

The example below shows a 32-bit read from the vendor ID and device ID register using the Achronix SDK's acx_pcie_peek_poke tool. The DBI gateway is mapped to BAR3 as described in PCIe BAR Mapping and Software Usage below.

Note
The acx_pcie_peek_poke tool can be found in the /tools/acx_pcie_peek_poke/ directory of the latest Achronix SDK release.

32-bit Read to DBI Gateway via PCIe

./acx_pcie_peek_poke read bar3 0x0 
0x00101b59

Error Response

A read transaction that has failed in the DBI gateway will return a SLVERR response on the write response channel as defined by the AXI protocol. That SLVERR will result in a runtime error if the transaction was initiated using PCIe or JTAG.

Write Transaction

A write to the PCIe registers can be completed by issuing an AXI write transaction to the DBI gateway address space. Supported AXI write transactions must have a burst size of 32 bits and a burst length of 1. The write transaction is complete once the AXI write response has been received by the AXI initiator.

Examples

Consider a design with the DBI gateway enabled in the Device Manager and placed on the NAP located at row 4, column 6. The DBI gateway address space will use the NAP address space for row 4, column 6 which, according to the NAP Address Translation table, is 0x042b0000000. The PCIe DMA register space for the PCIE ×16 (PCIE_1) interface begins at address 0x0310000 of the DBI gateway address space indicated by the DBI Gateway Address Mapping table. Therefore, a write to the DMA read engine enable register (offset 0x2c from the base address of the PCIe DMA register space) will be at NoC address 0x042b031002c .

The example below shows a 32-bit write to the DMA read engine enable register using a Tcl write command to the DBI gateway.

ac7t1500:noc_read 0x042b031002c
00000000
cmd ac7t1500:noc_write 0x042b031002c 00000001
cmd ac7t1500:noc_read 0x042b031002c
00000001

In the next example, consider the DBI gateway mapped to BAR3 using the compressed mapping as described in PCIe BAR Mapping and Software Usage below. ATU region 1 maps BAR3 offset 0x10000 to NoC address 0x042b0300000. If the target NoC address is 0x042b031002c, then the respective BAR3 offset would be 0x2002c (0x042b031002c − 0x042b0300000 + 0x10000).

The example uses the Achronix SDK's acx_pcie_peek_poke tool to perform a 32-bit write to the DMA read engine enable register using the DBI gateway.

32-bit Write to DBI Gateway via PCIe

./acx_pcie_peek_poke write bar3 0x2002c 0x00000001

Error Response

A write transaction that has failed in the DBI gateway will return a SLVERR response on the write response channel as defined by the AXI protocol. The SLVERR response will result in a runtime error if the transaction was initiated using PCIe or JTAG.

PCIe BAR Mapping

For a host system to access the DBI gateway over the PCIe link, the DBI gateway address space must be mapped to a PCIe BAR. The Achronix SDK uses BAR3 as the default BAR for access to the DBI gateway, although any BAR could be used. The full DBI gateway address space can be mapped into a single PCIe BAR, or the ATU's address match mode (AMM) can be used to map smaller subsets of the address space. By default the Achronix SDK supports two DBI gateway mapping modes: compressed and full. The BAR mapping configurations are shown in the sections below using screenshots from the ACE 10.2 GUI.

Compressed Mapping

In compressed mapping mode only a subset of the DBI gateway address space for a single physical function is exposed. Included are PCIe configuration header space and capability structures for a single physical function, the ATU registers, and the DMA registers. The compressed mapping is recommended because it allows for a much smaller BAR to be allocated than the full mapping (only 192K vs. 4M for one physical function.)

The compressed DBI gateway mapping utilizes that ATU's address match mode with two regions:

  • A 64 KB region 0 with base address at the target physical function's PCIe configuration header offset in the DBI gateway's address space
  • A 128 KB region 1 with base address at the ATU register space offset in the DBI gateway address space

Using these two ATU regions will result in the address table from the PCIe perspective shown below.

DBI Gateway Compressed Address Mapping from PCIe Perspective

DBI Gateway Address Space 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
PCIe Configuration Header Space and Capability Structures 0 0 0x0 Register Offset(†)
ATU Registers 0 1 Register Offset within the ATU Space
DMA Register 1 0 Register Offset within the DMA Space

Table Note
† Register space for PCIe configuration header and capability structure is within 0x0 to 0x700

Examples

Each PCIe physical function has its own set of BARs. Therefore, the BARs and ATU must be configured for each physical function that accesses the DBI Gateway. Two examples are given in this section. The first example is for a simple device that only has a single physical function. The second is a more complex example for a device that uses all four available physical functions.

Note
If a physical function is not used, then the compressed DBI gateway mapping for that physical function is not required to be mapped to a BAR. Additionally, all physical functions can have BARs that map and access the PCIe configuration header space and capability structures for any physical function. Therefore, the compressed DBI gateway mapping for physical function 0 (or any other physical function) can be mapped in a BAR for physical function 0, 1, 2, or 3 as well.

Example Using Physical Function 0

In this example, the ADM is set to column 6, row 4 and the target PCIe interface is PCIE_1. The corresponding NAP address is 0x42_b000_0000. The ATU regions will combine the base NAP address and an offset that represents the desired DBI gateway address space. The NAP address space is used by the DBI gateway as explained in the section. An example configuring BAR3 to use the compressed DBI gateway mapping for physical function 0 is shown below.

In the IORing Designer configuration for PCIe physical function 0, under BAR3, set the "Address Mode" to Address Match Mode in order to use multiple ATU regions. Set the number of ATU Regions to 2.

BAR3_address_match_mode.PNG

BAR3 Setting in Physical Function 0 Memory Map Page

On the BAR3 ATU region page for physical function 0, configure the two ATU regions with the addresses used for the compressed DBI gateway mapping as follows. Bits [19:18] of PF0 BAR3 Region 0 are set to "00" to map the region to the beginning of the PCIe configuration header space and capability structures of physical function 0. Bits [21:20] of PF0 BAR3 Region 1 are set to "11" to skip over infrequently used registers and map the region to the start of the ATU and DMA registers (which are shared by all of the physical functions).

PF0_PCIe_BAR_3_ATU_config.PNG

DBI Gateway Compressed Mapping for Physical Function 0 on BAR3

Example Using All Physical Functions

In this example, the ADM is set to column 6, row 4 and the target PCIe interface is PCIE_1. The corresponding NAP address is 0x42_b000_0000. The NAP address space is used by the DBI gateway as explained in the Addressing section. An example using BAR3 for the compressed DBI gateway mapping of each physical function is shown below.

Physical Function 0

In the IORing Designer configuration for PCIe physical function 0, under BAR3, set the "Address Mode" to Address Match Mode in order to use multiple ATU regions. Set the number of ATU Regions to 2.

BAR3_address_match_mode.PNG

BAR3 Setting in Physical Function 0 Memory Map Page

On the BAR3 ATU Region page for physical function 0, configure the two ATU regions with the addresses used for the compressed DBI gateway mapping as follows. Bits [19:18] of PF0 BAR3 Region 0 are set to "00" to map the region to the beginning of the PCIe configuration header space and capability structures of physical function 0. Bits [21:20] of PF0 BAR3 Region 1 are set to "11" to skip over infrequently used registers and map the region to the start of the ATU and DMA registers (which are shared by all of the physical functions).

PF0_PCIe_BAR_3_ATU_config.PNG

DBI gateway Compressed Mapping for Physical Function 0 on BAR3

Physical Function 1

In the IORing Designer configuration for PCIe physical function 1, under BAR3, set the "Address Mode" to Address Match Mode in order to use multiple ATU regions.

BAR3_address_match_mode_PF1.PNG

BAR3 Setting in Physical Function 1 Memory Map Page

On the BAR3 ATU Region page for physical function 1, configure the two ATU regions with the addresses used for the compressed DBI gateway mapping as follows. Bits [19:18] of PF0 BAR3 Region 0 are set to "01" to map the region to the beginning of the PCIe configuration header space and capability structures of physical function 1. Bits [21:20] of PF0 BAR3 Region 1 are set to "11" to skip over infrequently used registers and map the region to the start of the ATU and DMA registers (which are shared by all of the physical functions).

PF1_PCIe_BAR_3_ATU_config.PNG

DBI Gateway Compressed Mapping for Physical Function 1 on BAR3

Physical Function 2

In the IORing Designer configuration for PCIe physical function 2, under BAR3, set the "Address Mode" to Address Match Mode in order to use multiple ATU regions.

BAR3_address_match_mode_PF2.PNG

BAR3 Setting in Physical Function 2 Memory Map page

On the BAR3 ATU Region page for physical function 2, configure the two ATU regions with the addresses used for the compressed DBI gateway mapping as follows. Bits [19:18] of PF0 BAR3 Region 0 are set to "10" to map the region to the beginning of the PCIe configuration header space and capability structures of physical function 2. Bits [21:20] of PF0 BAR3 Region 1 are set to "11" to skip over infrequently used registers and map the region to the start of the ATU and DMA registers (which are shared by all of the physical functions).

PF2_PCIe_BAR_3_ATU_config.PNG

DBI Gateway Compressed Mapping for Physical Function 2 on BAR3

Physical Function 3

In the IORing Designer configuration for PCIe physical function 0, under BAR3, set the "Address Mode" to Address Match Mode in order to use multiple ATU regions.

BAR3_address_match_mode_PF3.PNG

BAR3 Setting in Physical Function 3 Memory Map Page

On the BAR3 ATU Region page for physical function 3, configure the two ATU regions with the addresses used for the compressed DBI gateway mapping as follows. Bits [19:18] of PF0 BAR3 Region 0 are set to "11" to map the region to the beginning of the PCIe configuration header space and capability structures of physical function 3. Bits [21:20] of PF0 BAR3 Region 1 are set to "11" to skip over infrequently used registers and map the region to the start of the ATU and DMA registers (which are shared by all of the physical functions).

PF3_PCIe_BAR_3_ATU_config.PNG

DBI Gateway Compressed Mapping for Physical Function 3 on BAR3

Full Mapping

The full DBI gateway mapping refers to the mapping of the whole DBI gateway address space for an individual PCIe IP to a single BAR. With this mapping all the registers for each physical function can be accessed. Mapping this address space will require a BAR size of at least 4 MB. The full mapping exposes PCIe configuration header space and capability structures for all physical and virtual functions in the PCIe controller. As a result, care must be taken to ensure that the correct PCIe function is accessed when the full mapping is in use.

Note
Due to the increased BAR memory requirements and potential for erroneous register space access Achronix recommends using the compressed mapping over the full mapping whenever possible.

Example

In this example, the ADM is set to column 6, row 4. The corresponding NAP address is 0x42_b000_0000. The NAP address space is used by the DBI gateway as explained in the Addressing section above. An example configuring BAR3 to use the full DBI gateway mapping for both PCI_1 and PCI_0 is shown below. BAR Match Mode in the ATU can be used since only a single continuous region is required.

BAR3_full_dbi_gateway_mapping_pcie1.PNG

Full DBI Gateway Mapping for PCIE_1

BAR3_full_dbi_gateway_mapping_pcie0.PNG

Full DBI Gateway Mapping for PCIE_0

Accessing The Gateway From The Achronix SDK

The Achronix SDK provides a set of software abstractions and an API to access the DBI gateway using the native DBI interface, as well as both the compressed and full gateway mapping modes. By default the Achronix SDK expects the DBI gateway to be mapped in the compressed mode at the start of BAR 3.

Defining The Default DBI Gateway Mapping Globally

The default DBI mapping behavior is defined within the SDK shared library at compile time by the Achronix_IP_config.c file in the src/Achronix_IP/ directory of the Achronix SDK. Within this file is a lookup table called ACX_G_IP_LOCATIONS which defines how Achronix provided IP is laid out in the PCIe BAR spaces. An example of the ACX_G_IP_LOCATIONS table from the Achronix SDK version 2.1.1 is shown bellow:

ACX_G_IP_LOCATIONS

const ACX_IP_location ACX_G_IP_LOCATIONS[ACX_IP_STATIC_SIZE] = 
{                              // A bar index of -1 indicates that the IP is not mapped, limit is exclusive
                               //{  bar_index, bar_offset, bar_limit, tag }
    [ACX_IP_DBI_X16_INTERFACE] = { -1,         0,          0x30,      "DBI_X16_INTERFACE"},
    [ACX_IP_DBI_GATEWAY_FULL]  = { -1,         0,          0x400000,  "DBI_GATEWAY_FULL"}, // Full 4MB gateway mapping
    [ACX_IP_DBI_GATEWAY_COMP]  = {  3,         0,          0x30000,   "DBI_GATEWAY_COMP"}, // Compressed 192K gateway mapping
};

This table can be modified to change the default location of the DBI gateway for designs that do not map the compressed gateway to the start of BAR 3. The first column of the table indicates the BAR index of the IP. A value of −1 indicates that the IP is not present. If both the compressed and full mappings are provided the SDK will use the full. The ACX_DBI_X16_INTERFACE is the native DBI interface in the PCIE_1 IP which support up to 16 lanes.

Modifying The DBI Gateway Mapping At Runtime

The Achronix SDK also provides a set of functions to modify the mapping of the DBI interface at runtime. The following functions are supported as of Achronix SDK version v2.1.1.

DBI Remapping Functions

ACX_SDK_STATUS acx_dbi_x16_overide_location(ACX_DEV_PCIe_device *device, int bar_index, uint32_t bar_offset);
ACX_SDK_STATUS acx_dbi_gateway_overide_location(ACX_DEV_PCIe_device *device, int compressed, int bar_index, uint32_t bar_offset);

An example snippet showing how to change the DBI gateway at runtime is shown bellow:

ACX_DEV_PCIe_device *device = acx_dev_init_pcie_device_idx(args.device_idx);
if (!device || device-status != ACX_SDK_STATUS_OK) {
    ACX_PRINT_ERROR("failed to open device\n");
    exit(EXIT_FAILURE);
}

//Move the compressed gateway to BAR 4 offset 0x3000

if (acx_dbi_gateway_overide_location(device, 1, 4, 0x3000) != ACX_SDK_STATUS_OK){
    ACX_PRINT_ERROR("failed to override gateway bar\n");
    exit(EXIT_FAILURE);
}

//Note the override function does not set the interface block in use. 
//To which interface is used the block must be set manually as shown bellow:
device-dbi_interface_block = &(device-ip_blocks[ACX_IP_DBI_GATEWAY_COMP]);
Was this article helpful?
0 out of 0 found this helpful
Return to top

Related articles

Powered by Zendesk