5.11. Linux DPDK - emllDpdk

The Real-time Ethernet Driver emllDpdk is based on the Data Plane Development Kit (DPDK V23.11). See https://www.dpdk.org for more information. It does not need the atemsys driver and uses Ethernet adapters that are bound to the DPDK interface. The network interface must be bound according to the interface type PCI or SOC.

The parameters to emllDpdk are setup-specific. The function CreateLinkParmsFromCmdLineDpdk() in EcSelectLinkLayer.cpp demonstrates how to initialize emllDpdk.

struct EC_T_LINK_PARMS_DPDK

Public Members

EC_T_LINK_PARMS linkParms

Common link parameters. Signature must be set to EC_LINK_PARMS_SIGNATURE_DPDK.

EC_T_DWORD dwPortId

DPDK port ID

EC_T_WORD wRxBufferCnt

Receive buffer count, 0: defaults to DPDK internal setting

EC_T_WORD wTxBufferCnt

Transmit buffer count, 0: default to DPDK internal setting

EC_T_BOOL bDontCheckLinkStatus

Don’t check link status (forced)

EC_T_BOOL bSetPromiscuousMode

Change promiscuous mode setting

EC_T_BOOL bEalInitDeinitByApp

DPDK EAL layer is initialized and deinitialized by user App

Note

  • Root privileges are required.

  • emllDpdk increases the stack size by DPDK_SEND_BURST_STACK_SIZE.

5.11.1. Linux System Requirements

  • Kernel configuration

In the Fedora OS and other common distributions, such as Ubuntu, or Red Hat Enterprise Linux, the vendor supplied kernel configurations can be used to run most DPDK applications. For other kernel builds, options which should be enabled for DPDK include:

HUGETLBFS
PROC_PAGE_MONITOR support

5.11.2. Huge pages setup

Create huge pages (not persistent)

mkdir -p /dev/hugepages
mountpoint -q /dev/hugepages || mount -t hugetlbfs nodev /dev/hugepages
echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

5.11.3. DPDK for PCI Network Adapter

First, the driver may need to be loaded, as it may be unloaded per default.

sudo modprobe uio_pci_generic

Next the network adapter needs to be bound with dpdk-devbind.py. Therefore, the PCI addresses have to be found out by calling

./usertools/dpdk-devbind.py --status

The output will look like this (here the addresses of the PCI ports are 02:00.0 and 02:00.1, the card is an Intel X710, to be bound to the DPDK i40e driver):

Other Network adapters
======================
0000:02:00.0 'Ethernet Controller X710 for 10GBASE-T 15ff' unused=i40e,vfio-pci
0000:02:00.1 'Ethernet Controller X710 for 10GBASE-T 15ff' unused=i40e,vfio-pci

For each port to be bound to emllDpdk, call

./usertools/dpdk-devbind.py --bind=uio_pci_generic <address>

Example:

./usertools/dpdk-devbind.py --bind=uio_pci_generic 02:00.0

Check https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html for more information on the drivers.

5.11.4. DPDK for DPAA

5.11.4.1. Port specific numbering

The LS1046A Ethernet Port IDs within the Linux device tree (dts) may be non-linear, e.g. 1, 5, 6, 10, whereas the corresponding DPDK Port IDs are linear (0, 1, 2, …).

The following example demonstrates how the DPDK Port ID can be determined from the Ethernet Port ID within the Linux device tree:

DPDK

dts

0

1

1

5

2

6

3

10

5.11.4.2. Reassigning Ports to DPAA and Linux in DTB/DTS

The following changes make the LS1046A ports (i.e. ports 1, 5, 6, 10) available to the emllDpdk, the Linux boot file /boot/fsl-ls1046a-frwy-sdk.dts/dtb.

Note

ethernet@0 (Port 1) is known to be not assignable to emllDpdk for FRWY-LS1046A.

The section dpaa-extended-args must be extended:

chosen {
    stdout-path = "serial0:115200n8";
    name = "chosen";

    dpaa-extended-args {

        fman0-extd-args {
            cell-index = <0>;
            compatible = "fsl,fman-extended-args";
            dma-aid-mode = "port";

            fman0_rx0-extd-args {
                cell-index = <0>;
                compatible = "fsl,fman-port-1g-rx-extended-args";
                vsp-window = <8 0>;
            };

            fman0_tx0-extd-args {
                cell-index = <0>;
                compatible = "fsl,fman-port-1g-tx-extended-args";
            };

            fman0_rx1-extd-args {
                cell-index = <1>;
                compatible = "fsl,fman-port-1g-rx-extended-args";
                vsp-window = <8 0>;
            };

            fman0_tx1-extd-args {
                cell-index = <1>;
                compatible = "fsl,fman-port-1g-tx-extended-args";
            };
        };

        fman1-extd-args {
            cell-index = <1>;
            compatible = "fsl,fman-extended-args";
            dma-aid-mode = "port";
            fman1_rx0-extd-args {
                cell-index = <0>;
                compatible = "fsl,fman-port-1g-rx-extended-args";
                vsp-window = <8 0>;
            };

            fman1_tx0-extd-args {
                cell-index = <0>;
                compatible = "fsl,fman-port-1g-tx-extended-args";
            };

            fman1_rx1-extd-args {
                cell-index = <1>;
                compatible = "fsl,fman-port-1g-rx-extended-args";
                vsp-window = <8 0>;
            };

            fman1_tx1-extd-args {
                cell-index = <1>;
                compatible = "fsl,fman-port-1g-tx-extended-args";
            };
        };
    };
};

Furthermore some or all ports must be assigned to DPDK instead of the native Linux driver in the section fsl,dpaa as follows:

fsl,dpaa {
    compatible = "fsl,ls1046a\\0fsl,dpaa\\0simple-bus";
    dma-coherent;
    phandle = <0x85>;

    /* assign Port 1 (ethernet@0) to emllDpdk (all ports assigned to emllDpdk must be contiguous at the start) */

    ethernet@0 {
        compatible = "fsl,dpa-ethernet-init";
        fsl,fman-mac = <0x34>;
        dma-coherent;

        fsl,bman-buffer-pools = <0x35 0x36 0x37>;
        fsl,qman-frame-queues-rx = <0x50 0x01 0x51 0x01>;
        fsl,qman-frame-queues-tx = <0x70 0x01 0x71 0x01>;
    };

    /* port listed in BSP, but not implemented */

    ethernet@1 {
        compatible = "fsl,dpa-ethernet";
        fsl,fman-mac = <0x38>;
        dma-coherent;
        status = "disabled";
    };

    ...

    /* assign Port 10 (ethernet@9) to the native Linux driver */

    ethernet@9 {
        compatible = "fsl,dpa-ethernet";
        fsl,fman-mac = <0x3e>;
        dma-coherent;
    };
    ...
};

5.11.5. DPDK for ENETC4

DPDK for ENETC4 on the i.MX95 SoC currently requires a specifically customized DPDK version from NXP. Therefore, a specially created version libemllDpdkEnetc4.so exists which is linked against NXP DPDK 22.11.

To use this version of emllDpdk, the EC_T_LINK_PARMS::szDriverIdent of EC_T_LINK_PARMS_DPDK::linkParms must be set to EC_LINK_PARMS_IDENT_DPDK_ENETC4:

EC_T_LINK_PARMS_DPDK* pLinkParmsAdapter = EC_NULL;
/* ... */
OsStrcpy(pLinkParmsAdapter->linkparms.szDriverIdent, EC_LINK_PARMS_IDENT_DPDK_ENETC4);

See also

CreateLinkParmsFromCmdLineDpdk() in EcSelectLinkLayer.cpp

5.11.5.1. Additional requirements

5.11.5.2. Setting up the ENETC4 Ethernet interfaces

The following steps describe the creation of VF (Virtual Function) and the binding of DPDK to it.

  1. Load Kernel modules

    modprobe uio_pci_generic
modprobe kpage_ncache

  2. Setup hugepages

    echo 448 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

  3. Identify the PF(Physical Function) PCI address

    dpdk-devbind.py --status

    The devices are displayed in the output under Network devices using kernel driver; in this case, 0001:00:00.0 is used

    Network devices using kernel driver
===================================
0001:00:00.0 'Device e101' if=end0 drv=fsl_enetc4 unused=
0001:00:08.0 'Device e101' if=end1 drv=fsl_enetc4 unused= *Active*

  4. Configure queues. Set the ring count on the PF to 1 for VF0 and VF1 and reload to apply the new configuration

    devlink dev param set pci/0001:00:00.0 name si_num_rings cmode driverinit value 0101
devlink dev reload pci/0001:00:00.0

  5. Create Virtual Function VF0

    echo 1 > /sys/bus/pci/devices/0001\:00\:00.0/sriov_numvfs

  6. Identify the VF0 PCI address

    dpdk-devbind.py --status

    The newly created devices are displayed in the output under Other Network devices; in this case, 0001:00:02.0

    Other Network devices
=====================
0001:00:02.0 'Device ef00' unused=uio_pci_generic
0001:00:10.0 'Device e101' unused=fsl_enetc4,uio_pci_generic

  7. Bind VF0 to DPDK

    dpdk-devbind.py -b uio_pci_generic 0001:00:02.0

  8. Enable trust for VF0

    ip link set end0 vf 0 trust on

See also

NXP RM00293

5.11.6. Limitations

If the adapter needs to be used multiple times, for cable redundancy or calling emInitMaster() multiple times, the application must set EC_T_LINK_PARMS_DPDK::bEalInitDeinitByApp = EC_TRUE and call rte_eal_init() and rte_eal_cleanup() itself.

To initialize DPDK following call is needed:

rte_eal_init();

To deinitialize DPDK following calls are needed:

rte_eth_dev_close(dwPortId);
rte_eal_cleanup();