# Linux DPDK - emllDpdk

The Real-time Ethernet Driver emllDpdk is based on the Data Plane Development Kit (DPDK V23.11). See [https://www.dpdk.org](https://www.dpdk.org) for more information.
It does not need the atemsys driver and uses Ethernet adapters that are bound to the DPDK interface. The network interface must be bound according to the interface type PCI or SOC.

The parameters to emllDpdk are setup-specific. The function `CreateLinkParmsFromCmdLineDpdk()` in [`EcSelectLinkLayer.cpp`](_downloads/bf62c71a4a7e317930818b9da4a9815c/EcSelectLinkLayer.cpp) demonstrates how to initialize emllDpdk.

### struct EC_T_LINK_PARMS_DPDK

### Public Members

### [EC_T_LINK_PARMS](emll_init.html.md#_CPPv415EC_T_LINK_PARMS) linkParms

Common link parameters. Signature must be set to EC_LINK_PARMS_SIGNATURE_DPDK. 

### [EC_T_DWORD](api.html.md#_CPPv410EC_T_DWORD) dwPortId

DPDK port ID 

### [EC_T_WORD](api.html.md#_CPPv49EC_T_WORD) wRxBufferCnt

Receive buffer count, 0: defaults to DPDK internal setting 

### [EC_T_WORD](api.html.md#_CPPv49EC_T_WORD) wTxBufferCnt

Transmit buffer count, 0: default to DPDK internal setting 

### [EC_T_BOOL](api.html.md#_CPPv49EC_T_BOOL) bDontCheckLinkStatus

Don’t check link status (forced) 

### [EC_T_BOOL](api.html.md#_CPPv49EC_T_BOOL) bSetPromiscuousMode

Change promiscuous mode setting 

### [EC_T_BOOL](api.html.md#_CPPv49EC_T_BOOL) bEalInitDeinitByApp

DPDK EAL layer is initialized and deinitialized by user App 

#### NOTE
- Root privileges are required.
- emllDpdk increases the stack size by DPDK_SEND_BURST_STACK_SIZE.

## Linux System Requirements

- Kernel configuration

In the Fedora OS and other common distributions, such as Ubuntu, or Red Hat Enterprise Linux, the vendor supplied kernel configurations can be used to run most DPDK applications.
For other kernel builds, options which should be enabled for DPDK include:

```default
HUGETLBFS
PROC_PAGE_MONITOR support
```

- Required Tools, Libraries and further system requirements can be checked under [https://doc.dpdk.org/guides/linux_gsg/sys_reqs.html](https://doc.dpdk.org/guides/linux_gsg/sys_reqs.html)

## Huge pages setup

Create huge pages (not persistent)

```console
$ mkdir -p /dev/hugepages
$ mountpoint -q /dev/hugepages || mount -t hugetlbfs nodev /dev/hugepages
$ echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
```

## DPDK for PCI Network Adapter

First, the driver may need to be loaded, as it may be unloaded per default.

```console
$ sudo modprobe uio_pci_generic
```

Next the network adapter needs to be bound with `dpdk-devbind.py`. Therefore, the PCI addresses have to be found out by calling

```console
$ ./usertools/dpdk-devbind.py --status
```

The output will look like this (here the addresses of the PCI ports are 02:00.0 and 02:00.1, the card is an Intel X710, to be bound to the DPDK i40e driver):

```default
Other Network adapters
======================
0000:02:00.0 'Ethernet Controller X710 for 10GBASE-T 15ff' unused=i40e,vfio-pci
0000:02:00.1 'Ethernet Controller X710 for 10GBASE-T 15ff' unused=i40e,vfio-pci
```

For each port to be bound to emllDpdk, call

```console
$ ./usertools/dpdk-devbind.py --bind=uio_pci_generic <address>
```

Example:

```console
$ ./usertools/dpdk-devbind.py --bind=uio_pci_generic 02:00.0
```

Check [https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html](https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html) for more information on the drivers.

## DPDK for DPAA

### Port specific numbering

The LS1046A Ethernet Port IDs within the Linux device tree (`dts`) may be non-linear, e.g. 1, 5, 6, 10, whereas the corresponding DPDK Port IDs are linear (0, 1, 2, …).

The following example demonstrates how the DPDK Port ID can be determined from the Ethernet Port ID within the Linux device tree:

|   DPDK |   dts |
|--------|-------|
|      0 |     1 |
|      1 |     5 |
|      2 |     6 |
|      3 |    10 |

### Reassigning Ports to DPAA and Linux in DTB/DTS

The following changes make the LS1046A ports (i.e. ports 1, 5, 6, 10) available to the emllDpdk, the Linux boot file `/boot/fsl-ls1046a-frwy-sdk.dts/dtb`.

#### NOTE
ethernet@0 (Port 1) is known to be not assignable to emllDpdk for FRWY-LS1046A.

The section dpaa-extended-args must be extended:

```dts
chosen {
    stdout-path = "serial0:115200n8";
    name = "chosen";

    dpaa-extended-args {

        fman0-extd-args {
            cell-index = <0>;
            compatible = "fsl,fman-extended-args";
            dma-aid-mode = "port";

            fman0_rx0-extd-args {
                cell-index = <0>;
                compatible = "fsl,fman-port-1g-rx-extended-args";
                vsp-window = <8 0>;
            };

            fman0_tx0-extd-args {
                cell-index = <0>;
                compatible = "fsl,fman-port-1g-tx-extended-args";
            };

            fman0_rx1-extd-args {
                cell-index = <1>;
                compatible = "fsl,fman-port-1g-rx-extended-args";
                vsp-window = <8 0>;
            };

            fman0_tx1-extd-args {
                cell-index = <1>;
                compatible = "fsl,fman-port-1g-tx-extended-args";
            };
        };

        fman1-extd-args {
            cell-index = <1>;
            compatible = "fsl,fman-extended-args";
            dma-aid-mode = "port";
            fman1_rx0-extd-args {
                cell-index = <0>;
                compatible = "fsl,fman-port-1g-rx-extended-args";
                vsp-window = <8 0>;
            };

            fman1_tx0-extd-args {
                cell-index = <0>;
                compatible = "fsl,fman-port-1g-tx-extended-args";
            };

            fman1_rx1-extd-args {
                cell-index = <1>;
                compatible = "fsl,fman-port-1g-rx-extended-args";
                vsp-window = <8 0>;
            };

            fman1_tx1-extd-args {
                cell-index = <1>;
                compatible = "fsl,fman-port-1g-tx-extended-args";
            };
        };
    };
};
```

Furthermore some or all ports must be assigned to DPDK instead of the native Linux driver in the section `fsl,dpaa` as follows:

```dts
fsl,dpaa {
    compatible = "fsl,ls1046a\\0fsl,dpaa\\0simple-bus";
    dma-coherent;
    phandle = <0x85>;

    /* assign Port 1 (ethernet@0) to emllDpdk (all ports assigned to emllDpdk must be contiguous at the start) */

    ethernet@0 {
        compatible = "fsl,dpa-ethernet-init";
        fsl,fman-mac = <0x34>;
        dma-coherent;

        fsl,bman-buffer-pools = <0x35 0x36 0x37>;
        fsl,qman-frame-queues-rx = <0x50 0x01 0x51 0x01>;
        fsl,qman-frame-queues-tx = <0x70 0x01 0x71 0x01>;
    };

    /* port listed in BSP, but not implemented */

    ethernet@1 {
        compatible = "fsl,dpa-ethernet";
        fsl,fman-mac = <0x38>;
        dma-coherent;
        status = "disabled";
    };

    ...

    /* assign Port 10 (ethernet@9) to the native Linux driver */

    ethernet@9 {
        compatible = "fsl,dpa-ethernet";
        fsl,fman-mac = <0x3e>;
        dma-coherent;
    };
    ...
};
```

## DPDK for ENETC4

DPDK for ENETC4 on the i.MX95 SoC currently requires a specifically customized DPDK version from NXP. Therefore, a specially created version `libemllDpdkEnetc4.so` exists which is linked against NXP DPDK 22.11.

To use this version of emllDpdk, the [`EC_T_LINK_PARMS::szDriverIdent`](emll_init.html.md#_CPPv4N15EC_T_LINK_PARMS13szDriverIdentE) of [`EC_T_LINK_PARMS_DPDK::linkParms`](#_CPPv4N20EC_T_LINK_PARMS_DPDK9linkParmsE) must be set to `EC_LINK_PARMS_IDENT_DPDK_ENETC4`:

```cpp
EC_T_LINK_PARMS_DPDK* pLinkParmsAdapter = EC_NULL;
/* ... */
OsStrcpy(pLinkParmsAdapter->linkparms.szDriverIdent, EC_LINK_PARMS_IDENT_DPDK_ENETC4);
```

#### SEE ALSO
`CreateLinkParmsFromCmdLineDpdk()` in [`EcSelectLinkLayer.cpp`](_downloads/bf62c71a4a7e317930818b9da4a9815c/EcSelectLinkLayer.cpp)

### Additional requirements

- Kernel Module `kpage_ncache.ko` [https://github.com/nxp-qoriq/dpdk-extras](https://github.com/nxp-qoriq/dpdk-extras)
- devlink tool from iproute2 [https://github.com/mikedanese/iproute2/tree/master/devlink](https://github.com/mikedanese/iproute2/tree/master/devlink)
- NXP DPDK 22.11 [https://github.com/NXP/dpdk](https://github.com/NXP/dpdk)

### Setting up the ENETC4 Ethernet interfaces

The following steps describe the creation of VF (Virtual Function) and the binding of DPDK to it.

1. Load Kernel modules
   ```console
   $ modprobe uio_pci_generic
   $ modprobe kpage_ncache
   ```
2. Setup hugepages
   ```console
   $ echo 448 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
   ```
3. Identify the PF(Physical Function) PCI address
   ```console
   $ dpdk-devbind.py --status
   ```

   The devices are displayed in the output under *Network devices using kernel driver*; in this case, 0001:00:00.0 is used
   ```default
   Network devices using kernel driver
   ===================================
   0001:00:00.0 'Device e101' if=end0 drv=fsl_enetc4 unused=
   0001:00:08.0 'Device e101' if=end1 drv=fsl_enetc4 unused= *Active*
   ```
4. Configure queues. Set the ring count on the PF to 1 for VF0 and VF1 and reload to apply the new configuration
   ```console
   $ devlink dev param set pci/0001:00:00.0 name si_num_rings cmode driverinit value 0101
   $ devlink dev reload pci/0001:00:00.0
   ```
5. Create Virtual Function VF0
   ```console
   $ echo 1 > /sys/bus/pci/devices/0001\:00\:00.0/sriov_numvfs
   ```
6. Identify the VF0 PCI address
   ```console
   $ dpdk-devbind.py --status
   ```

   The newly created devices are displayed in the output under *Other Network devices*; in this case, 0001:00:02.0
   ```default
   Other Network devices
   =====================
   0001:00:02.0 'Device ef00' unused=uio_pci_generic
   0001:00:10.0 'Device e101' unused=fsl_enetc4,uio_pci_generic
   ```
7. Bind VF0 to DPDK
   ```console
   $ dpdk-devbind.py -b uio_pci_generic 0001:00:02.0
   ```
8. Enable trust for VF0
   ```console
   $ ip link set end0 vf 0 trust on
   ```

#### SEE ALSO
[NXP RM00293](https://docs.nxp.com/bundle/RM00293/page/topics/data_plane_development_kit.html)

## Limitations

If the adapter needs to be used multiple times, for cable redundancy or calling [`esInitSimulator()`](api_general.html.md#_CPPv415esInitSimulator10EC_T_DWORDPK25EC_T_SIMULATOR_INIT_PARMS) multiple times,
the application must set [`EC_T_LINK_PARMS_DPDK::bEalInitDeinitByApp`](#_CPPv4N20EC_T_LINK_PARMS_DPDK19bEalInitDeinitByAppE) = EC_TRUE
and call rte_eal_init() and rte_eal_cleanup() itself.

To initialize DPDK following call is needed:

```default
rte_eal_init();
```

To deinitialize DPDK following calls are needed:

```default
rte_eth_dev_close(dwPortId);
rte_eal_cleanup();
```