4.4. Linux

4.4.1. OS optimizations

Linux itself is not real-time capable, so it is recommended to use it with the additional PREEMPT_RT patch.

The power management can disrupt cyclical processing, it is advisable to disable the CPUIDLE sub-system and CPUFREQ sub-system. The sub-systems can be disabled by changing the kernel command line parameters in the boot loader. On x86, x86_64 systems this is usually grub, on embedded devices with ARM, ARM64 is usually u-boot. It is also possible to build a custom kernel without these sub-systems.

Running a EC-Master application on a dedicated CPU core that is isolated from the Linux scheduler (ISOLCPUS) can provide additional stability.

4.4.1.1. CPUIDLE sub-system

Check if CPUFREQ sub-system is enabled:
ls /sys/devices/system/cpu/

If cpuidle appears in the list, it is enabled.

Disable CPUIDLE via the kernel command-line in grub:
linux  /boot/vmlinuz-4.19.0-16-rt-amd64 cpuidle.off=1

4.4.1.2. CPUFREQ sub-system

Check if CPUFREQ sub-system is enabled:
ls /sys/devices/system/cpu/

If cpufreq appears in the list, it is enabled.

Disable CPUFREQ sub-system via the kernel command-line grub:
linux  /boot/vmlinuz-4.19.0-16-rt-amd64 cpufreq.off=1

If CPUFREQ is not to be deactivated, the governor should be set to performance.

The currently active governor can be determined as follows:
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
The available governors with:
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_available_governors
To change governor use:
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

4.4.1.3. ISOLCPUS

Isolate CPU core number 4 of a quad-core processor via the kernel command-line grub:
linux  /boot/vmlinuz-4.19.0-16-rt-amd64 isolcpus=3
Running EcMasterDemo on the isolated CPU core by setting the CPU affinity -a:
./EcMasterDemo -a 3

4.4.2. atemsys kernel module

To use Real-time Ethernet Driver under Linux, the atemsys kernel module must be compiled and loaded. atemsys grants direct access to hardware to improve the performance.

All necessary scripts, source code and a detailed description of the installation can be found on https://github.com/acontis/atemsys. A ready-to-use Yocto recipe is also available on https://github.com/acontis/meta-acontis

4.4.2.1. atemsys as Device Tree Ethernet Driver

atemsys can also be used as a device tree driver to avoid certain conflicts between the Real-time Ethernet Driver and the Linux kernel, e.g. power management, shared MDIO bus, etc..

A detailed guide on how to customize the device tree accordingly can also be found on https://github.com/acontis/atemsys. Example device tree modifications for different Real-time Ethernet Drivers/SoC can be found in https://github.com/acontis/atemsys/wiki.

Note

This is the preferred solution on all embedded devices with device tree support.

4.4.2.2. atemsys and PHY OS Driver

To use the PHY OS Driver, the acontis kernel module atemsys has to be included in the kernel device tree as an official driver for the Ethernet controller and doesn’t required any additional configuration at the application level. As a result atemsys can interact with Linux drivers.

4.4.3. Unbind Ethernet Driver instance

Ethernet Driver instances used by Real-time Ethernet Drivers may not be bound by kernel drivers modules! Unbind can be done by unloading the kernel driver module, via the unbind interface of the driver or by modifying the device tree.

4.4.3.1. Unbind from kernel driver

The following command unbinds an instance without unloading the kernel driver module:

PCI

echo "<Instance-ID>" > /sys/bus/pci/drivers/<driver-name>/unbind

Example:

echo "0000:00:19.0" > /sys/bus/pci/drivers/e1000e/unbind

This call requires the PCI bus, device, function codes (in the above example it is 0000:00:19.0). The codes can be found using Linux commands like, for example:

ls /sys/bus/pci/drivers/e1000e

SoC

echo "<Instance-ID>" > /sys/bus/platform/drivers/<driver-name>/unbind

Example:

echo "2188000.ethernet" > /sys/bus/platform/drivers/fec/unbind

4.4.3.2. Unload kernel driver

Not all drivers allow unbinding of network adapters. If unbinding is not supported the corresponding Linux kernel driver must not be loaded.

The following command lists the loaded kernel modules that may conflict with Real-time Ethernet Driver:

lsmod | egrep "<module-name>"

Example:

lsmod | egrep "e1000|e1000e|igb"

PCI/PCIe: The command lspci -v shows which driver is assigned to which network card, e.g.:

lspci -v

...
11:0a.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)
...
Kernel driver in use: e1000e

Modules can be prevented from loading with the following commands:

echo blacklist <module-name> | sudo tee -a /etc/modprobe.d/blacklist.conf
update-initramfs -k all -u
sudo reboot

The following table shows the Kernel modules related to the Real-time Ethernet Driver:

Chip

Real-time Ethernet Driver

Kernel driver(s)

Remarks

Broadcom Genet

emllBcmGenet

genet

Unbind not supported

Beckhoff CCAT

emllCCAT

ec_bhf

CPSW

emllCPSW

ti_cpsw

Generic

emllDpdk

DesignWare 3504

emllDW3504

stmmac

emllEG20T

Freescale TSEC/eTSEC v1/2

emllETSEC

gianfar_driver

Freescale FEC and ENET controller

emllFslFec

fec, fec_ptp

Xilinx Zynq-7000/Ultrascale

emllGEM

Intel Pro/1000

emllI8254x

igb, e1000, e1000e

Intel Pro/1000

emllIntelGbe

igb, e1000, e1000e

Intel Pro/100

emllI8255x

e100

ICSS

emllICSS

prueth,pruss

Unbind not supported

RDC R6040

emllR6040

Realtek RTL8139

emllRTL8139

8139too, 8139cp

Realtek RTL8169 / RTL8111 / RTL8168

emllRTL8169

r8169

Unbind not supported

SuperH

emllSHEth

sh_eth

Unbind not supported

Generic

emllSockRaw

4.4.4. Docker

It is possible to operate EC-Master within a Docker container with realtime priority. The atemsys kernel module should be installed on the host in order to operate the container with the lowest possible capabilities and privileges.

The following additional settings, permissions for docker run are required:

Add atemsys device to container
--device=/dev/atemsys:/dev/atemsys
Allow max realtime priority
--ulimit rtprio=99
Add capability to set priority an lock memory
--cap-add=sys_nice
--cap-add=ipc_lock
Publish RAS server port 6000
-p 6000:6000

4.4.5. Setting up and running EcMasterDemo

  1. Unbind Ethernet Driver instance, e.g.

    echo 0000:00:19.0 > /sys/bus/pci/drivers/e1000e/unbind

  2. Load atemsys kernel module

    insmod atemsys.ko

  3. Copy files from EC-Master package /bin and a eni.xml to directory e.g. /tmp.

  4. Adjust LD_LIBRARY_PATH search locations for Real-time Ethernet Driver if necessary, e.g.

    export LD_LIBRARY_PATH=/tmp:$LD_LIBRARY_PATH

  5. Run EcMasterDemo

    cd /tmp
./EcMasterDemo -f eni.xml -intelgbe 1 1 -perf

See also

Running EcMasterDemo

4.4.5.1. Run in Docker container

  1. Unbind Ethernet Driver instance and load atemsys on the host.

  2. Create a directory on the host (e.g. ~/docker) and copy files from EC-Master package /bin and exi.xml into this directory.

  3. Start bash console in container
    sudo docker run -it \
    --name atem_container \
    --device=/dev/atemsys:/dev/atemsys \
    --ulimit rtprio=99 \
    --cap-add=sys_nice --cap-add=ipc_lock \
    -v ~/docker:/home/docker \
    -p 6000:6000 \
    ubuntu bash

    Command line arguments:

    • -it Allocate a pseudo-TTY and run container

    • --name atem_container Container name

    • --device=/dev/atemsys:/dev/atemsys Add atemsys device to container

    • --ulimit rtprio=99 Allow max realtime priority

    • --cap-add=sys_nice Add Linux capability to set priority

    • --cap-add=ipc_lock Add Linux capability to lock memory

    • -v ~/docker:/home/docker Mount previously create directory to container

    • -p 6000:6000 Publish RAS server port 6000

    • ubuntu bash Use Docker image ubuntu and start bash

  4. Run EcMasterDemo in container

    cd /home/docker
export LD_LIBRARY_PATH=.
./EcMasterDemo -f eni.xml -intelgbe 2 1 -perf

4.4.6. OS Compiler settings

Besides the general settings from Compiling the EcMasterDemo the following settings are necessary to build the example application for Linux

Possible ARCHs (see ATECAT_ARCHSTR in SDK/INC/Linux/EcOsPlatform.h):

  • aarch64

  • armv4t-eabi

  • armv6-vfp-eabihf

  • PPC

  • riscv64

  • x64 (aka amd64)

  • x86 (aka i686)

armv4t-eabi and armv6-vfp-eabihf are incompatible with each other. A potentially armv6-vfp-eabihf compatible system returns success on
readelf -A /proc/self/exe | grep Tag_ABI_VFP_args
Extra include paths
<InstallPath>/SDK/INC/Linux
<InstallPath>/Examples/Common/Linux
Extra source paths
<InstallPath>/Examples/Common/Linux
<InstallPath>/Sources/OsLayer/Linux
Extra library paths to the main EtherCAT components
<InstallPath>/SDK/LIB/Linux/<Arch>
Extra libraries (in this order)
EcMasterRasServer EcMaster pthread dl rt