4.4. Linux

4.4.1. OS optimizations

Linux itself is not real-time capable, so it is recommended to use it with the additional PREEMPT_RT patch.

The power management can disrupt cyclical processing, it is advisable to disable the CPUIDLE sub-system and CPUFREQ sub-system. The sub-systems can be disabled by changing the kernel command line parameters in the boot loader. On x86, x86_64 systems this is usually grub, on embedded devices with ARM, ARM64 is usually u-boot. It is also possible to build a custom kernel without these sub-systems.

Running a EC-Master application on a dedicated CPU core that is isolated from the Linux scheduler (ISOLCPUS) can provide additional stability.

4.4.1.1. CPUIDLE sub-system

Check if CPUFREQ sub-system is enabled:

ls /sys/devices/system/cpu/

If cpuidle appears in the list, it is enabled.

Disable CPUIDLE via the kernel command-line in grub:

linux  /boot/vmlinuz-4.19.0-16-rt-amd64 cpuidle.off=1

4.4.1.2. CPUFREQ sub-system

Check if CPUFREQ sub-system is enabled:

ls /sys/devices/system/cpu/

If cpufreq appears in the list, it is enabled.

Disable CPUFREQ sub-system via the kernel command-line grub:

linux  /boot/vmlinuz-4.19.0-16-rt-amd64 cpufreq.off=1

If CPUFREQ is not to be deactivated, the governor should be set to performance.

The currently active governor can be determined as follows:

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

The available governors with:

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_available_governors

To change governor use:

echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

4.4.1.3. ISOLCPUS

Isolate CPU core number 4 of a quad-core processor via the kernel command-line grub:

linux  /boot/vmlinuz-4.19.0-16-rt-amd64 isolcpus=3

Running EcMasterDemo on the isolated CPU core by setting the CPU affinity -a:

./EcMasterDemo -a 3

4.4.2. atemsys kernel module

To use Optimized Link Layers under Linux, the atemsys kernel module must be compiled and loaded. atemsys grants direct access to hardware to improve the performance.

All necessary scripts, source code and a detailed description of the installation can be found on https://github.com/acontis/atemsys. A ready-to-use Yocto recipe is also available on https://github.com/acontis/meta-acontis

4.4.2.1. atemsys as Device Tree Ethernet Driver

atemsys can also be used as a device tree driver to avoid certain conflicts between the link layer and the Linux kernel, e.g. power management, shared MDIO bus, etc..

A detailed guide on how to customize the device tree accordingly can also be found on https://github.com/acontis/atemsys. Example device tree modifications for different link layers/SoC can be found in https://github.com/acontis/atemsys/wiki.

Note

This is the preferred solution on all embedded devices with device tree support.

4.4.2.2. atemsys and PHY OS Driver

To use the PHY OS Driver, the acontis kernel module atemsys has to be included in the kernel device tree as an official driver for the Ethernet controller and doesn’t required any additional configuration at the application level. As a result atemsys can interact with Linux drivers.

4.4.3. Unbind Link Layer instance

Link Layer instances used by optimized Link Layers may not be bound by kernel drivers modules! Unbind can be done by unloading the kernel driver module, via the unbind interface of the driver or by modifying the device tree.

4.4.3.1. Unbind from kernel driver

The following command unbinds an instance without unloading the kernel driver module:

PCI

echo "<Instance-ID>" > /sys/bus/pci/drivers/<driver-name>/unbind

Example:

echo "0000:00:19.0" > /sys/bus/pci/drivers/e1000e/unbind

This call requires the PCI bus, device, function codes (in the above example it is 0000:00:19.0). The codes can be found using Linux commands like, for example:

ls /sys/bus/pci/drivers/e1000e

SoC

echo "<Instance-ID>" > /sys/bus/platform/drivers/<driver-name>/unbind

Example:

echo "2188000.ethernet" > /sys/bus/platform/drivers/fec/unbind

4.4.3.2. Unload kernel driver

Not all drivers allow unbinding of network adapters. If unbinding is not supported the corresponding Linux kernel driver must not be loaded.

The following command lists the loaded kernel modules that may conflict with optimized Link Layers:

lsmod | egrep "<module-name>"

Example:

lsmod | egrep "e1000|e1000e|igb"

PCI/PCIe: The command lspci -v shows which driver is assigned to which network card, e.g.:

lspci -v

...
11:0a.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)
...
Kernel driver in use: e1000e

Modules can be prevented from loading with the following commands:

echo blacklist <module-name> | sudo tee -a /etc/modprobe.d/blacklist.conf
update-initramfs -k all -u
sudo reboot

The following table shows the kernel modules related to the Optimized Link Layers:

Chip	Link Layer Name	Kernel driver(s)	Remarks
Broadcom Genet	emllBcmGenet	genet	Unbind not supported
Beckhoff CCAT	emllCCAT	ec_bhf
CPSW	emllCPSW	ti_cpsw
DesignWare 3504	emllDW3504	stmmac
	emllEG20T
Freescale TSEC/eTSEC v1/2	emllETSEC	gianfar_driver
Freescale FEC and ENET controller	emllFslFec	fec, fec_ptp
Xilinx Zynq-7000/Ultrascale	emllGEM
Intel Pro/1000	emllI8254x	igb, e1000, e1000e
Intel Pro/100	emllI8255x	e100
ICSS	emllICSS	prueth,pruss	Unbind not supported
RDC R6040	emllR6040
Realtek RTL8139	emllRTL8139	8139too, 8139cp
Realtek RTL8169 / RTL8111 / RTL8168	emllRTL8169	r8169	Unbind not supported
SuperH	emllSHEth	sh_eth	Unbind not supported
Generic	emllSockRaw

4.4.4. Docker

It is possible to operate EC-Master within a Docker container with realtime priority. The atemsys kernel module should be installed on the host in order to operate the container with the lowest possible capabilities and privileges.

The following additional settings, permissions for docker run are required:

Add atemsys device to container

--device=/dev/atemsys:/dev/atemsys

Allow max realtime priority

--ulimit rtprio=99

Add capability to set priority an lock memory

--cap-add=sys_nice
--cap-add=ipc_lock

Publish RAS server port 6000

-p 6000:6000

4.4.5. Setting up and running EcMasterDemo

1. Unbind Link Layer instance, e.g.

   echo 0000:00:19.0 > /sys/bus/pci/drivers/e1000e/unbind
   

2. Load atemsys kernel module

   insmod atemsys.ko
   

3. Copy files from EC-Master package /bin and a eni.xml to directory e.g. /tmp.

4. Adjust LD_LIBRARY_PATH search locations for Optimized Link Layers if necessary, e.g.

   export LD_LIBRARY_PATH=/tmp:$LD_LIBRARY_PATH
   

5. Run EcMasterDemo

   cd /tmp
   ./EcMasterDemo -f eni.xml -i8254x 1 1 -perf
   

See also

Running EcMasterDemo

4.4.5.1. Run in Docker container

1. Unbind Link Layer instance and load atemsys on the host.

2. Create a directory on the host (e.g. ~/docker) and copy files from EC-Master package /bin and eni.xml into this directory.

3. Start bash console in container

   sudo docker run -it \
       --name atem_container \
       --device=/dev/atemsys:/dev/atemsys \
       --ulimit rtprio=99 \
       --cap-add=sys_nice --cap-add=ipc_lock \
       -v ~/docker:/home/docker \
       -p 6000:6000 \
       ubuntu bash
   

   Command line arguments:

   • -it Allocate a pseudo-TTY and run container

   • --name atem_container Container name

   • --device=/dev/atemsys:/dev/atemsys Add atemsys device to container

   • --ulimit rtprio=99 Allow max realtime priority

   • --cap-add=sys_nice Add Linux capability to set priority

   • --cap-add=ipc_lock Add Linux capability to lock memory

   • -v ~/docker:/home/docker Mount previously create directory to container

   • -p 6000:6000 Publish RAS server port 6000

   • ubuntu bash Use Docker image ubuntu and start bash

4. Run EcMasterDemo in container

   cd /home/docker
   export LD_LIBRARY_PATH=.
   ./EcMasterDemo -f eni.xml -i8254x 2 1 -perf
   

4.4.6. OS Compiler settings

Besides the general settings from Compiling the EcMasterDemo the following settings are necessary to build the example application for Linux

Possible ARCHs (see ATECAT_ARCHSTR in SDK/INC/Linux/EcOsPlatform.h):

• armv4t-eabi

• armv6-vfp-eabihf

• aarch64

• x64 (aka amd64)

• x86 (aka i686)

• PPC

armv4t-eabi and armv6-vfp-eabihf are incompatible with each other. A potentially armv6-vfp-eabihf compatible system returns success on

readelf -A /proc/self/exe | grep Tag_ABI_VFP_args

Extra include paths

<InstallPath>/SDK/INC/Linux
<InstallPath>/Examples/Common/Linux

Extra source paths

<InstallPath>/Examples/Common/Linux
<InstallPath>/Sources/OsLayer/Linux

Extra library paths to the main EtherCAT components

<InstallPath>/SDK/LIB/Linux/<Arch>

Extra libraries (in this order)

AtemRasSrv EcMaster pthread dl rt