10. RTOS Devices (Partitioning)

10.1. Overview

By default, all devices are assigned for non Real-time guests. To assign a specific device to Real-time guests, the RTOSVisor has to be configured appropriately. For each device, a device assignment script has to be created which assigns such device to RTOS guests in general. In addition, a device configuration file for RTOS guests must be created which assigns such device to a specific RTOS guest.

Caution

The current script files only support device assignment to a single RTOS guest.

10.2. PCI/PCIe Devices

Hint

This section is valid for all kinds of PCI/PCIe devices. For assigning PCI/PCIe Ethernet devices you should follow the steps described in PCI/PCIe Ethernet Devices

10.2.1. Detect pci bus info

With lspci command a list of all available pci devices will be presented. The first number sequence (f.ex. 00:00.0) is the main part of the desired pci info. The number sequence is the vendor:device.function information. An additonal -D parameter shows the domain number.

sudo lspci -k

Following an example output:

hvuser@HV-TP106:~$ sudo lspci -k
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 Processor Family DRAM Controller (rev 09)
        DeviceName:  Realtek 8211
        Subsystem: Fujitsu Technology Solutions Xeon E3-1200 Processor Family DRAM Controller
        Kernel driver in use: ie31200_edac
        Kernel modules: ie31200_edac
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (Lewisville) (rev 04)
        Subsystem: Fujitsu Technology Solutions 82579LM Gigabit Network Connection (Lewisville)
        Kernel driver in use: e1000e
        Kernel modules: e1000e
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
        Subsystem: Fujitsu Technology Solutions 6 Series/C200 Series Chipset Family USB Enhanced Host Controller
        Kernel driver in use: ehci-pci
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b4)
        Kernel driver in use: pcieport
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b4)
        Kernel driver in use: pcieport
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b4)
        Kernel driver in use: pcieport
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
        Subsystem: Fujitsu Technology Solutions 6 Series/C200 Series Chipset Family USB Enhanced Host Controller
        Kernel driver in use: ehci-pci
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4)
00:1f.0 ISA bridge: Intel Corporation C202 Chipset LPC Controller (rev 04)
        Subsystem: Fujitsu Technology Solutions C202 Chipset LPC Controller
        Kernel driver in use: lpc_ich
        Kernel modules: lpc_ich
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 04)
        Subsystem: Fujitsu Technology Solutions 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller
        Kernel driver in use: ahci
        Kernel modules: ahci
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 04)
        Subsystem: Fujitsu Technology Solutions 6 Series/C200 Series Chipset Family SMBus Controller
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801
02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
        Subsystem: Fujitsu Technology Solutions 82574L Gigabit Network Connection
        Kernel driver in use: e1000e
        Kernel modules: e1000e
03:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) (rev 05)
        Subsystem: Fujitsu Technology Solutions MGA G200e [Pilot] ServerEngines (SEP1)
        Kernel driver in use: mgag200
        Kernel modules: mgag200

Example output with additional -D parameter:

hvuser@HV-TP106:~$ sudo lspci -kD
0000:00:00.0 Host bridge: Intel Corporation Xeon E3-1200 Processor Family DRAM Controller (rev 09)
        DeviceName:  Realtek 8211
        Subsystem: Fujitsu Technology Solutions Xeon E3-1200 Processor Family DRAM Controller
        Kernel driver in use: ie31200_edac
        Kernel modules: ie31200_edac
0000:00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (Lewisville) (rev 04)
        Subsystem: Fujitsu Technology Solutions 82579LM Gigabit Network Connection (Lewisville)
        Kernel driver in use: e1000e
        Kernel modules: e1000e
...

10.2.2. Case 1: PCIe Devices

In case a non-ethernet PCI Express (PCIe) device should be assigned to the real-time part, the supplied hv_addpcidev command should be used. The device <bus info> identified must be used as the first parameter of the hv_addpcidev command. The second (mandatory) parameter is a unique device name used on the Real-time side. The third (mandatory) parameter is a unique number (to be increased sequentially) for each assigned device, starting with 1.

hv_addpcidev 0000:02:00.0 rtos_pcidev1 1

This call creates two files: /hv/config/rtos_pcidev1.sh and /hv/config/rtos_pcidev1.config.

In case additional devices shall be assigned:

hv_addpcidev [DETECTED_BUS_INFO] rtos_pcidev2 2

Hint

The RTOS device name is mandatory. This parameter must be a unique name that is used to identify the device. This name will also be used in filenames that are created by the hv_addpcidev command. In this tutorial the default name used is rtos_pcidev1.

Caution

If devices with same name are assigned to the same Rtos the names of the keys in rtos_pcidev1.sh, rtos_pcidev1.config, etc. must be altered!

10.2.3. Case 2: Legacy PCI Devices

In case a non-ethernet Legacy PCI device should be assigned to the real-time part, the method described in this section has to be applied (all other information described for non-ethernet PCIe devices are valid, though). Compared with non-ethernet PCIe devices, a fourth parameter of the hv_addpcidev command is required.

sudo hv_addpcidev [DETECTED_BUS_INFO] rtos_pcidev1 1 <interrupt type>

Possible values of <interrupt type> are legacy or no_interrupt. If this parameter is not provided, the default MSI interrupt is used which is not feasible for Legacy PCI devices.

10.3. PCI/PCIe Ethernet Devices

Hint

All commands to be executed in the following guide have to be input via the shell. To open the shell right click on the desktop and select ‘Open Terminal here’ or press CRTL + ALT + T.

10.3.1. Device Identification

In a first step, it is required to determine the Ethernet device that shall be used by the Real-time geust. There are several ways how to detect the desired adapter.

10.3.1.1. Identify by hardware information

An easy way to identify an adapter is its hardware information:

lshw -class network

returns

*-network:1
    description: Ethernet interface
    product: 82545EM Gigabit Ethernet Controller (Copper)
    vendor: Intel Corporation
    physical id: 6
    bus info: pci@0000:02:06.0
    logical name: enp2s0
    version: 01
    serial: 00:0c:29:94:bb:c3
    size: 1Gbit/s
    capacity: 1Gbit/s
    width: 64 bits
    clock: 66MHz
    capabilities: bus_master cap_list rom ethernet physical logical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
    configuration: autonegotiation=on broadcast=yes driver=e1000 driverversion=5.15.0-88-acontis duplex=full ip=172.17.10.26 latency=0 link=yes mingnt=255 multicast=yes port=twisted pair speed=1Gbit/s
    resources: irq:16 memory:fd580000-fd59ffff memory:fdfe0000-fdfeffff ioport:2080(size=64) memory:fd520000-fd52ffff

We can see many information helping on identification: The network adapter enp2s0 is an 'Intel' type '82545EM' with MAC-ID '00:0c:29:94:bb:c3' and current link state 'link=yes'.

Hint

If RT-Linux will be used as RTOS, than remember the currently used driver (in this example: driver=e1000), as this driver can also be used in RT-Linux with the attached device. This could be achived by a modprobe [driver_name] call in the rtos console window.

10.3.2. Case 1: PCIe Ethernet Devices

In case a PCI Express (PCIe) device is assigned, the method described in this section has to be applied. The device name identified above must be used as the first parameter of the hv_addeth command. The second (mandatory) parameter is a unique device name used on the Real-time side. The third (mandatory) parameter is a unique number (to be increased sequentially) for each assigned device, starting with 1.

hv_addeth enp2s0 rtos_eth1 1

This call creates two files: /hv/config/rtos_eth1.sh and /hv/config/rtos_eth1.config.

In case additional network devices shall be assigned:

hv_addeth [DETECTED_NAME] rtos_eth2 2

Hint

The RTOS device name is mandatory. This parameter must be a unique name that is used to identify the device. This name will also be used in filenames that are created by the hv_addeth command. In this tutorial the default name used is rtos_eth1.

Caution

If devices with same name are assigned to the same Rtos the names of the keys in rtos_eth1.sh, rtos_eth1.config, etc. must be altered!

10.3.3. Case 2: Legacy PCI Ethernet Devices

In case a Legacy PCI device is assigned, the method described in this section has to be applied (all other information described for PCIe devices are valid, though). Compared with PCIe devices, a fourth parameter of the hv_addeth command is required.

sudo hv_addeth [DETECTED_NAME] rtos_eth 1 <interrupt type>

Possible values of <interrupt type> are legacy or no_interrupt. If this parameter is not provided, the default MSI interrupt is used which is not feasible for Legacy PCI devices.

10.4. Legacy Serial (COM) Devices (non PCI)

Run the following command to get an overview of the COM devices on your hardware.

sudo cat /proc/tty/driver/serial | grep -v "uart:unknown"

You will then get a list of devices, in the example below the hardware provides 2 serial ports, COM1 and COM2.

serinfo:1.0 driver revision:
0: uart:16550A port:000003F8 irq:4 tx:0 rx:0
1: uart:16550A port:000002F8 irq:3 tx:0 rx:0

The device names for the COM devices are ttyS0 (COM1), ttyS1 (COM2) etc. Before using a legacy serial device (COM1, COM2 etc.) in the RTOS, you need to unbind the Hypervisor Host driver. In a first step, determine the device numbers for the serial port, e.g. for COM1:

sudo tree /sys/class/tty/ttyS0/device/driver

You will then get the following information.

/sys/class/tty/ttyS0/device/driver
├── 00:02 -> ../../../../devices/pnp0/00:02
├── bind
├── uevent
└── unbind

The device numbers in the above example are “00:02”.

To unbind the driver for COM1, you need to run the following command.

echo -n "00:02" | sudo tee -a /sys/class/tty/ttyS0/device/driver/unbind

If you want to use the COM1 port in the Hypervisor Host again, unbind it.

echo -n "00:02" | sudo tee -a /sys/bus/pnp/drivers/serial/bind

You may add the following lines in /hv/config/usr_hvpart.sh to automatically bind/unbind the COM1 port:

[ "$cmd" == "add" ] && echo -n "00:02" | sudo tee -a /sys/class/tty/ttyS0/device/driver/unbind
[ "$cmd" == "delete" ] && echo -n "00:02" | sudo tee -a /sys/bus/pnp/drivers/serial/bind

10.5. RTOS assignment (general)

To assign a specific device to Real-time guests in general, the partitioning script /hv/config/usr_hvpart.sh must be adjusted.

The device assignment scripts <RTOS device name>.sh usually shall be executed automatically on system startup. To accomplish this, you need to add the respective <RTOS device name>.sh calls into the file /hv/config/usr_hvpart.sh. In our tutorial we use rtos_eth1 as the unique <RTOS device name>, as mentioned earlier.

gedit /hv/config/usr_hvpart.sh

The usr_hvpart.sh file should contain at least the following string after editing: source $HV_CONFIG/rtos_eth1.sh $cmd

The example below shows how the device with the unique name rtos_eth1 is assigned.

#!/bin/bash

cmd="add"
[ $1 == "delete" ] && cmd="delete"

# unbind devices (assign to RTOS)
source $HV_CONFIG/rtos_eth1.sh $cmd

Please run the hv_hvpart command with the parameter add or reboot the system to make the change effective.

hv_hvpart add

You may use the delete parameter to assign all RTOS devices back to the Hypervisor Host.

hv_hvpart delete

Hint

The $HV_BIN/hvpart.sh script will be automatically started via the systemd service controlled via /etc/systemd/system/hv_part.service. This script will call the usr_hvpart.sh script which includes user specific partitioning commands. This service can be enabled or disabled as shown below (by default, it is enabled)

sudo systemctl enable /hv/services/hv_part.service
sudo systemctl disable hv_part

Caution

The System Manager configuration tool will write the device assignment into /hv/bin/hvpart.sh, you must not change this file. Effectively, all devices listed in /hv/bin/hvpart.sh and /hv/config/usr_hvpart.sh will be assigned to the RTOS guests.

10.5.1. Verification

10.5.1.1. de-assignment check using Hypervisor Host

In a first step, check if the Hypervisor Host’s original driver is not used in conjunction with the devices assigned to the Real-time guest.

lspci -k

The output will look similar like the following excerpt:

    :         :        :         :        :         :        :         :        :         :
    :         :        :         :        :         :        :         :        :         :
    :         :        :         :        :         :        :         :        :         :
01:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
        Subsystem: Intel Corporation I210 Gigabit Network Connection
        Kernel driver in use: igb
        Kernel modules: igb
02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Backplane Connection (rev 03)
        Subsystem: Intel Corporation I210 Gigabit Backplane Connection
        Kernel driver in use: pci-stub
        Kernel modules: igb
03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Backplane Connection (rev 03)
        Subsystem: Intel Corporation I210 Gigabit Backplane Connection
        Kernel driver in use: pci-stub
        Kernel modules: igb
    :         :        :         :        :         :        :         :        :         :
    :         :        :         :        :         :        :         :        :         :
    :         :        :         :        :         :        :         :        :         :

In the above example, the instance 01:00.0 is used by Ubuntu (driver: igb, e1000e etc.) and the instances 02:00.0 and 03:00.0 are assigned to a Real-time guest (driver: pci-stub).

10.6. Assign a device to a specific Real-time guest

After creating the device configuration file <RTOS device name>.config, it needs to be included into the guest configuration file to become effective for the respective guest. The next steps describe how to add this device to the example Real-time Linux guest, we use rtos_eth1 as the unique <RTOS device name>.

cd /hv/guests/examples/rt-linux
gedit ./usr.config

Add the corresponding rtos_eth1.config entry to the includes section of the configuration file.

The following example shows the ‘modified’ usr.config file:

RtosConfig
;-----------------------------------------------------------------------
; acontis technologies GmbH
;
; Guest user configuration
;-----------------------------------------------------------------------

#include "/hv/config/rtos_eth1.config"

;-----------------------------------------------------------------------
; End of file
;-----------------------------------------------------------------------

Please adjust the RTOS-32 guest configuration file as well because we will use an RTOS-32 application for device assignment verification later.

cd /hv/guests/examples/rtos-32
gedit ./usr.config

10.6.1. Real-time guest assignment check

10.6.1.1. Real-time Linux

After assigning a device to the RT-Linux a driver could be loaded for that device.

Run the rtos:

cd /hv/guests/examples/rt-linux
hv_guest_start -view

Log in into Real-Time Linux and run modprobe [driver_name]:

vmf64 login: root
password:  root
modprobe e1000

Hint

In the default case the previously used Hypervisor Host driver can still be used in RT-Linux.

10.6.1.2. RTOS-32

We use the RTOS-32Demo application to quickly verify, if the assigned netwok adapter is visible to the rtos part.

Adjust the guest configuration setting to prepare starting the RTOS-32Demo:

cd /hv/guests/examples/rtos-32
gedit usr_guest_config.sh

and add the following line:

export osImage=$HV_ROOT/guests/examples/rtos-32/Loader.bin

Adjust the link to the demo application:

cd /hv/guests/examples/rtos-32
rm rtos32app.dlm
ln -s /hv/guests/examples/rtos-32/files/RTOS-32Demo.dlm rtos32app.dlm

Run the demo:

cd /hv/guests/examples/rtos-32
hv_guest_start -view

The output should look like this for the used network adapter (8086:1533) in this tutorial:

PCI BIOS Information

Vendor Device Bus Dev Func Class  Int IRQ TLat MSI Type
-------------------------------------------------------
FFFE   FFFE     0   0   0  000000  -    0    0  -  Invalid class code
FFFE   FFFE     0   2   0  000000  -    0    0  -  Invalid class code
FFFE   FFFE     0  20   0  000000  -    0    0  -  Invalid class code
FFFE   FFFE     0  22   0  000000  -    0    0  -  Invalid class code
FFFE   FFFE     0  26   0  000000  -    0    0  -  Invalid class code
FFFE   FFFE     0  27   0  000000  -    0    0  -  Invalid class code
FFFE   FFFE     0  28   0  000000  -    0    0  -  Invalid class code
FFFE   FFFE     0  29   0  000000  -    0    0  -  Invalid class code
FFFE   FFFE     0  30   0  000000  -    0    0  -  Invalid class code
FFFE   FFFE     0  31   0  000000  -    0    0  -  Invalid class code
8086   1533     1   0   0  020000  A    5    0  X  Network controller
FFFE   FFFE     2   0   0  000000  -    0    0  -  Invalid class code
10EC   8169     3   5   0  020000  A    4   64  -  Network controller
Current Date/time: 01/01/2018 00:00:00
Current Date/time: 01/01/2018 00:00:01
Current Date/time: 01/01/2018 00:00:02
Current Date/time: 01/01/2018 00:00:03
Current Date/time: 01/01/2018 00:00:04

List of threads:
Name            Prio State      R.Del FStck  MStck Scheds  CTime  CT%
---------------------------------------------------------------------
Main Task          5 Current          63060  61276     11  -      -
Idle Task          0 Ready             2164   2164    526  -      -
Comm               5 Delaying       1 32832  31724    527  -      -
IPTASK             6 BlckdWait         4128   3940      2  -      -
IPTIMER            6 Delaying      11  4120   3940     44  -      -

Interrupts:
 IRQ    Calls  FreeStack  Doubles  Time
-------------------------------------------
   0      526        744        0  -

Network:
 Ping will respond at IP address 192.168.157.2

In the above example, one 8086:1533 device at BDF 1,0,0 is assigned to the Real-time guest.

Finally, terminate the console connection to the real-time guest and stop the Real-time guest OS:

CTRL + C

cd /hv/guests/examples/rtos-32
hv_guest_stop

10.7. RTOS de-assignment

To de-assign a specific device from Real-time guests, the partitioning script /hv/config/usr_hvpart.sh must be adjusted.

In a first step, please de-assign all devices from RTOS guests using the delete parameter of the hv_hvpart command (assign all RTOS devices back to the Hypervisor Host):

hv_hvpart delete

Then you need to remove or uncomment the respective <RTOS device name>.sh calls from the file /hv/config/usr_hvpart.sh. Here we use rtos_eth1 as the unique <RTOS device name>, as mentioned earlier.

gedit /hv/config/usr_hvpart.sh

The usr_hvpart.sh file should contain at least the following string after editing: source $HV_CONFIG/rtos_eth1.sh $cmd

The example below shows how the device with the unique name rtos_eth1 is assigned.

#!/bin/bash

cmd="add"
[ $1 == "delete" ] && cmd="delete"

# unbind devices (assign to RTOS)
# source $HV_CONFIG/rtos_eth1.sh $cmd       --> uncomment or remove this line

Finally, run the hv_hvpart command with the parameter add or reboot the system to assign the remaining devices to RTOS guests again.

hv_hvpart add