KVM+Virtual Function Configuration on CloudLab

Posted on In Disqus:

Here is a post about How to run mTCP applications in VM with Virtual Function.

1. Environment

1.1 about nodes

At CloudLab, we launch two c220g2 nodes, where X520 10Gb NIC supports SR-IOV, at Wisc site. We choose Ubuntu 16.04 64bit STD image.

To enable the SR-IOV support, you may edit /etc/default//grub to add intel_iommu=on in GRUB_CMDLINE_LINUX Then sudo update-grub and reboot the system.

To configure VF for each NIC, you may just reload the NIC driver with addtional parameters. (NOTE: pay attention to your driver, it may be ixgbe, i40e or something else.)

1
2
sudo rmmod ixgbe
sudo modprobe ixgbe max_vfs=2

To see whether your configuration is done correctly, just run sudo lspci | grep -i ether to see whether there are pci devices with Virtual Functions.

1.2 about KVM

1
2
3
4
5
### install some dependencies
sudo apt-get install -y qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils virt-manager libosinfo-bin libguestfs-tools virt-top

### add your current user to group libvirtd
sudo adduser `id -un` libvirtd

2. Test

2.1 Launch a VM

1
2
### create a disk 
qemu-img create -f qcow2 ubuntu.qcow2 20G
1
2
3
4
5
6
7
8
9
10
11
12
13
### launch a VM named test here
sudo virt-install \
--name test \
--connect qemu:///system \
--ram 10240 \
--disk path=/path-to-disk/ubuntu.qcow2,format=qcow2,bus=virtio,size=8 \
--vcpus 8 \
--os-type linux \
--network network=default \
--graphics none \
--console pty,target_type=serial \
--location=/path-to-img/ubuntu-14.04.5-server-amd64.iso \
--extra-args 'console=ttyS0,115200n8 serial’

Here, we define a domain named test with 8 vCPUs and 10GB memory. We add a default network interface just for SSH connection.

2.2 insert a pci-passthrough device

To inject a pci-passthrough device, namely VF in this setting, we only need to know the pci address at our host and modify the VM configuration xml file.

To check VFs' pci addresses, just run ip link show or lshw -c network -businfo. And add this info to sudo virsh edit test.

1
2
3
4
5
6
7
8
...
<interface type='hostdev' managed='yes'>
	<mac address='52:54:00:b1:fa:4f'/>
	<source>
		<address type='pci' domain='0x0000' bus='0x06' slot='0x10' function='0x0'/>
	</source>
</interface>
...

When you reluanch your VM, you may use ifconfig -a to check whether you have done correctly.

2.3 small modifications to mTCP

File /mtcp/src/dpdk_module.c, comment lines related to set TX/RX flow control of the NIC.

1
2
3
4
5
6
7
8
9
10
11
/* retrieve current flow control settings per port */
memset(&fc_conf, 0, sizeof(fc_conf));
ret = rte_eth_dev_flow_ctrl_get(portid, &fc_conf);
if (ret != 0)
	rte_exit(EXIT_FAILURE, "Failed to get flow control info!\n");

/* and just disable the rx/tx flow control */
fc_conf.mode = RTE_FC_NONE;
ret = rte_eth_dev_flow_ctrl_set(portid, &fc_conf);
if (ret != 0)
	rte_exit(EXIT_FAILURE, "Failed to set flow control info!: errno: %d\n", ret);

And if you rename VF nic name or set IP to it, the MAC may be all-zero. So just modify dpdk-17.08/lib/librte_eal/linuxapp/igb_uio/igb_uio.h to enable ifconfig $if_name hw ether $mac_addr.

1
2
3
4
5
6
7
8
9
10
11
12
13
...
static int
netdev_set_mac(struct net_device *netdev, void *p)
{
	struct sockaddr *addr = p;
	if (!is_valid_ether_addr((unsigned char *)(addr->sa_data)))
		return -EADDRNOTAVAIL;
	memcpy(netdev->dev_addr, addr->sa_data, netdev->addr_len);
	return 0;
}
...
	.ndo_set_mac_address    = netdev_set_mac,
...

Miscellaneous

If you use an image like Ubuntu 16.04 64-bit as the guest OS, you may lost your console when you boot your VM. This is because that grub file may not add the required setting by default. To solve this problem, just modify the grub and bootentry and then reboot.

Specifically,

1
2
sudo guestmount -d test -i /mnt
sudo vim /mnt/etc/default/grub

modify related fields (NOTE: you can set your own baud rate):

1
2
3
GRUB_CMDLINE_LINUX='console=tty0 console=ttyS0,115200n8'
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1”

and update the boot entry:

1
2
3
sudo vim /mnt/boot/grub/grub.cfg
### add additional parameters
console=ttyS0,19200 earlyprint=serial,ttyS0,19200

References

about how to add a pci passthrough host network device