vhost-device-vsock

Design

The crate introduces a vhost-device-vsock device that enables communication between an application running in the guest i.e inside a VM and an application running on the host i.e outside the VM. The application running in the guest communicates over VM sockets i.e over AFVSOCK sockets. The application running on the host connects to a unix socket on the host i.e communicates over AFUNIX sockets. The main components of the crate are split into various files as described below:

• packet.rs
  • Introduces the VsockPacket structure that represents a single vsock packet processing methods.
• rxops.rs
  • Introduces various vsock operations that are enqueued into the rxqueue to be sent to the guest. Exposes a RxOps structure.
• rxqueue.rs
  • rxqueue contains the pending rx operations corresponding to that connection. The queue is represented as a bitmap as we handle connection-oriented connections. The module contains various queue manipulation methods. Exposes a RxQueue structure.
• thread_backend.rs
  • Multiplexes connections between host and guest and calls into per connection methods that are responsible for processing data and packets corresponding to the connection. Exposes a VsockThreadBackend structure.
• txbuf.rs
  • Module to buffer data that is sent from the guest to the host. The module exposes a LocalTxBuf structure.
• vhostuservsock_thread.rs
  • Module exposes a VhostUserVsockThread structure. It also handles new host initiated connections and provides interfaces for registering host connections with the epoll fd. Also provides interfaces for iterating through the rx and tx queues.
• vsock_conn.rs
  • Module introduces a VsockConnection structure that represents a single vsock connection between the guest and the host. It also processes packets according to their type.
• vhu_vsock.rs
  • exposes the main vhost user vsock backend interface.

Usage

Run the vhost-device-vsock device: vhost-device-vsock --guest-cid=<CID assigned to the guest> \ --socket=<path to the Unix socket to be created to communicate with the VMM via the vhost-user protocol> \ --uds-path=<path to the Unix socket to communicate with the guest via the virtio-vsock device> \ [--tx-buffer-size=<size of the buffer used for the TX virtqueue (guest->host packets)>] or vhost-device-vsock --vm guest_cid=<CID assigned to the guest>,socket=<path to the Unix socket to be created to communicate with the VMM via the vhost-user protocol>,uds-path=<path to the Unix socket to communicate with the guest via the virtio-vsock device>[,tx-buffer-size=<size of the buffer used for the TX virtqueue (guest->host packets)>]

Specify the --vm argument multiple times to specify multiple devices like this: vhost-device-vsock \ --vm guest-cid=3,socket=/tmp/vhost3.socket,uds-path=/tmp/vm3.vsock \ --vm guest-cid=4,socket=/tmp/vhost4.socket,uds-path=/tmp/vm4.vsock,tx-buffer-size=32768

Or use a configuration file: vhost-device-vsock --config=<path to the local yaml configuration file>

Configuration file example: yaml vms: - guest_cid: 3 socket: /tmp/vhost3.socket uds_path: /tmp/vm3.sock tx_buffer_size: 65536 - guest_cid: 4 socket: /tmp/vhost4.socket uds_path: /tmp/vm4.sock tx_buffer_size: 32768

Run VMM (e.g. QEMU):

qemu-system-x86_64 \ <normal QEMU options> \ -object memory-backend-file,share=on,id=mem0,size=<Guest RAM size>,mem-path=<Guest RAM file path> \ # size == -m size -machine <machine options>,memory-backend=mem0 \ -chardev socket,id=char0,reconnect=0,path=<vhost-user socket path> \ -device vhost-user-vsock-pci,chardev=char0

Working example

sh shell1$ vhost-device-vsock --vm guest-cid=4,uds-path=/tmp/vm4.vsock,socket=/tmp/vhost4.socket or if you want to configure the TX buffer size sh shell1$ vhost-device-vsock --vm guest-cid=4,uds-path=/tmp/vm4.vsock,socket=/tmp/vhost4.socket,tx-buffer-size=65536

sh shell2$ qemu-system-x86_64 \ -drive file=vm.qcow2,format=qcow2,if=virtio -smp 2 -m 512M -mem-prealloc \ -object memory-backend-file,share=on,id=mem0,size=512M,mem-path="/dev/hugepages" \ -machine q35,accel=kvm,memory-backend=mem0 \ -chardev socket,id=char0,reconnect=0,path=/tmp/vhost4.socket \ -device vhost-user-vsock-pci,chardev=char0

Guest listening

iperf

```sh

https://github.com/stefano-garzarella/iperf-vsock

guest$ iperf3 --vsock -s host$ iperf3 --vsock -c /tmp/vm4.vsock ```

netcat

```sh guest$ nc --vsock -l 1234

host$ nc -U /tmp/vm4.vsock CONNECT 1234 ```

Host listening

iperf

```sh

https://github.com/stefano-garzarella/iperf-vsock

host$ iperf3 --vsock -s -B /tmp/vm4.vsock guest$ iperf3 --vsock -c 2 ```

netcat

```sh host$ nc -l -U /tmp/vm4.vsock_1234

guest$ nc --vsock 2 1234 ```

Sibling VM communication

If you add multiple VMs, they can communicate with each other. For example, if you have two VMs with CID 3 and 4, you can run the following commands to make them communicate:

sh shell1$ vhost-device-vsock --vm guest-cid=3,uds-path=/tmp/vm3.vsock,socket=/tmp/vhost3.socket \ --vm guest-cid=4,uds-path=/tmp/vm4.vsock,socket=/tmp/vhost4.socket shell2$ qemu-system-x86_64 \ -drive file=vm1.qcow2,format=qcow2,if=virtio -smp 2 -m 512M -mem-prealloc \ -object memory-backend-file,share=on,id=mem0,size=512M,mem-path="/dev/hugepages" \ -machine q35,accel=kvm,memory-backend=mem0 \ -chardev socket,id=char0,reconnect=0,path=/tmp/vhost3.socket \ -device vhost-user-vsock-pci,chardev=char0 shell3$ qemu-system-x86_64 \ -drive file=vm2.qcow2,format=qcow2,if=virtio -smp 2 -m 512M -mem-prealloc \ -object memory-backend-file,share=on,id=mem0,size=512M,mem-path="/dev/hugepages2" \ -machine q35,accel=kvm,memory-backend=mem0 \ -chardev socket,id=char0,reconnect=0,path=/tmp/vhost4.socket \ -device vhost-user-vsock-pci,chardev=char0

```sh

nc-vsock patched to set .svm_flags = VMADDR_FLAG_TO_HOST

guestcid3$ nc-vsock -l 1234 guestcid4$ nc-vsock 3 1234 ```

License

This project is licensed under either of

• Apache License, Version 2.0
• BSD-3-Clause License