Gns3-gui: Using a virtual disk to transfer data to/from QEMU VMs

[b-ehlers]

Is your feature request related to a problem? Please describe.

Currently you can't easily transfer data to/from a QEMU VM. Especially transfering configurations would be very helpful. Issue #1095 wants to create a very comfortable way to share configurations similar to the current Dynamips/IOU solution. But it's also very complex to implement. For every device type a special solution must be found.

This idea should be ways simpler to implement, but is not as comfortable as #1095. This can serve as a transitional solution until a full import/export tool is available.

Describe the solution you'd like

Each QEMU device gets an empty 1MB FAT non-cloned raw image added as harddisk D. As almost every system supports FAT disks, the user gets an external storage device, where he can save configurations, but also other informations. It's a sort of virtual USB stick. GNS3 implements methods to transfer the files from this image via the GNS3 GUI to the user's host.

Transfering files from the image to the GUI:

• GUI sends request to server
• Server uses the mtools (https://www.gnu.org/software/mtools/) to extract the files from the raw image, puts them into a .zip archive and sends it to the GUI.
• The GUI stores that .zip file or optionally extracts it

Transfering files from the GUI to the image:

• GUI puts the files into a .zip archive and sends it to the server
• Server extracts the files from the zip archive, adds them with the mtools into an empty FAT image and uses this as the new harddisk D.

Advantages:

• Much simpler to implement as #1095
• Can be used for all file types, it's not limited to configuration files
• Supports a wide range of devices, no device specific code is needed
• Can store multiple files/configurations
• Uses only user mode tools to access the image, no root privileges needed

Disadvantages:

• The user must manually tranfer configurations from/to this virtual disk, the configuration doesn't get automatically activated.
• For performance reasons the virtual disk size should be limited, it's not intended for exchanging big files
• Works only when the VM is switched off

[b-ehlers]

My implementation is now complete. What remains is to test it under Windows, but I will refrain from that.

[grossmj]

Thanks @b-ehlers, I will check it out soon.

[grossmj]

PRs https://github.com/GNS3/gns3-gui/pull/3024 and https://github.com/GNS3/gns3-server/pull/1801 opened.

[grossmj]

@b-ehlers

It seems your implementation could be adapted to support extracting configs from IOSv and IOSvL2: https://github.com/GNS3/gns3-server/issues/1315#issuecomment-542883665 I haven't looked into it in too much details but it should work in theory.

We could use libguestfs for it but mtools seems simpler and more portable.

[b-ehlers]

Quite some time ago I made some few tests with libguestfs and found it quite slow. When I remember correctly, something around a second. That doesn't sound much, but if you have a topology with 10 routers and have to wait 10 seconds on every start and stop it could be annoying. Therefore I stopped using it. But maybe that has changed, or I made something wrong.

I made some tests using the new version of the IOUtools (https://github.com/GNS3/gns3-server/issues/1627) with IOSv to ensure that config extraction and insertion works. So basically the IOUtools should be ready.

I tested with

• vios-adventerprisek9-m.vmdk.SPA.156-2.T
• vios_l2-adventerprisek9-m.vmdk.SSA.152-4.0.55.E
• vios_l2-adventerprisek9-m.03.2017.qcow2

[b-ehlers]

I saw a lot of test failures in https://ci.appveyor.com/project/gns3-build/gns3-server/builds/34178776. That's mainly because the qemu base manager gns3server/compute/qemu/__init__.py creates an empty config image config.ing in the image directory. That irritates quite some tests.

[grossmj]

Quite some time ago I made some few tests with libguestfs and found it quite slow. When I remember correctly, something around a second. That doesn't sound much, but if you have a topology with 10 routers and have to wait 10 seconds on every start and stop it could be annoying. Therefore I stopped using it. But maybe that has changed, or I made something wrong.

I believe it is slow because libguestfs actually launches a VM in the background to mount images (I haven't checked that myself).

I made some tests using the new version of the IOUtools (GNS3/gns3-server#1627) with IOSv to ensure that config extraction and insertion works. So basically the IOUtools should be ready.

That's great, thanks for the confirmation :)

I saw a lot of test failures in https://ci.appveyor.com/project/gns3-build/gns3-server/builds/34178776. That's mainly because the qemu base manager gns3server/compute/qemu/__init__.py creates an empty config image config.ing in the image directory. That irritates quite some tests.

Yes, I noticed too. I will fix the tests later.

[grossmj]

I believe it is slow because libguestfs actually launches a VM

This confirms it: http://libguestfs.org/guestfs-performance.1.html

I can also see a lot can go wrong or limit performances. Using mtools looks like a better solution at the moment.

[b-ehlers]

The use of mtools is portable, lightweight and supports a wide range of qemu VM types. Of course a solution using libguestfs and IOUtools would be more comfortable, but limited for IOS based VMs. For me this is a base solution, until for some VM types, a full import/export tool is available.

Interesting is, what VIRL v2 is doing, see https://github.com/GNS3/gns3-gui/issues/1095#issuecomment-618936289. In my view that's too simple and limited.

[grossmj]

Interesting is, what VIRL v2 is doing, see #1095 (comment). In my view that's too simple and limited.

I agree, issuing some commands on the console port to export configs is not something I want to do.

[grossmj]

Implemented. We will work on extracting configs from IOSv, IOSvL2 etc. in the 3.x versions.

[b-ehlers]

@grossmj , thanks a lot for your integration work. In the near future I will remove my now obsolete patches at https://git.b-ehlers.de/ehlers/gns3-qemu-config.

Even though some parts of its README file are outdated now, other parts might give some useful hints. I attach it here so that the information is not lost.

Furthermore, I've adding the file config.img.zip

---

gns3-qemu-config - QEMU config disk

Currently you can't easily transfer data to/from a QEMU VM.
Especially transfering configurations would be very helpful.
But to share configurations similar to the current
Dynamips/IOU solution is very complex.
For every device type a special solution must be found.

This extension is much simpler to implement.
It adds a small DOS formatted disk as HD-D to every QEMU VM.
The user can use this storage device to save configurations,
but also any other data.
The GNS3 server converts this disk image to a ZIP archive,
which is accessible in the GUI by import/export configuration.
To access the files in the disk image the server needs the
mtools installed (https://www.gnu.org/software/mtools/).

The config disk uses the disk interface of HD-D, unless that
is of type "ide". If it is, the HD-A disk interface is used.
This prevents the use of "ide", unless HD-A is also using it.

Advantages:

• Much simpler to implement as a full configuration import/export.
• Can be used for all file types, it's not limited to configuration files.
• Supports a wide range of devices, no device specific code is needed.
• Can store multiple files/configurations.

Disadvantages:

• The user must manually tranfer configurations from/to this disk,
  the configuration doesn't get automatically activated.
• The user has to manually pack/unpack the ZIP files.
• For performance reasons the virtual disk size is limited,
  it's not intended for exchanging big files.
• Limitation: Works only, if HD-D is not used.

Patching the GNS3 sources

Apply this addition to a GNS3 sources with:

cd <gns3-gui source directory>
git am <path to patches>/gns3-gui*.patch
cd <gns3-server source directory>
git am <path to patches>/gns3-server*.patch

Afterwards do the normal installation from source of GNS3.

Installing mtools

Linux: mtools packages are available for most distributions,
see https://pkgs.org/download/mtools.

macOS: mtools is available in the Homebrew
package manager, install it with brew install mtools.

Windows: mtools binaries for Win32 are available at
https://github.com/foone/mtools_win32/releases.
They will also run on 64-bit Windows.
I have not tested my GNS3 extension on Windows, so
I have no idea if mtools works in this environment.

Optional: Create an empty DOS formatted disk image

This extension needs a DOS formatted disk image config.img
in the QEMU image directory of the GNS3 server.
If this image is missing, a 1 MB image will be created.

If you need a different size, create it as shown below:

disk_size=1024      # disk size in KiByte
dd if=/dev/zero bs=1024 count=$disk_size of=config.img status=none
printf 'drive x:\n  file="config.img"\n  partition=1\n' > mtoolsrc
MTOOLSRC=mtoolsrc mpartition -I -c -s 8 -h 16 -t $(($disk_size / 64)) x:
MTOOLSRC=mtoolsrc mformat x:
rm mtoolsrc

Afterwards copy it into the QEMU image directory of the GNS3 server.

config.img.zip contains the 1 MB image together with the build script.

Remaining steps

• Needs some tests on Windows, but I have no plans for that.
• Optional: pack/unpack the zip file on the GUI side.

[b-ehlers]

Just want to report a special use of the config disk. As junOS uses an unix OS, I was able to set the /config directory to the config disk (device /dev/ad3s1) in /etc/fstab on junOS/FreeBSD. That way I have the actual configuration directly accessible in GNS3.

I'm using only very old versions of junOS, so I don't know if that works with more recent version.

Olive 12.3R6.6:

root@% cat /etc/fstab
#version 3
# Device       Mountpoint  FStype  Options  Dump  Pass#
/dev/ad0s1a    /           ufs     rw        1    1
/dev/ad3s1     /config     msdosfs rw,noauto 0    0
proc           /proc       procfs  rw        0    0
/dev/ad0s1f    /var        ufs     rw,noauto 2    0
/dev/ad0s1b    none        swap    sw        0    0
/dev/ad0s1b    /tmp        mfs     rw,noauto 0    0
/dev/ad0s1b    /mfs        mfs     rw,noauto 0    0

vSRX 12.1X47-D20.7:

root@% cat /etc/fstab
# Device       Mountpoint  FStype  Options  Dump  Pass#
/dev/md0       /           cd9660  ro        0    0
proc           /proc       procfs  rw        0    0
/dev/ad3s1     /config     msdosfs rw        0    0
/dev/bo0s1b    none        swap    sw        0    0

[b-ehlers]

With the archive feature of IOS the changed configs can be automatically saved to the config disk. That has the additional benefit, that the user gets a backup of his configurations.

Add to IOS config:

 path flash1:$h
 write-memory