Etcher: Create a test suite of real images that we will analyze on CI servers

[jviotti]

This suite will contain all sorts of images, from hybrid ones, DMGs, UDF, VMWare, etc, and each of them will be analyzed for certain characteristics (e.g: bootable or not).

The suite of images should be stored somewhere in the cloud, and the CI servers can download, analyze, assert, delete.

[jhermsmeier]

We might want to look into using S3 as a network fs; there seem to be plenty of modules for that around: https://npms.io/search?q=S3+fs

[jhermsmeier]

Images that would be interesting to have (feel free to edit this comment to add):

Connects to: #1189

Partitioned / Raw (.img, .bin, etc.)

• [ ] MBR (Master Boot Record) only
• [ ] MBR & GPT (GUID Partition Table)
• [ ] MBR & corrupt primary GPT, intact secondary GPT
• [ ] MBR & GPT, missing secondary GPT
• [ ] Protective MBR & GPT
• [ ] Hybrid MBR & GPT
• [ ] VBR (Volume Boot Record, flat single file system, unpartitioned)
• [ ] MBR with extended partitions (EBRs)
• [ ] APM (Apple Partition Map)

Alternative Raw Formats

• [ ] UDF/ISO (Plain, El Torito, Hybrids (containing a MBR / GPT))

Multi-Purpose Containers

• [ ] WIM (Windows Imaging File Format)
• [ ] DMG/UDIF (Apple Universal Disk Image Format)
• [ ] DMG/NDIF (Apple New Disk Image Format)

Virtual Disk Formats

• [ ] DISK (Linux Virtual)
• [ ] VHD (Fixed, Dynamic, Diff, Split)
• [ ] VHDX
• [ ] AVHD (Hyper-V Snapshot)
• [ ] VMDK
• [ ] VDI
• [ ] OVF

Vendor Images

• [ ] Apple Mac OS (DMG)
• [ ] Microsoft Windows (UDF/ISO with El Torito boot)
• [ ] Tizen IVI (Downloads: http://download.tizen.org/releases/daily/tizen/ivi-3.0/, features .bmaps)
• [ ] Fedora
• [ ] Canonical Ubuntu
• [ ] RPF Raspbian
• [ ] resinOS (https://resinos.io)
• [ ] Zorin OS (https://zorinos.com)

Note: Interesting info on Hybrid ISOs:
https://lists.ubuntu.com/archives/ubuntu-devel/2011-June/033495.html

[jviotti]

Wow, this is amazing information!

[lurch]

Is it possible to have a GPT without an MBR?
Is a VBR the same thing I've seen described as "superfloppy" formatting elsewhere?

There's also RiscOS (still used by some RPi users) which has a crazy boot / partitioning scheme of its own (which caused problems when I was working on NOOBS).

[lurch]

https://github.com/resin-io/etcher/issues/553 also suggests that VMWare images use some "strange" formatting.

[jhermsmeier]

use some "strange" formatting.

Just plain, non-hybrid UDF, as most .iso images, I suppose.

[lurch]

You're probably more aware of the intricate details than me, but should we also test ISO9660?

Also, it occurs to me that some of the images we might want to test, might not be publicly distributable. We could get around that by storing them in S3 as encrypted images, and then using the secure environment variables on the CI servers to decrypt them :)

[jhermsmeier]

Is it possible to have a GPT without an MBR?

No, the MBR (whether protective, or hybrid) is required, as it indicates the position of the primary GPT through an EFI system partition entry (with ID 0xEE or 0xEF).

Is a VBR the same thing I've seen described as "superfloppy" formatting elsewhere?

SuperFloppy refers to either SuperDisk or high-capacity floppies, such as Zip100 etc. I believe – and doesn't have much to do with on-disk data structures.
A VBR or PBR is a Volume / Partition Boot Record, with the same structure as a MBR, used for bootstrapping the OS. It's just called a VBR if the device is unpartitioned, a PBR if it resides in the first sector of a partition on a partitioned device, and an EBR if it's an extended partition record (a PBR without bootloader code pointing to partitions within a partition).

There's also RiscOS (still used by some RPi users) which has a crazy boot / partitioning scheme of its own (which caused problems when I was working on NOOBS).

Interesting – haven't come across that yet. It's a partition table & file system in one (which means it probably shouldn't co-exist with a MBR on disk).
I'll have a deeper look at the ADFS some time.

[jhermsmeier]

You're probably more aware of the intricate details than me, but should we also test ISO9660.

UDF is a subset of ISO9660, which was also published as ECMA-119, which in turn is based on ECMA-107, and extended by ECMA-167 – and then there's also the El Torito spec, which is also an extension on top of UDF for bootable CD-ROMs.

TLDR; We're covering ISO9660 quite well with UDF

Also, it occurs to me that some of the images we might want to test, might not be publicly distributable. We could get around that by storing them in S3 as encrypted images, and then using the secure environment variables on the CI servers to decrypt them :)

Considering the size and amount of images, and that for most tests we'll only need to read a few blocks, I think we'd be better off running a SMB or NFS server or something along those lines to have block-level access to those images.

Another (very compact) way to go about this would be to generate dynamic VHDs from the images, containing only the blocks we need for testing. That wouldn't work very well for blockmapped images, though.

[lurch]

...superfloppy / VBR ...

Yeah, looks like we're indeed talking about different names for the same thing http://reboot.pro/topic/4560-tool-to-see-if-superfloppy-or-harddisk-usb/?p=34730
I remember when PC BIOSes could only boot one style or the other, and you had to experiment with the 3 or 4 different "Live USB" downloads offered by each Linux distro, to see which one your PC worked with ;-)

UDF is a subset of ISO9660

I remembered UDF as coming significantly _after_ ISO9660 (back in the day I had a https://en.wikipedia.org/wiki/Amiga_A570 which was one of the very first home CD-ROM drives), and https://en.wikipedia.org/wiki/ISO_9660#Extensions_and_improvements clarifies that UDF is an _evolution_ of a _subset_ of an _extension_ of ISO9660 :rofl: