Dosbox-x: WfW 3.11 - corrupted progman GRPs when using imgmount + internal shell + S3 driver

[diaphone0]

Hi, and sorry if my English is not perfect :)

Describe the bug
This might be a niche, but when I'm trying to run WfW 3.11 from a img-mounted drive and with the internal dosbox shell.
In the first time after running win.com the program manager loads with no issues, but on subsequent runs most of the program groups are missing, and their grp files are reported as corrupted.

To Reproduce

Steps to reproduce the behavior:

1. start dosbox-x (default config, no conf file is present)
2. imgmount c test.img
   note: this img file has WfW 3.11 (hebrew edition) installed
   i've attached this img file here: https://drive.google.com/file/d/1U-qXs5FtpOTAS2llv5iFdcTd4Rr6LTIX/view?usp=sharing

1. C:
   cd windows
   win.com (should start with no issues)
   (close progman to exit windows)
   run win.com again
   (at this point there should be warnings about corrupted GRP files and many program groups should be missing)

2. this doesn't happen on the following cases:
   a. booting test.img and running win.com from ms-dos:
   boot -l c (F1)
   win.com
   (this image is bootable and includes ms-dos 6.22 as well)
   b. using basic VGA driver instead of S3 driver, I've replaced it via windows setup:
   cd windows
   setup
   (select VGA for display driver)
   c. mounting the same content with standard dirctory mount (and not imgmount)

Screenshots (things are in hebrew...)
program manager after first win.com run

warnings after second win.com run
(it says "Program group (...grp) file is invalid or corrupted. Please recreate the program group")

then, most of program groups are gone:

Environment (please complete the following information):

• Operating system: Windows 10 64bit
• DOSBox-X release version or commit SHA1 this issue occurred on
  Tested on:
  0.83.4 (073b2d6)
  0.83.2
  0.82.26

• Used configuration, e.g. dosbox-x.conf
  none

[rderooy]

I can confirm some kind of corruption occurs when I follow your instructions.

Here is MAIN.GRP. Left is the corrupted version, right is the unmodified original.

[rderooy]

I can also confirm that the issue does not occur if I copy the contents of the disk image to a host folder, and mount that host folder as my C: drive and then run WIN.COM

[rderooy]

I tried to duplicate the problem with the English WFW 3.11. It has the S3 driver installed, but the issue does not occur.

I noticed however that your using an older S3 driver, and are running in backward compatibility mode.

DOSBox-X emulates the S3 Trio64 which is the 764 chipset. Your using the 864 driver, which (confusingly) actually pre-dates the 764. Could you try using the S3 driver from here: https://www.vogons.org/viewtopic.php?t=9405

[diaphone0]

Thanks for the response,
I've tried to fresh install English wfw3.11 (got it at winworldpc.com) using dosbox shell,
And the GRP corruption happend already in the gui portion of windows installation.

So I've tried again t install from booted DOS 6.22 and after the installation there was no corruption even when starting windows from dosbox shell with the s3_764 driver.

i've made some more tests so i'll list them in short:

• fresh-install English from dosbox shell - has corruption even during setup
• fresh-install Hebrew from dosbox shell - has corruption even during setup
• fresh-install English from booted dos + S3_764 - no corruption at all
• fresh-install English from booted dos + S3_864 - has corruption under dosbox shell
• fresh-install English from booted dos + S3_764 AND switching to 864 - no corruption at all.
• fresh-install Hebrew from booted dos + S3_764 - still has corruption...

Thanks!

[Wengier]

I guess there exists some bug in the FAT driver which causes the issue, which has already existed in the code for some time. But the fact that the issue is especially associated with Hebrew version and/or the S3_864 driver makes it even more mysterious...

[joncampbell123]

Does the corruption depend on the name of the files? The characters used within? Does corruption happen if everything in the GRP is only plain ASCII?

Also, is this the code page Windows is using? https://en.wikipedia.org/wiki/Code_page_862

[Wengier]

I used the test image @diaphone0 provided in the first post as the test case, and I don’t think the corruption depends on the file name, because the group files are all plain ASCII names like MAIN.GRP, GAMES.GRP, etc, but rather, the corruption seems to occur if there are at least 2 icons in the group, even if they are plain ASCII (more verifications can be done for this, but seem to be correct). Also, the group file will not corrupted if it had the read-only attribute. I tested these with the default code page (437) for DOSBox-X, but with the Hebrew Windows he provided. I could not check or change the Windows settings because they are all in Hebrew which I do not understand.

[diaphone0]

@joncampbell123 CP862 was used mainly in Hebrew-enabled MSDOS, while Windows 3x and onwards uses "windows-1255" encoding for hebrew letters (in ANSI format, unicode cases excluded)

Anyway, I've noticed in the hexdump-diff from the screenshot @rderooy provided that everything is one byte-shifted in the left side, so an exrta byte was added somewhere.
I'll check if that the same case for all corrupted files.

I also tested this on the Daum build and there are no corruption issues there, so indeed it might be related somehow to the FAT driver enhancements...
I'll try to test it on earlier releases that have FAT changes mentioned in the changelog, in attempt to pin-point the commit that causes this issue.

[diaphone0]

@Wengier I'm not sure if that's related only to hebrew encoding, but to make things easier I've attached an image with English WfW3.11 & S3_864 intalled:
https://drive.google.com/file/d/1gVV5ck6FPzfh1golefKcypJmeRvdSR5i/view?usp=sharing
In this case the corruption errors appear on the third time that windows ia launched.
(This image also include both 764 & 864 drivers under C:\INSTALL, I've considered to include wfw eng & heb setup files as well but I'm not sure if thats appropriate, after all GitHub is now MS-owned... )

The corruption issues appear also during the English windows installation, as during the GUI stage of the installation the program manager is being loaded twice, first time for intial GRP creation and second time for auto adding searched dos programs (in my case Z:\EDIT.COM):

[diaphone0]

It seems that the last release that doesn't have this issue is this one: (tested with heb 3.11 + S3_864)
https://github.com/joncampbell123/dosbox-x/releases/tag/dosbox-x-v0.82.21-beta-27

The next FAT-related commit is this one:
https://github.com/joncampbell123/dosbox-x/commit/6626bc359ccc036f51cf018eb4ae26bba84a3eda

at the moment I have difficulties to build dosbox from this point (currntly i'm using an old mingw environment which is kinda messy)

Update:
I eventually managed to compile at this commit (6626bc359ccc036f51cf018eb4ae26bba84a3eda) and the corruption happens with it.
Then I've checked again with the previous commit (6548ef865cd1a826427f0f143b3fda433b22688c) and then there is no corruption at all, so I think that this is the earliest point which has the problem

[Wengier]

@joncampbell123 and @diaphone0 I can also confirm that the issue only occurs with the change in commit 6626bc3 applied. If I comment out line 418 in drive_fat.cpp in the latest code the problem will immediately disappear:

if (sizedec == 0) goto finalizeWrite;

@joncampbell123 probably knows how to properly change this code to fix both this issue and the issue you intended to fix in that commit.

[joncampbell123]

That line was added so that writing to the end of a cluster doesn't preemptively allocate the next cluster. It was needed to silence SCANDISK warnings about "lost clusters", where the extra cluster was not counted by file size and therefore lost.

The intent there is to flush the sector, and then exit before allocating the next cluster. If you modify that line like below, does it resolve the issue?

if (sizedec == 0) { curSectOff = 0; goto finalizeWrite; }

[Wengier]

@joncampbell123 Yes! I can confirm that the modification does resolve the issue. Thanks @joncampbell123 for this and the explanation of that commit.

[diaphone0]

I can confirm as well, works like a charm!
Many thanks!