Pkp-lib: supplementary files migration

[bozana]

The supp files in OJS 2.4.x had the filestage = 6 (ARTICLE_FILE_SUPP).
When migrated to OJS 3.x those files get filestage = 2 (SUBMISSION_FILE_SUBMISSION). This leads to an error when the editor clicks on "Change File" galleys grid row action.
Thus, the file stage of supp files should be changed to 10 (SUBMISSION_FILE_PROOF) when migrating to OJS 3.x. That would mean that they are not visible in the Submission Files grid.

[bozana]

PRs:
ojs master: https://github.com/pkp/ojs/pull/1667
ojs ojs-stable-3_1_0: https://github.com/pkp/ojs/pull/1706

Additional PRs to fix the function repairSuppFilesFilestage:
ojs master: https://github.com/pkp/ojs/pull/1929
ojs ojs-stable-3_1_1: https://github.com/pkp/ojs/pull/1930

Patches for those that do not want to migrate supp files as galleys:
3.1.1: https://github.com/pkp/ojs/pull/2021/files
3.1.1-2: https://github.com/pkp/ojs/pull/2021/files

[bozana]

@asmecher, could you take a look? Do you agree that those files should have the file stage = 10 (SUBMISSION_FILE_PROOF) instead of 2 (SUBMISSION_FILE_SUBMISSION)? Do you see any reason why it is not good to only have them in the Galley grid and not in Submission Files grid? THANKS!!!

[ajnyga]

Just wanted to know if this affects updating from 3.0.2/3.1 to 3.1.1?

And while I am here: I do not see any problem in storing the suppfiles only in the Galleys grid because in OJS2 there was no difference between the supplementary files in the submission stage and the production stage (can not remember the OJS2 terminology anymore). Connected to this is the problem that many OJS2 journals store sensitive data to the supplementary files, because they do not show up in the front end if the reader tools are disabled. Of course you can access them even without reader tools, but not without some knowledge of the system. So basically the OJS3 upgrade probably reveals sensitive supplementary files for many journals.

[bozana]

@ajnyga, yes, it affects that upgrade from 3.0.2/3.1 to 3.1.1.
And yes, the supp files are presented on the article page...

[ajnyga]

Ok. Because it turned out during our update that most of the supplementary files the journals had were not intended to be published. I could imagine this is the case also with a lot of other OJS2 journals as well.

So what I did is that I cretead excel lists of the supplementary files and asked all journals to check which files are meant to be public (not a lot of them were). Using these lists I did a small post upgrade script that remvoved the unwanted files from the galleys grid which I ran after the OJS2=>OJS3 upgrade in January. Meaning that I only removed the galley link - leaving all the files visible in the submission stage grid.

So if I am reading this issue right, the suggested change would remove those unlinked files also from the submissions grid. Which is a bit problematic to some of our journals because with OJS2 they used the supplementary files feature to store some editorial documents (agreements etc.) and with this update those would be lost (because representing them in the galleys grid again is out of the question of course).

But I guess that with the OJS3.1.1 upgrade I will just remove or edit that part from the script, because the solution I made in january has worked fairly well.

[ajnyga]

The issue is discussed here: https://forum.pkp.sfu.ca/t/all-the-files-words-figures-letters-have-been-published-with-the-new-ojs-3-0-version/26533
I am fairly sure that more an more OJS2 journals will encounter this problem when upgrading.

[bozana]

Hmmm... I see... Maybe than the best solution would be to copy that files, so that they stay in the submission files grid and also be in the galleys grid... I'll take a look...

[ajnyga]

I think that @asmecher pointed out last year that the scenario I described is not the way the supplementary files feature in OJS2 was meant to be used like. I agree, but at the same time I have a feeling that our journals are not the only ones that have used it the wrong way.

I am of course more than willing to just edit the 3.1.1. upgrade script to take into account the changes I made.

[asmecher]

@ajnyga, have you checked whether your supplementary files were inadvertently published e.g. via the Reading Tools or the OAI interface? I suspect they were. The change with OJS3 is that they're much more evident -- but I don't believe it publishes anything that wasn't already published in OJS2 one way or another.

[asmecher]

@jmacgreg may also be maintaining tweaks to work around this on our own upgrades. James?

[ajnyga]

hi @asmecher
Yes, definitely the supplementary files were published in OJS2 as well. In technical sense.

But most of our journal editors did not seem to understand that they were public because most of them did not use reading tools and did not see the files on the article page. Also OAI is something that the editors really do not know about.

So when the OJS3 upgrade came, they were surprised to see those files visible. And as you can see from the forum issue I posted above, our journals were not the only ones.

I do not have a problem in handling this in our own server, but suspect that may bump into other similar situations where the journals want to preserve the supplementary files they had in OJS2, but do not want them to be public.

[bozana]

I realized that migrating the supp files so that they are both in submission files grid and galley grid does not bring much -- those use cases as @ajnyga mentioned will however have to adapt the upgrade code to remove those migrated as galleys.
Maybe a kind of upgrade configuration option, how to migrate supp files -- as submission files or galleys -- would be helpful. @asmecher, would that maybe be possible?

[ajnyga]

Hi,

Just to make things more difficult, in our case there were both journals that had used supplementary files correctly and those that had not. In this regard I would think that having them on both by default would be a good option. But as I said above, in our part the problem has been solved already.

[bozana]

Hmmm... So maybe to leave it then as it should be i.e. as galleys.
In that case, @asmecher, could you review the PR: https://github.com/pkp/ojs/pull/1667 ?

[asmecher]

@bozana, please go ahead. I'm still curious whether PKP hosting is maintaining a patch for different behavior when upgrading installs with supplementary files they don't want published -- @mfelczak, any idea?

[mfelczak]

Hi @asmecher and @bozana , yes it would be very helpful to have the option to not publish supp files during upgrade. Many OJS2 editors have been using these to collect additional submission files for internal use.

[bozana]

I will close the issue, but maybe @jmacgreg and @mfelczak can however write it here then. Eventually we could provide an explanation here what is to be done/changed a) if supp files should not be migrated, b) if they should be migrated only as submission files (and not galley files).

[bozana]

Oh, I see now the message from @mfelczak...

[bozana]

I will then provide an explanation, eventually a patch, how to change the code. The only thing that I cannot do is the general fix/explanation if the situation is journal specific, in an multi-journal installation...

[bozana]

Here is the patch to migrate supp files from OJS 2.4.x to OJS 3.1.0 and 3.1.1 as submission files, and not as galleys: https://github.com/pkp/ojs/pull/1718/files

[bozana]

I believe that the other option, not to migrate supp files at all is actually not needed -- it would mean to remove them also from different DB tables (e.g. submission_files, submission_supplementary_files, metrics) and from the files folder. The patch to migrate them as submission files seems fine/enough for all cases to me. What do you think, @jmacgreg and @mfelczak ?

[mfelczak]

Thanks @bozana, yes having the option to migrate the supp files as submissions files should be sufficient.

[israelcefrin]

@bozana I've committed a new patch to be applied into OJS3.1.1, given new OJS version shows hunk fail when I attempt apply current one: https://github.com/israelcefrin/ojs/commit/26ac913eefa6076c5827455746e8cfa0bf4cd06e
Should I create a pull request to master branch as you did?

[bozana]

@israelcefrin, I am not sure if GitHub will keep that link forever, also when you delete your branch or that commit disappears for any other reason... Maybe you could try? Else, in case of such a PR it will... THANKS!!!

[bozana]

@israelcefrin, I will put the link to the PR here: https://github.com/pkp/ojs/pull/1906/files, so the PR can be closed... and you can do whatever you want (also remove it) with your branch/commit...

[israelcefrin]

Tks @bozana, I will use this link as source for future upgrades from OJS 2.4.x.

[bozana]

@israelcefrin, sorry, I think, this line should be also removed: https://github.com/bozana/ojs/blob/b942b210ce893e7714f226d1a912f1e14c3e7db9/dbscripts/xml/upgrade.xml#L162. Could you add it to your patch? -- for those upgrading from 3.x to 3.1.1

EDIT: maybe we will remove that function for all -- that function tries to repair supp files but it is not possible to differentiate if the supp file is from ojs 2.4.x or from 3.x. (that could be any files of a category defined as supplementary) S. also: https://forum.pkp.sfu.ca/t/ojs-3-1-1-bug-image-files-missing-for-all-papers-including-published-papers/40645/9

[bozana]

@asmecher, here are the PRs to remove the function repairSuppFilesFilestage:
ojs master: https://github.com/pkp/ojs/pull/1925
ojs ojs-stable-3_1_1: https://github.com/pkp/ojs/pull/1926
OK so?

[ajnyga]

Hi @bozana to sum up: we will never migrate the old OJS2 supplementary files back to supplementary files in journals that have already upgraded to OJS3 earlier?

[bozana]

Yes @ajnyga, that's correct -- the supp files from OJS 2 are there, in the system, but they are in the submission files grid instead of galleys grid -- apparently there were not so many cases because we haven't heard from any. We only have a few cases where other files from submission files grid were migrated wrongly to the galleys grid with that function/try.

[bozana]

The function convertSupplementaryFiles, s. https://github.com/pkp/ojs/blob/master/classes/install/Upgrade.inc.php#L1054, is still there, for those migrating from OJS 2 now. It will migrate supp files from OJS 2 to the galleys grid. If this is not wished (because the journals actually never wanted those files to be public), that the patch above has to be applied first, before doing the upgrade.

[asmecher]

@bozana, yes, that's probably not a bad solution -- but did you intend to leave the code in place in Upgrade.inc.php? I don't think it's called anywhere else once this is gone.

Is it possible to write a quick query up returning the files affected by this e.g. to an arbitrary stage? In my test a file that was in the submission stage disappeared from all the grids; it would be good to have something for users to run that would make these files visible, in case they've already upgraded and run in production that way.

[bozana]

Ah, yes, now I understand -- those files have SUBMISSION_FILE_PROOF and are in the proof folder, but they do not have an associated galley -- that function repairSuppFilesFilestage did not create a galley for such files... because the convertSupplementaryFiles did it before, but this is however not true for new supp files uploaded to OJS 3.x... Grrrr...
I actually should consider assoc_type and assoc_id in that function... Grrrr...

[bozana]

@asmecher, I fixed the function repairSuppFilesFilestage, so that it considers only supp files in the submission stage that have assoc_type = galley:
ojs master: https://github.com/pkp/ojs/pull/1929
ojs ojs-stable-3_1_1: https://github.com/pkp/ojs/pull/1930
OK so?

Then I will provide an SQL for those already used the wrong repairSuppFilesFilestage function, to find what files are affected and a script to help them appropriately restore them...

[bozana]

The function to find the wrongly migrated supp files using the old, buggy function repairSuppFilesFilestage would be:

SELECT ssf.*, s.context_id
FROM submission_supplementary_files ssf, submission_files sf, submissions s
WHERE sf.file_id = ssf.file_id AND sf.assoc_type IS NULL AND sf.file_stage = 10 AND sf.revision = ssf.revision AND s.submission_id = sf.submission_id

Then it have to be seen, in what file_stage those files actually are, in order to migrate them back correctly.
If we would like to have them all in the submission file stage then this function could be used:

>

function repairRepairSuppFilesFilestage() {

 $submissionFileDao = DAORegistry::getDAO('SubmissionFileDAO');

 import('lib.pkp.classes.submission.SubmissionFile');
 import('lib.pkp.classes.file.SubmissionFileManager');

 // get reviewer file ids
 $result = $submissionFileDao->retrieve(
     'SELECT ssf.*, s.context_id
     FROM submission_supplementary_files ssf, submission_files sf, submissions s
     WHERE sf.file_id = ssf.file_id AND sf.assoc_type IS NULL AND sf.file_stage = ? AND sf.revision = ssf.revision AND s.submission_id = sf.submission_id',
     array((int)SUBMISSION_FILE_PROOF)
 );
 while (!$result->EOF) {
     $row = $result->GetRowAssoc(false);
     $submissionFileRevision = $submissionFileDao->getRevision($row['file_id'], $row['revision']);
         $submissionFileManager = new SubmissionFileManager($row['context_id'], $submissionFileRevision->getSubmissionId());
         $basePath = $submissionFileManager->getBasePath() . '/';
         $generatedOldFilename = $submissionFileRevision->getServerFileName();
         $oldFileName = $basePath . $submissionFileRevision->_fileStageToPath($submissionFileRevision->getFileStage()) . '/' . $generatedOldFilename;
         $submissionFileRevision->setFileStage(SUBMISSION_FILE_SUBMISSION);
         $generatedNewFilename = $submissionFileRevision->getServerFileName();
         $newFileName = $basePath . $submissionFileRevision->_fileStageToPath($submissionFileRevision->getFileStage()) . '/' . $generatedNewFilename;
         if (!file_exists($newFileName)) {
             if (!file_exists($path = dirname($newFileName)) && !$submissionFileManager->mkdirtree($path)) {
                 error_log("Unable to make directory \"$path\"");
             }
             if (!rename($oldFileName, $newFileName)) {
                 error_log("Unable to move \"$oldFileName\" to \"$newFileName\".");
             } else {
                 $submissionFileDao->updateObject($submissionFileRevision);
             }
         }
     $result->MoveNext();
 }
 $result->Close();
 return true;

}

@asmecher , shall I then provide it in the upgrade or as a tool script?

[bozana]

@asmecher, the PR from @israelcefrin would then be the right one -- to migrate supp files from OJS 2 to submission files grid only, and removing the function repairSuppFilesFilestage: https://github.com/pkp/ojs/pull/1906/files. Could you take a quick look i.e. confirm, if OK so?

[ajnyga]

(waiting for repairRepairRepairSuppFilesFilestage 🤣 )

[bozana]

:rofl: Correct... Sorry... :worried:

[bozana]

@ajnyga, the latest decision is that we will not migrate supp files as galleys but only to submission files grid... -- it seems that this is the case how mostly the supp files were used in OJS 2...

[ajnyga]

So the supplementary files will not be added as galleys at all after upgrade from OJS2 to OJS3? Meaning that if there is an article that has actual supplementary files, these have to be added manually after the upgrade? Right?

[bozana]

Yes, I think so. S. https://forum.pkp.sfu.ca/t/ojs-3-1-1-bug-image-files-missing-for-all-papers-including-published-papers/40645/12.
We could then maybe provide a patch the other way around -- for those that do want to have them as galleys... :-)

[bozana]

@asmecher, I would just like to confirm with you if we want to leave the supp file migration to galleys after all and just fix the wrong function repairSuppFilesFilestage (that is only used for an upgrade from OJS 3) like here:
ojs master: https://github.com/pkp/ojs/pull/1929
ojs ojs-stable-3_1_1: https://github.com/pkp/ojs/pull/1930

Or if we would like to migrate supp files into the submission files grid and remove the function repairSuppFilesFilestage like here: https://github.com/pkp/ojs/pull/1906/files

Thanks!

[bozana]

I think I would still prefer the first solution...

[asmecher]

I trust your judgement and the code looks clean (though I haven't run a test upgrade). Please go ahead!

[bozana]

I am leaving the migration of supp files from OJS 2.4.x to galleys in OJS 3.1.1-1 as is and I've just merged the fix for the buggy repairSuppFilesFilestage function for those already updated to OJS 3.
Thus, tagging @ajnyga and @israelcefrin here as well, for them to know.
Thus, DO NOT FORGET: if someone does not want the supp files from OJS 2.4.x to be publicly available and migrated as galleys in OJS 3, as till now, the patch that @israelcefrin provides is necessary: https://github.com/pkp/ojs/pull/1906/files.

[bozana]

@israelcefrin, here the patch for the 3.1.1-2: https://github.com/pkp/ojs/pull/2021/files

[israelcefrin]

Thanks @bozana !!

[carzamora]

Hi, I have some questions about this issue... I have the OJS 3.1.1.0 and I found some files in the proof folder but the manuscript is only in the review stage, I think there are supplementary material because of the content of them (appendix and big table), but these files are not accessible from the OJS, I can access only in cPanel, my installation is not upgraded from OJS 2.x, this issue is resolved if upgrade to 3.1.1-2??

[bozana]

Hi @t4x0n, yes unfortunately I wrongly moved all supp files to the proof stage. This issue is solved in 3.1.1-2 but the affected 3.x installations will/cannot be repaired automatically :-( I hope you can fix those manually? -- That would be best and most secure way to solve it. Else, if you think that there are too many cases, let me know -- eventually you could try to run a script to move those supp files from the proof stage into the submission stage...

[carzamora]

thanks, I think this is a not big deal, maybe just a few submissions have problems.

[spiderkga]

Good afternoon!

I made an update to version 3 but the articles are being appreciated by the public.

Is there a way to fix this, or do I need to do the update process again?

[asmecher]

@spiderkga, have a look at this thread in our support forum: https://forum.pkp.sfu.ca/t/issue-with-supplementary-files-in-upgrading-from-ojs-2-4-8-1-to-ojs-3-2-1-1/61683/2

[spiderkga]

@asmecher I

think it was not what I asked for, I am sending a screenshot of the screen I need these files of the red frame to be hidden for public access

[asmecher]

@spiderkga, that thread discusses the same thing you're asking about: the content you're pointing out is actually published in your OJS 2.x, but is not easy to find. OJS 3.x is designed to make accessing that content easier, but does not publish anything that was not previously published. Some users misunderstood the feature in OJS 2.x, and only discovered that the documents were published when they upgraded to 3.x. The patch linked by Israel in the linked thread can be applied to change the behaviour when you run the upgrade. If you've already run the upgrade, you might need to review your submissions manually. In any case, that thread is the relevant place to continue the conversation.

[bdgregg]

OK - different error message on my end:

[cid:be355ae1-89fe-4e42-ad43-de959a29aaac]

I think I hit our production site on the first try. Are we sending the same attributes that we are doing for pitt.primo.exlibrisgroup.com?

-Brian.

Brian D. Gregg

Solutions Architect
University Library System
University of Pittsburgh
7500 Thomas Blvd.
Pittsburgh, PA 15208
voice: 412-648-3264
fax: [phone]

---

From: Alec Smecher [e-mail]
Sent: Tuesday, August 11, 2020 4:31 PM
To: pkp/pkp-lib [e-mail]
Cc: Subscribed [e-mail]
Subject: Re: [pkp/pkp-lib] supplementary files migration (#3016)

@spiderkgahttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fspiderkga&data=02%7C01%7Cbdgregg%40pitt.edu%7C6de25037d6644fb440ae08d83e357f36%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C637327746768748719&sdata=Z1UbweNKEH4aMrCILzVqx038gdH6MmOzWyFX3Q%2B5rps%3D&reserved=0, that thread discusses the same thing you're asking about: the content you're pointing out is actually published in your OJS 2.x, but is not easy to find. OJS 3.x is designed to make accessing that content easier, but does not publish anything that was not previously published. Some users misunderstood the feature in OJS 2.x, and only discovered that the documents were published when they upgraded to 3.x. The patch linked by Israel in the linked thread can be applied to change the behaviour when you run the upgrade. If you've already run the upgrade, you might need to review your submissions manually. In any case, that thread is the relevant place to continue the conversation.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpkp%2Fpkp-lib%2Fissues%2F3016%23issuecomment-672263758&data=02%7C01%7Cbdgregg%40pitt.edu%7C6de25037d6644fb440ae08d83e357f36%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C637327746768748719&sdata=4BWcBlWv%2BqrIBXyDtKU%2FpWj6UprSnDhxjwEzT%2Fy3ZgY%3D&reserved=0, or unsubscribehttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAC5B3RGNCLUB3U3GQJK7YTSAGTBHANCNFSM4ECFG73Q&data=02%7C01%7Cbdgregg%40pitt.edu%7C6de25037d6644fb440ae08d83e357f36%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C637327746768748719&sdata=OPWP%2FhwvuyXAFYXcMHeg2avwakUIoB7%2BA3BuOBDhRiI%3D&reserved=0.

[spiderkga]

@asmecher
But can I apply this patch after updating?

Could you tell me how you do it?

[asmecher]

@spiderkga, sorry, the patch can only be effective if it's applied before the upgrade process is run. Afterwards you'll have to review your content manually.