Dataverse: Setting for public only installations, new API endpoint for restricting files

[scolapasta]

As we work with groups like SBGrid and MOC, we've designed these workflows where all files need to be public (i.e. restricting a file on the dataverse side isn't useful, since the files are available publicly outside the system). So we need to have a setting to make the dataverse installation public only. If this setting is set, then we would not render UIs that allow users to restrict files and would have those commands throw unsupported exceptions.

[mheppler]

UI impact (reference doc):

• Dataset
  
  
  
  • [ ] ~Restricted lock icons~ (will not be applying render logic, because we don't need to turn off something that can't be turned on)
  
  
  • [ ] ~Request Access button~ (ditto)
  
  
  • [x] Edit button > Permissions, conditional render for "Files"
  
  
  • [x] Edit Files button > Restrict/Unrestrict options
  
  
  • [x] Terms tab > Restricted Files + Terms of Access panel, conditional render for "Terms of Access" header text
  
  
  • [x] Terms tab > Restricted Files + Terms of Access panel > Restricted Files, hide label and text
  
  
• Edit/Create Dataset
  
  
  
  • [x] Create Dataset > Warning msg... "File Access - Files are stored on a publicly accessible storage server."
  
  
• Edit Dataset Terms
  
  
  
  • [x] Terms tab > Restricted Files + Terms of Access panel, conditional render for "Terms of Access" header text
  
  
  • [x] Request Access, hide option + label
  
  
• Upload Files (both create/upload workflows)
  
  
  
  • [ ] ~Restricted lock icons~
  
  
  • [x] Restrict button
  
  
  • [x] Edit/Upload Files > Warning msg... "File Access - Files are stored on a publicly accessible storage server."
  
  
• Edit Files
  
  
  
  • [ ] ~Restricted lock icons~
  
  
  • [x] Restrict button, hide
  
  
  • [x] Edit/Upload Files > Warning msg... "File Access - Files are stored on a publicly accessible storage server."
  
  
• Manage File Permissions
  
  
  
  • Page link removed from dataset pg
  
  
• Files
  
  
  
  • [ ] ~Restricted lock icons~
  
  
  • [ ] ~Request Access button~
  
  
  • [x] Edit button > Restrict/Unrestrict options
  
  
• Dataverse
  
  
  
  • [x] Access facets, remove
  
  
  • [ ] ~Restricted lock icons~
  
  
  • [x] dataverseLinksStayOnPage param removed, no longer in use #1521
  
  
• Bundle
  
  
  
  • [x] Create Dataset + Edit/Upload Files > dataset.message.publicInstall > "File Access - Files are stored on a publicly accessible storage server."
  
  
• User Guide > Dataset + File Management
  
  
  
  • [x] File Handling + Uploading
  
  
  • [x] Tabular Data Files
  
  
  • [x] File Tags
  
  
• API Guide
  
  
  
  • [x] Native API
  
  
• Installation Guide
  
  
  
  • [x] Configuration
  
  

[pdurbin]

@ferrys as I mentioned to you, @bsilverstein95 and @raprasad at lunch yesterday, I encourage you to consider making the correctness of the code you are writing for this issue testable in an automated fashion via API tests. One challenge will be that currently the ability to restrict files is a GUI-only feature. Note that just yesterday @mdmADA asked in #3873 if we could add the ability to restrict files via API (a duplicate of #2497) and related to #3440. I gave @bsilverstein95 #3431 about getting set up to write API tests using REST Assured so you may want to check in with him and take a look at the API tests I asked him to write last summer in pull request #3299 (in ConfirmEmailIt.java).

[dlmurphy]

For #3747 I added this warning to the Dataset Management page of the User Guide and the Config page of the Installation Guide:

Note: At present, any file restrictions that users apply in Dataverse will not be supported in Swift. This means: if you set a file on Dataverse as "restricted", a user without proper permissions could bypass that restriction by accessing the file through Swift. For now, do not rely on file restrictions to limit access to data in a cloud-enabled Dataverse.

This warning will need to be edited or removed to reflect that fact that public-only installations are now an option.

[pdurbin]

Interesting. I didn't know you added that, @dlmurphy . Looks good. Thanks.

[dlmurphy]

Also, are we concerned that public-only installations will still connect with the same set of guides that still explicitly refer to and explain the file restriction feature? Is it possible to connect public-only installations with an alternate set of guides without those sections?

Edit: A low-impact (but potentially slightly confusing) solution might be to add a note in those sections mentioning that the feature is disabled in public-only installations.

[pdurbin]

@dlmurphy I say no to alternate guides. The guides should be a description of what's possible with the product (Dataverse) and it should mention, where appropriate, how configuration options affect the behavior that users will see. You've already done this by mentioning how Swift being turned on interacts with file permissions and we should continue in this vein.

[ferrys]

I was chatting with @mheppler and @TaniaSchlatter about adding an initial message for contributors to a Cloud Dataverse which informs them that they are using a public install (since it would not be obvious), so they know their data will be public. If we choose to implement this, we could also mention the file restriction features there.

[dlmurphy]

@pdurbin, that's a good point. I agree, let's not have alternate guides. I'll go through the guides and make any edits needed to make sure that users are clear that file restrictions aren't enabled on public-only installations.

@ferrys, that initial message sounds like a great idea. I was concerned that users wouldn't know whether their install was public-only or not, so getting them oriented up front with a message like that is really helpful. Keep me posted on that message, and if you need any help writing it or proofreading, let me know!

[ferrys]

Observations from the current implementation of the code:

• Restricting a file on an existing dataset through "Upload Files" and restricting a file on an existing dataset through the "Edit Files" tab both call the UpdateDatasetCommand to update the metadata.
• Restricting a file on the File Page and restricting a file by editing the metadata through the "Edit Files" tab both call the database to update the metadata.

So, considering all things, I think I will take the time to create something similar to a RestrictFileCommand so it is both accessible through the API (#3873) and able to be tested in an automated fashion.

[pdurbin]

Let's retest #3653 when we put this issue through QA.

[pdurbin]

Let's also retest #3576 when we put this issue through QA.

[pdurbin]

3065 would be nice to re-test as well when this issue goes to QA.

[pameyer]

Looks good to me. It would probably be helpful to update user/dataset-management.rst to mention that restricting files may not be available depending on installation configuration.

[pdurbin]

@pameyer thanks, speaking of documentation, I just set up job to build hourly at http://guides.dataverse.org/en/3776-public-only/ for @dlmurphy and everyone to take a look. We need these docs to work well for both "public only" installations and "non-public only" installations. What should we call non-public only installations?

[dlmurphy]

Reviewed all documentation to ensure that users of public-only installations don't get confused by mentions of file restriction. I have elected only to add disclaimers to sections of the docs that discuss _the act of restricting_ a file, not sections that discuss how to _access_ restricted files. In the former case, users may look for a feature and find it missing. In the latter case, users will never encounter the feature in the first place and won't have the opportunity to get confused about its absence.

[dlmurphy]

@mheppler and @pdurbin mentioned that public-only installations have in-app messaging that explains the absence of file restrictions. I'm interested in seeing a screenshot or demo of this so I can review the messaging.

[pdurbin]

@dlmurphy yes, here's the screenshot you wanted. It says, " File Access - Files are stored on a publicly accessible storage server."

[pdurbin]

@dlmurphy also, code has been deployed to the dvn-build server if you'd like to click around the UI to see the changes.

[sekmiller]

@ferrys can you add a test to RestrictFileCommand where publicInstall is set to true?

[dlmurphy]

I think that the "File Access" message at the top of the Create Dataset page does get the concept across fine, but I worry that it's not emphatic enough. My understanding is that ultimately this message is supposed to be a warning that keeps researchers from uploading sensitive data to a public-only installation. If this is accurate, then this message might need more of a cautionary tone.

What do folks think of this messaging:
"Warning - Files are stored on a publicly accessible storage server. All data you upload can be accessed by anyone."
or
"Public File Access - Files are stored on on a publicly accessible storage server. Consider the sensitivity of your data before uploading."

[pdurbin]

@dlmurphy yeah, I agree that it wouldn't hurt to make the message a bit more cautionary. I'd love to hear @pameyer weigh in on this since his users are going to be reading it. We could consider making in configurable, similar to :DatasetPublishPopupCustomText: http://guides.dataverse.org/en/4.7/installation/config.html#datasetpublishpopupcustomtext . For context, this was requested in #2833.

[pdurbin]

@dlmurphy is going to run the text change past @pameyer and then give it to @bsilverstein95 to commit. Thanks, all!

[dlmurphy]

Consulted with @pameyer and @ferrys and this is the message we're going with for the Create Dataset page: "Warning: Once your data is published, you will not be able to restrict access to it."

[mheppler]

Added a minor CSS fix here, to return the selected tab text color to it's original grey, and not the blue that is used for non-selected tabs. Here is a screenshot of the fixed UI, with the "Files" selected tab text styled grey.