Orleans: Avoiding the OrleansAdoNetContent folder

[palpha]

The solution to #4243 causes repositories and projects to be polluted by the OrleansAdoNetContent folder. One has to employ various hacks to ensure that the folder is removed after package restore, and in the case of dotnet restore there doesn't even seem to exist any target to use with AfterTargets to get rid of the files.

I see how the files could be useful for someone just starting out with Orleans, but not for an established project. In my case, the files are possibly harmful, as I'm running on a modified schema and these files would only confuse my colleagues.

A possible solution would be to move the copying of these files into the OrleansSQLUtils project. One can then reference the individual packages to bypass the copying of the SQL files.

Has anyone else spent way too much time fighting these files? :) Any other ideas as to what can be done about it?

[sergeybykov]

If these files are causing troubles, I don't see a problem with moving them somewhere else. OrleansSQLUtils though is only a meta-package with zero of its own content. So I'm not sure we want to pollute it.

I wonder what @veikkoeeva thinks.

[palpha]

It's precisely because it is a meta package that I think it's a good place for the copying of these files -- using the meta project is a convenience, an advanced user can use the individual projects instead.

Another possible solution could be to check some variable that can be set if one wants to skip the copying.

[veikkoeeva]

I agree this is annoying. Less of a trouble though that people are storing the files in gists or copying from the GH since they aren't easily found anymore (since the Nuget directory isn't easily found). The previous solution was mor or less a solution to this particular problem. As alluded by @palpha .

I don't have an opinion on other solutions other than keeping the files easily found. That being written, a suggestion solving:

1. Easily to be find place.
2. Doesn't interfere with development.

And we have a good point for refactoring.

I think a good solution would be just to lay them into one place in some "well known place", but what could it be? I think it's near mandatory to allow modified scripts in larger scale deployments due to deployment differences (physical, data quantity, GDPR, whatnot) so fixing it inside assembly or some other metadata would be a very opinionated move.

I have a (personal) feeling it'd be just plain easier if there were only one script even if it would create a few more tables in the DB in some cases (user doesn't use all the features). It's less of a trouble for maintenace and testing too (albeit the changes have been implemented already). I feels it should be less work to deploy the DB using whatever solution one wants to use (my current is just files copied to .sqlproj and using post deployment to deploy them to appropriate schemas and file groups, and yes, te extra files do annoy a bit). Then have a meta package that contains core ADO.NET logic and refer this one script. As it stands, the current solution is set up differently (namespace pragmas and all, a tad annoying from a dev perspective :)).

Do you have change proposal, @palpha ? In short, it would be nice to solve this somehow. :)

[palpha]

My humble opinion is that the OrleansSQLUtils meta project would be a good place. I'm not a NuGet expert, but isn't contentFiles a thing? If the files are packaged as content, they can be excluded when consuming the package (at least with Paket). In any case, bypassing the meta package will bypass the files.

As for the separation of the scripts, I don't mind and it's kind of helpful when shadowing changes in my modified scripts (eg https://github.com/palpha/orleans/blob/postgres-with-schema/src/AdoNet/Orleans.Clustering.AdoNet/PostgreSQLWithSchema-Clustering.sql) that it's in smaller chunks.

[veikkoeeva]

@palpha So the idea would be:

1. Put the separate scripts to the meta-package as opposed to the individual packages. This would add logic to the "advance user case" you noted that they'd like to have the individual packages. As in this case file existence would need to be checked. In fact, it should be checked already now so that modified files aren't overriden (albeit they can show in multiple projects).
2. Make the scripts accessible within the project structure but do not show in Visual Studio.

Partially the Nuget is like it is now since that one I got to work reliably. There's plenty of quirks in the new .csproj format, even between versions, and the old file format when it come to distribute content files, native libraries and so forth. I would personally like to have the scripts accessible in within the project structure in some well-known place and not show up in the VS unless I explicitly add them. I'm not sure about the benefit of adding the files into a meta-packages, especially if they're all in there (as they've been split).

[veikkoeeva]

Discussion and options to solve also at https://github.com/dotnet/orleans/issues/5168, specifically at https://github.com/dotnet/orleans/issues/5168#issuecomment-436969615 .

[sergeybykov]

We removed metapackages including Microsoft.Orleans.OrleansSqlUtils in #5946.

What if we simply add a .md file with links to the scripts on GitHub?

[palpha]

Perhaps give the scripts their very own NuGet package, referenced from the online documentation and some strategically chosen intellisense comment? I very much doubt that anyone would attempt to set up Orleans without consulting the documentation, and if the scripts are in a NuGet package that is versioned together with the ADO.NET bits there should be no issues with getting the wrong schema for the version of Orleans in use.

I dislike the idea of putting non-functional files in people's projects. However, if my opinion about this does not represent a large part of the user base, I have no problems working around the issue -- I can delete an .md file automatically just like I delete the scripts :).

[sergeybykov]

I like the idea of a separate NuGet package.

[veikkoeeva]

I think the important thing is just to give notice the scripts are to be found easily enough. I have been looking issues like https://github.com/NuGet/Home/issues/4856, https://github.com/dotnet/project-system/issues/3302 etc. I haven't got far enough to see if those would be helpful or if the files could be moved, say, closer to .sln root or what could work. Maybe important to note there may be options here.

<edit: @palpha You are right there, something should be done. Maybe I'm in minority to research options more throughout before taking action. Albeit slowly it seems since this doesn't come up so often. Maybe due to existence of this issue that people find...

[veikkoeeva]

Would it make sense to separate the ADO.NET bits that are currently conditionally included with #ifdef also this assembly (which would make it a bit easier to refactor/update the code)?

What is the story now to get these files somehow "close to the project" so they are noticeable to include? I include this comment here and not in the PR so as not to mix things up here. Just gauging an opinion if it would be a good idea perhaps do this refactoring later now if there will be an additional Nuget package.

[sergeybykov]

What is the story now to get these files somehow "close to the project" so they are noticeable to include?

Should we add a readme.md to the main packages with information about and links to the scripts package? I have no idea of that's a practice that people use. Just a thought...

[veikkoeeva]

@sergeybykov I think we need something to tell where to find the scripts. If the scripts are moved to separate Nuget package, is it different from excluding the .target from the current package so the scripts wouldn't be visible anymore as they wouldn't be copied from the Nuget cache directory on install or restore?

From that observation comes the question too, since if a new Nuget package is created, should we also put the common code into that and just combine the separate scripts into one. As far as I'm concerned, I don't mind there are a few extra rows in Query table and a few extra tables as it's easier than separate scripts where I need to remember the run the base script first.

As for deployment options, I wrote about three options I see at https://gitter.im/dotnet/orleans?at=5da09304894dee56e525220b if it matters. The problem is that once the Nuge package of membership, reminders or persistence is downloaded, it should be somehow easy to see a script needs to be run with some suitable mention. Documentation is likely needed. Does it matter if the documentation shows in the solution explorer? If so, can it be hidden and if it can, can these script files?

Does this make sense? :) I don't oppose changes, but it looks a bit if it's not clear why take this solution as opposed to others that are available and what would follow taking other options.

[sergeybykov]

@veikkoeeva Thanks for your thoughts. I was under the assumption, maybe an invalid one, that the scripts are needed only during initial development/learning, and after that everyone morphs them into their own customer/environment-specific variations. Is that wrong to assume?

The problem is that once the Nuge package of membership, reminders or persistence is downloaded, it should be somehow easy to see a script needs to be run with some suitable mention.

I was trying to address this with my question above about including a readme.md into each of those packages with a short documentation note about scripts, how to get them, etc.

As @palpha mentioned at the very top:

I see how the files could be useful for someone just starting out with Orleans, but not for an established project. In my case, the files are possibly harmful, as I'm running on a modified schema and these files would only confuse my colleagues.

Unlike the scripts that always need to be tweaked, such a doc file would be immutable. Would a readme.md address these concerns?

[veikkoeeva]

@veikkoeeva Thanks for your thoughts. I was under the assumption, maybe an invalid one, that the scripts are needed only during initial development/learning, and after that everyone morphs them into their own customer/environment-specific variations. Is that wrong to assume?

Your understanding is correct. Maybe it is unclear to me what would happen if the files are in a Nuget package? Are they retrieved from the local Nuget cache with the exception of having a .md file showing (and polluting?) the solution explorer?

Could the solution be to add them just once when installing the Nuget the first time?

The problem is that once the Nuge package of membership, reminders or persistence is downloaded, it should be somehow easy to see a script needs to be run with some suitable mention.

I was trying to address this with my question above about including a readme.md into each of those packages with a short documentation note about scripts, how to get them, etc.

Do we a similar problem then that the readme.md will show in the solution explorer whereas it would not be wanted? If it can be deleted and is not readded during Nuget restore, can we use the same technique to add the scripts to some content folder common to all .csproj files in the solution?

As @palpha mentioned at the very top:

I see how the files could be useful for someone just starting out with Orleans, but not for an established project. In my case, the files are possibly harmful, as I'm running on a modified schema and these files would only confuse my colleagues.

Unlike the scripts that always need to be tweaked, such a doc file would be immutable. Would a readme.md address these concerns?

In OneBoxDeployment I added the scripts to DACPAC and even modified the schemas and filegroups (security, backups/disaster recorvery etc.), so in that sense the original scripts aren't neeeded anymore.

So I just put the files basically to new place and I don't need the original files. So in that sense they clutter the view.

I would appreciate a solution that creates a solution file content directory and puts the scripts there so that they don't show up in the solution directory or that if I remove them they do not reappear (maybe have a .md file to explain this). Would this be doable?

[sergeybykov]

I would appreciate a solution that creates a solution file content directory and puts the scripts there so that they don't show up in the solution directory or that if I remove them they do not reappear

I don't know how do achieve this. I also suspect this path is fraught with perils - automated builds, CI/CD, multiple dev machines, etc. I think it is safer to be simple and deterministic.

Do we a similar problem then that the readme.md will show in the solution explorer whereas it would not be wanted?

I think it's different - unlike the scripts, it's an immutable readme file. In my opinion, the choice is between including readme.md to help with discoverability and not including it, and relying on people looking a the docs.

I wonder what @palpha thinks - would readme.md in the solution be a nuisance or not.

[palpha]

First of all, thank you for having this conversation. I know it's not the most interesting or pressing issue that can be discussed :).

To reiterate my thoughts about the scripts: they are required to get started, but had it been my product I would expect the developer to incorporate the scripts, potentially modified, in their own documentation or tooling.

An appropriately named .md file might be acceptable, but I would still do my best to have it automatically deleted and .gitignored. There are other packages that drop such files, but in general they have utilized the concept of content files, which made them easy to avoid with Paket. It is not entirely clear from the documentation I can find, but it appears that content files are deprecated.

As for leaving it up to a developer to figure out from the documentation how to get the scripts: in addition to the idea of including a mention of the script package in some IntelliSense documentation a developer will likely encounter when wiring up the ADO.NET bits, the package could be promoted in a related exception thrown when the "interface" table (OrleansQuery) does not exist. Helpful and verbose exception messages guiding a developer during their struggle to grok a project is an underrated approach.

[sergeybykov]

Resolved via #6049 and #6062.