Msbuild: Proposal: DependsOnTask for project items Include

[ghost]

For Problem statement and Part 1, see https://github.com/Microsoft/msbuild/issues/3219

Part 2:

<Compile
  Include="$(MSBuildProjectDirectory)\SharedDTOs\RemoteFile.vb"
  DependsOnTask="DownloadMySourceFile"
  Condition="!Exist($(MSBuildProjectDirectory)\SharedDTOs\RemoteFile.vb)" />

this will indicate the design-time or first-evaluation-time to run the task before evaluating this line, ignore the task result and continue evaluation once task is finished. If task fails (due to insufficient credentials / no internet connection / wrong URL / any other reason), a reason warning will be issued and subsequent steps will fail as they fail today:

CSC : error CS2001: Source file 'C:\testprojnonexisting.cs' could not be found.

for <Compile> and for <Content>

error : File not found: 'C:\testproj\mockimage.png'

etc.

[rainersigwald]

I'm not so sure about this. Would it apply to evaluation-time (as opposed to created-during-execution) items? That is, is it legal to have

<Project>
  <ItemGroup>
    <I Include="foo.bar" DependsOnTask="CreateFoo" />
  </ItemGroup>
</Project>

? What about

<Project InitialTargets="IncludeFoo">
  <Target Name="IncludeFoo">
    <ItemGroup>
      <I Include="foo.bar" DependsOnTask="CreateFoo" />
    </ItemGroup>
  </Target>
</Project>

? How does the execution differ?

Do you want to call a task or a target with this? If a task, how would you pass arguments to the task, or would it be a requirement that the task not take any arguments?

[ghost]

I was thinking about calling it a Task, as in a predefined task with arguments set. If we have multiple tasks with same name then one completion would suffice.

How does the execution differ?

One difference in context of VS is, if <I> == <Content>, then VS will not display the file in Solution Explorer.

[dasMulli]

I'm not sure that downloading / "heavy logic" is good during evaluation or even design-time builds. (both for performance and determinism due to network or other logic).
Ideally, the download should hook into the Restore target.

A few different things could be done here to hook some targets into restore based on some conventions involving metadata on items.
e.g. (simplified)

<ItemGroup>
  <Content Include="CompanyLogo.png" DownloadSourceIfMissing="https://my.company.com/logo.png" />
</ItemGroup>

…
<Target Name="DownloadMissingContent" BeforeTargets="some-restore-hook">
    <DownloadFile Source="%(Content.DownloadSourceIfMissing)" 
         Target="%(Content.Identity)"
         Condition="!Exists%(Content.Identity)) and '%(Content.DownloadSourceIfMissing)'" />
</Target>

Might be a good candidate for an MSBuild SDK and see how many ppl need it.

[CZEMacLeod]

I think you might need some kind of metabase to store the etag, last modified, cache control settings of the downloaded file. Also control over when the file is updated - some kind of max query rate control that can be applied by the consumer. If this is just to restore missing files, that is one thing, but if it is to download and keep a file up to date that is another...
Is it applicable to have DownloadFiles as an ItemGroup and maybe make it visible separately in the UI (e.g. Visual Studio) with a property page to set the metadata.
I think the downloaded files would be stored in the IntermediateOutputFiles path or similar...

<ItemGroup>
    <DownloadFile Include="https://my.company.com/logo.png">
        <TargetGroup>Content</TargetGroup>
        <MaxUpdate>86400</MaxUpdate>
        <Pack>true</Pack>
        <PackagePath>content\images\</PackagePath>
    </DownloadFile>
</ItemGroup>

If using that approach, then I think you could make the DownloadFiles target just be BeforeTargets="Build"?

[ghost]

Good points @dasMulli and @CZEMacLeod. I like the overall approach and line of reasoning, except for <Pack>true</Pack> bit; that would make this task too specific in terms of scope IMHO.

The idea is to download the file separately and use the existing semantics for build inclusion/exclusion <Content>, <Compile> to include it in build and packaging targets. While DownloadFile can still be used to download something that is not related to build target (like download a .sig file and verify signature or md5 checksum in deterministic build or download a file called whitelist.txt that will package only files listed in it and filter out the rest etc. etc.).

Separate context but related:

In Visual Studio project system, design time, it would be nice to introduce this concept of "deferred" or "to be downloaded" or (what they call in C/C++ paradigm) "delayed linking/inclusion" content, such that the Solution Explorer and related artifacts don't appear curious and unhappy with exclamation sign when the file is not (yet) present, and react accordingly when DownloadFile fails.

[CZEMacLeod]

@kasper3 The Pack, and PackagePath properties were there as an example to indicate that all metadata was copied to the appropriate TargetGroup ItemGroup in the DownloadFiles target.
I was thinking something like a resolve downloads, where the filename would be inferred from the url unless it was specified in via content-disposition etc.
This would probably act a little like the Microsoft.NuGet.targets ResolveNuGetPackageAssets target.

In the project UI, deferred, or remote items should be able to be downloaded, refreshed or force refreshed.