Cli: "Retrieve Source From Org" doesn't update originating source file

[ChuckJonas]

Summary

"Retrieve Source From Org" doesn't update originating source file when run from a single file outside the default directory.

It works as expected when refreshed from manifest.

Steps To Reproduce:

1. > create project with manifest
2. attach to some sandbox with apex code
3. clear everything but apex classes from package.xml
4. Right click on manifest, > retrieve source from org
5. Create the following directory structure: force-app/other/classes
6. Move all the files from force-app/main/default/classes to force-app/other/classes. (Optionally delete the force-app/main/default/classes folder when done)
7. Right click on manifest, > retrieve source from org

Expected: the second time the source is retrieved (step 7), the files under force-app/other/classes will have been updated.

Result: ✅

---

Retrieve Single File

1. navigate to any .cls class under force-app/other/classes
2. right click -> retrieve source from org

Expected: The file you ran the command against on would be refreshed. EG: force-app/other/classes/Foo.cls

Result: ❌
The retrieved code will instead be placed in the default project directory: force-app/main/default/classes

Additional information

LOG:

=== Retrieved Source
FULL NAME               TYPE       PROJECT PATH
──────────────────────  ─────────  ───────────────────────────────────────────────────────────
ABTestingTriggerHelper  ApexClass  force-app/other/classes/ABTestingTriggerHelper.cls
ABTestingTriggerHelper  ApexClass  force-app/other/classes/ABTestingTriggerHelper.cls-meta.xml
23:12:47.72 sfdx force:source:retrieve --sourcepath /Users/jamesbond/Documents/clients/bug-repo/force-app/other/classes/ABTestingTriggerHelper.cls ended with exit code 0

VS Code Version: Version 1.33.1 (1.33.1)

SFDX CLI Version: sfdx-cli/7.5.0-e6ca93eee8 darwin-x64 node-v10.15.3

OS and version: osx 10.14.4

[douglascayers]

Seems related to https://github.com/forcedotcom/cli/issues/45

[clairebianchi]

@ChuckJonas thank you for filing this, it is a bug. Currently, the CLI does not support multiple package directories in the same SFDX project which is related to the issue you are seeing. I have added this issue to our backlog and hope that we can address it sometime this summer.

[justgrumpy]

@clairebianchi, I don't believe your statement is entirely accurate. If you look at #45, the CLI does support multiple package directories, but not all parameters work the same. Behind the scenes the extension is using force:source:retrieve with the --sourcepath SOURCEPATH parameter. This is exactly the same issue as #45.

[ChuckJonas]

This example is only working with a single PackageDirectory but with with multiple "modules" paths inside that package directory.

If you actually follow the reproduction steps, this will be clear, as we do not modify the standard sfdx-project.json.

{
  "packageDirectories": [
    {
      "path": "force-app",
      "default": true
    }
  ],
  "namespace": "",
  "sfdcLoginUrl": "https://login.salesforce.com",
  "sourceApiVersion": "45.0"
}

And this isn't supported, then whats the point of having the ridiculously long default structure of /main/default/ if you can't work with code in other sibling directories?

[ChuckJonas]

Any chance this will be addressed? It's a major pain-point

[ralphcallaway]

@clairebianchi for what it's worth when you create a new piece of metadata you get to choose the page directory, so seems like the concept is there, just got missed in the design. hopefully the team sees this a bug and not an improvement. can't see how it would make sense to refresh a file and have the contents end up in a separate location

[ralphcallaway]

this pretty much means people can't use any directory structure other than main/default which was one of the major goals of salesforcedx

[rendall]

This is a bump & upvote.

[amorek]

Is it on roadmap?

[jonathanwiesel]

Any news on this?

[clairebianchi]

Hi all, yes, this is on the roadmap. We are currently working on some refactoring that would allow multiple package directory support which should address this as well. If it does not then we will pick this up and work on it as a separate item in the coming months.

[RaithZx]

This feature could be very useful!

[shetzel]

It's absolutely useful! We are expecting to publish a canary release sometime today with source:deploy and source:retrieve support for multiple package directories (mpd). Please watch for an announcement on this repo very soon.

[ChuckJonas]

@shetzel by multiple package directories do you mean that it will work only when there are multiple packages defined in the sfdx-project.json?

While that is a nice feature, this issue is about the fact that you can only effectively use default/main because if you add MD to other locations, it gets duplicated in default/main when you run source:retrieve.

We should have the ability to use multiple folders, WITHOUT having to create separate packages.

[shetzel]

@ChuckJonas - great question/callout! Sorry, I wasn't clear in my post. As part of the mpd support for source:deploy and source:retrieve (and push/pull for that matter) the file mapping was changed so that this usecase should now be supported whether multiple package directories are used or not. You're absolutely right that this usecase isn't about mpd but about proper file mapping.

If the canary release doesn't fix this usecase please mention me and I'll make sure it does. This should definitely work and I'm sorry it's taken this long to get it fixed.

[ChuckJonas]

@shetzel thanks for clarifying! Super excited to try it out.

This will be a massive game changer for us 💪

[shetzel]

@ChuckJonas @RaithZx et. al - if you could give the latest salesforce-alm@mpd canary release a try with your source:deploy and source:retrieve usecases we would greatly appreciate any feedback. It's an initial, early version that will probably have a few issues but hopefully nothing huge. Thanks for your patience with this!

sfdx plugins:install salesforce-alm@mpd

This should install version 49.0.20-5

[ChuckJonas]

@shetzel just rang through my reproduction steps in the original post and it friggin work as expected now!

🙇 Finally, we can start using different folder structures

[shetzel]

@ChuckJonas - awesome! Thanks for the validation. Please note that installing the salesforce-alm@mpd plugin overwrites the default core plugin of the same name so this is just a temporary solution until that code is promoted to a patch release and part of the core plugin.

I'm hopeful this will be part of an upcoming patch release. It all depends on testing efforts over the next week or 2.

[ChuckJonas]

@shetzel FTI,

With 49.0.20-5, I'm having issues with the sfdx force:source:retrieve -x manifest/package.xml command. It seems to just respond without actually updating/retrieving anything.

Not related to this issue, but wasn't sure where I should report it.

[shetzel]

Tagging me and reporting here works. Thanks! I'll have a look and fix asap.

[shetzel]

@ChuckJonas - I can't repro this. Would you happen to have repro steps? BTW, we should be announcing an updated mpd release more publicly and if that goes well this will go into a patch release soon.

[ChuckJonas]

@shetzel Weird, I just reinstalled it and can no longer reproduce... Must have been doing something dumb 🤷‍♂️