Spring 19: "shane:mdapi:pull --all" now missing some MD types

[camisotro]

Last Monday I did a "shane:mdapi:pull" of a production org and committed to a Git repo.
Today I did the same and noticed that some folders originally present are missing, including:

• applications
• profiles
• standardValueSets
• workflows
• homePageComponents
  Did something change in the API version that makes it no longer possible to pull these, or is this a version incompatibility that can be remedied?

[mshanemc]

Are you running the current version? And did you “downgrade” sfdx from prerel back to latest if you were using prerel?

…

Sent from my iPhone

On Feb 11, 2019, at 1:43 PM, camisotro ***@***.***> wrote: Last Monday I did a "shane:mdapi:pull" of a production org and committed to a Git repo. Today I did the same and noticed that some folders originally present are missing, including: applications profiles standardValueSets workflows homePageComponents Did something change in the API version that makes it no longer possible to pull these, or is this a version incompatibility that can be remedied? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[camisotro]

I updated the plugin last week before the pull. I checked the repo this morning and found no changes since the last pull so didn't try to update.

The only change I can think of is that the org was Winter 18 last week and is Spring 18 this week.

[camisotro]

Just wanted to add, I've tried this weekly against a large org. Sometimes I get everything. Sometimes a random scattering of MD components are missing.

I then tried pulling the remaining ones using "--type=ApexComponent" etc. But I tried doing so in multiple parallel command prompts to speed up the process. I noticed that it didn't operate very nicely in parallel, maybe because one pull was cleaning up the temp package directory before the other had finished the convert command or such?

Anyway it made me wonder if the massive operation of "--all" could be failing due to some steps executing out of sequence. The command line does seem to correctly report the names of all the components it attempted to pull, and it does generate a temp package.xml for each one of them during retrieval. But then some fail to actually get converted into the force-app folder. It doesn't help that "--all" in parallel results in my CPU & Memory maxing out as all the little node.js sub-processes spiral out of control. It's not like I'm using a low-power computer either. I have seen this both on my work laptop and my home computer which are both Core i7 machines.

Aside from finding the root cause, one or both of these might help the end-user take some more control over the situation:

• Allow the command-line to specify max number of simultaneous processes that can be launched during retrieval.
• Provide a command that takes the same possible arguments as shane:mdapi:pull but simply generates one package.xml from all the items found but does not retrieve them. (I can then use force:source:retrieve or force:mdapi:retrieve or Workbench/ANT/etc with it.)

[gbreavin]

I'm seeing the same behaviour. I'm seeing types reported as being successfully retrieved, but not actually being pulled. This isn't just on --all, but on other switches as well. For example, I tried -s (as I was missing custom objects), and ExternalDataSource wasn't retrieved. In fact, running with -t ExternalDataSource is reporting success but not actually fetching anything (despite their being an External Data Source in my org).

Is there anything I can do to help poke around what the issue might be?

[camisotro]

As I mentioned above, it does seem that each individual retrieval lands in pkgTempDir, but sporadically some of them fail to be converted into DX and merged in. I wonder if maybe some of the retrieval processes are accidentally deleting the entirety of pkgTempDir, before the other processes have finished their conversions?

[mshanemc]

Could be...Will have to explore what’s happening. I’m trying to distribute the process by mdtype (for limits) and then parallelize as much as possible. I’m not sure that the sfdx source retrieve works properly on all types. What do you seem to be missing consistently?

…

Sent from my iPhone

On Jun 15, 2019, at 10:01 AM, camisotro ***@***.***> wrote: As I mentioned above, it does seem that each individual retrieval lands in pkgTempDir, but sporadically some of them fail to be converted into DX and merged in. I wonder if maybe some of the retrieval processes are accidentally deleting the entirety of pkgTempDir, before the other processes have finished their conversions? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[camisotro]

It's never consistent. It's just that if I run a retrieve that includes many types (especially "all"), some types at random will fail to extract.

[gbreavin]

@mshanemc I think I have found other ways to achieve what I'm trying to do, so probably won't need this command.

Thanks for all your work on this plugin - so many useful commands!

[ghost]

Hi @mshanemc,

First a fall, I like to thank you for creating this useful plugin which resolve a lot of problem. I'm totally new to this salesforce area hence doesn't know about this much and learning continuously. Before discuss about my issues, I like to pull
@gbreavin / @camisotro also here to get some suggestion.
My business requirement is to bring all prod org source code (more than 700mb probably bit up and down) into bitbucket repository and use same to deploy into destination org. Getting metadata (code, config,…) into a developer sandbox is no problem at all. Salesforce handles that. Getting record data is a challenge for us now. the ultimate aim is to get all source including record data into git. secondly, i like to know how do I retrieve all metadata types like apexclass, custom field etc whichever linked to profiles. So that I come to know complete metadata types of profile and use same to retrieve from org using package.xml file and store in git repo as complete profile.

Any help on this case is really appreciable.

[ghost]

on top of that I'm getting below issue when i try to run --all pull command towards my test org

$ sfdx shane:mdapi:pull --all -u sf-test-org
ERROR running shane:mdapi:pull: Command failed: sfdx force:source:retrieve -x pkgDirTemp/CustomIndex/package.xml -w 30 -u [email protected] --json

asking your org about its metadata... done

I'm not sure, what issue this will bring up and how to resolve it. Can you post your comments

[ghost]

team Any thoughts ?