amplify env add/checkout change underlying file structure in unpredictable ways.

[eettaa]

Before opening, please confirm:

• I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
• I have searched for duplicate or closed issues.
• I have read the guide for submitting bug reports.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

v14.18.1

Amplify CLI Version

7.5.3

What operating system are you using?

Ubuntu 18.04.5 LTS

Amplify Categories

Not applicable

Amplify Commands

env

Describe the bug

I am struggling to understand what is actually expected to happen under the hood for amplify env add and amplify env checkout X. It matters to me because many of these files are version-controlled (as Amplify recommends) and therefore I need to understand what file changes should be expected in pull requests with environment interaction.

The files that seem to change do not match my understanding after reading the documentation I could find. Can someone help me understand why amplify's behavior makes sense (or confirm my thought that it does not?).

Starting case: Env 'envone' with auth, lambda (one function called functionone), api enabled. envone is pushed to cloud and re-pulled, amplify status says 'no update'

"Bug" one: amplify env add envtwo

Expected: #current-cloud-backend/ is updated to be almost empty (no categories enabled). backend/ is unchanged. Amplify status says "create" for auth, api, lambda.
Justification: creating a new environment is not meant to update the local backend in backend/. The new environment has no categories enabled, so #current-cloud-backend/ should not have categories. Status shows the difference between these two states.
Actual: #current-cloud-backend/ retains all the categories from envone. The two amplify-meta.json files (one in #current-cloud-backend/ and one in backend/) are updated. Amplify status says "create api, auth, functionone"

... Next, let's remove our local changes:

"Bug" two: amplify env pull --restore

Expected: all backend/ categories are deleted. amplify status now has no entries.
Justification: there should be no backend categories in #current-cloud-backend, and this should be copied over into backend/
Actual: amplify status as expected. but backend/ retains all the categories from envone, just as #current-cloud-backend/ retained the categories while adding the new environment.

If you then run amplify push, you get a message saying "this is up to date" i.e. no push is needed. No underlying files change.

... Next, let's add a new functiontwo to envtwo:
amplify add function .... // defaults are fine, name the function functiontwo
amplify status runs as expected: functiontwo designated as 'create'
amplify push // seems to work as expected. I note that after this stage, #current-cloud-backend/ has a function/functiontwo/ directory. This is strange as the previous amplify push on the fresh environment did not update (remove) #current-cloud-backend/ directories...

.. Next, let's switch back to envone:

"Bug" 3: amplify env checkout envone

Expected: #current-cloud-backend/ is modified to remove functiontwo. I would expect it to add back in the three categories that might have been removed as part of "bug" 1, but since they weren't, no change there. No backend/ change. amplify status shows: functiontwo: create, functionone: delete, api: delete, auth: delete.
Justification: envtwo has only functiontwo. backend/ doesn't change with amplify env checkout. therefore, the difference should be that functiontwo would be created and the other components would be deleted.
Actual: #current-cloud-backend/ is modified to remove functiontwo. Amplify status says functiontwo: Create, unctionone: no change, api: no change, auth: no change.

What is especially confusing to me is the apparent discrepancy between "bugs" 1 and 3, where adding an empty environment does not remove #current-cloud-backend/ contents but checking out envone again does remove #current-cloud-backend/functiontwo/ directories.

I assume that the above behavior is intentional, that my perceived "bugs" are not really bugs, and my understanding of expected #current-cloud-backend/ and backend/ changes is incorrect somehow. However, I cannot find any documentation on file changes for these commands. Can anyone offer insight?

Expected behavior

(as described above)

Reproduction steps

(as described above)

GraphQL schema(s)

(irrelevant, this behavior would occur with only e.g. lambda functions enabled)

Log output

No response

Additional information

No response

[jhockett]

Hey @eettaa,

It matters to me because many of these files are version-controlled (as Amplify recommends) and therefore I need to understand what file changes should be expected in pull requests with environment interaction.

The #current-cloud-backend folder is recommended to be in .gitignore

[eettaa]

Hi @jhockett, thanks so much for your response!

Yes, totally acknowledge that #current-cloud-backend/ (and the amplify-meta.json files) are meant to be .gitignore'ed. But, many files that I would expect to be updated in the backend/ directory are supposed to be VC'ed. See, for example, my stated (admittedly perhaps confused) expectations for "Bug 2" above, where I'd naively expect the backend/ directory to be wiped.

The #current-cloud-backend folder is recommended to be in .gitignore

Does this mean Amplify makes no promises about what's going on in these directories i.e. "as a user you shouldn't try to understand this?" Even if I don't need to worry about it for VC reasons, I'm still sort of curious to understand Amplify at a deeper level. But I understand if you don't have time to answer questions that don't pose immediate problems to your users.

Thanks Again!

[jhockett]

@eettaa apologies for taking so long to get back to you initially and thanks for clarifying what you were asking.

The reason amplify/backend/ is not wiped, is because we want to give customers the opportunity to clone their existing environment. This is similar to creating a new git branch from an existing branch.

If you want a blank environment, you would need to run amplify remove <category> to remove the resources after amplify env add, but if that's the case, perhaps a new project is better suited to what you're trying to accomplish?

[eettaa]

Hi @jhockett, thanks for the extra info.

The reason amplify/backend/ is not wiped, is because we want to give customers the opportunity to clone their existing environment. This is similar to creating a new git branch from an existing branch.

This makes sense to me when a new environment is created (and it matches my expectation as stated in "bug" 1). But what I do not understand is why backend/ is not updated when amplify env pull --restore is run. I am envisioning a flow like this:

• Developer is on some random git branch with some custom amplify environment
• git checkout prod // Dev switches to shared, canonical branch, which also has an associated amplify environment
• amplify env checkout prod // Switch to prod amplify environment- I'd expect #current-cloud-backend/ to be updated.
• amplify env pull --restore // Shouldn't this fully update the backend/ directory to match prod??

Current behavior at the end of this flow seems to be that git status will show backend/ as having a combination of folders from both environments. But I would instead hope it only contains the contents of the prod amplify environment. As-is, it would be bad if the dev then commits the backend/ directory back to the prod branch, because AFAICT it is still tainted with remnants of their custom amplify environment.

The fact that in "bug" 2 I would expect backend/ to be largely wiped is incidental- what I should really say is that I'd expect it to exactly match the environment state of the pulled environment (which in that case should have been empty I think, as no categories have been added*).

* In reality, #current-cloud-backend/ was not empty (see 'Actual' results in "Bug" 1), but I don't understand that behavior, either.

Thanks for helping me understand these intricacies- my team is excited to be learning about Amplify

[jhockett]

#current-cloud-backend is a folder we use internally to Amplify CLI, but it isn't intended for developers to consume. amplify status [--verbose] should be considered your source of truth for the current state of the environment.

amplify env pull --restore // Shouldn't this fully update the backend/ directory to match prod??

This sounds correct to me given the steps you laid out. Can you share the git status of the changes so we can see which files are not in sync?

Does amplify pull --restore(no env) yield a different result?

[eettaa]

@jhockett thanks so much for your comment.
Sure, let me get you the status dumps for the first post. I apologize in advance for the length of the dump, but hopefully it will help us understand what's going on.

In order to be extra explicit, I've removed the amplify files from my .gitignore. I do understand some of them are meant to be hidden, notably #current-cloud-backend/ and amplify-meta.json, and that our larger conversation is focused on just the files that aren't excluded normally (notably backend/ contents). But maybe the extra changes will help us identify a pattern/understand amplify in totality.

Please note that this is exactly the process I used when formulating my initial question, so you'll see these results map exactly onto my original post (except that I do not include "Bug" 3 here as we can already identify problems at 2).

Starting Case (before bug one):

$ git status
nothing to commit, working tree clean

$ amplify status --verbose

    Current Environment: envone
    
┌──────────┬────────────────────────┬───────────┬───────────────────┐
│ Category │ Resource name          │ Operation │ Provider plugin   │
├──────────┼────────────────────────┼───────────┼───────────────────┤
│ Api      │ apiname         │ No Change │ awscloudformation │
├──────────┼────────────────────────┼───────────┼───────────────────┤
│ Function │ helloworld             │ No Change │ awscloudformation │
├──────────┼────────────────────────┼───────────┼───────────────────┤
│ Auth     │ authname │ No Change │ awscloudformation │
└──────────┴────────────────────────┴───────────┴───────────────────┘

---

After amplify env add envtwo :

$ amplify status
    Current Environment: envtwo
    
┌──────────┬────────────────────────┬───────────┬───────────────────┐
│ Category │ Resource name          │ Operation │ Provider plugin   │
├──────────┼────────────────────────┼───────────┼───────────────────┤
│ Api      │ apiname         │ Create    │ awscloudformation │
├──────────┼────────────────────────┼───────────┼───────────────────┤
│ Function │ helloworld             │ Create    │ awscloudformation │
├──────────┼────────────────────────┼───────────┼───────────────────┤
│ Auth     │ authname │ Create    │ awscloudformation │
└──────────┴────────────────────────┴───────────┴───────────────────┘

$ git status

	modified:   amplify/#current-cloud-backend/amplify-meta.json
	modified:   amplify/#current-cloud-backend/awscloudformation/build/root-cloudformation-stack.json
	modified:   amplify/.config/local-aws-info.json
	modified:   amplify/.config/local-env-info.json
	modified:   amplify/backend/amplify-meta.json
	modified:   amplify/backend/awscloudformation/build/root-cloudformation-stack.json
	modified:   amplify/team-provider-info.json

Note in particular that for some reason #current-cloud-backend/ still retains the api, function and auth categories from envone, even though my understanding is that we have just "switched" to a new, blank environment. To me this is unexpected (even if users shouldn't worry about this as it is .gitignore'd

$ ls amplify/#current-cloud-backend
amplify-meta.json  api  auth  awscloudformation  backend-config.json  function  tags.json

For good measure, we can also see that backend/ retains these directories. We both agree this is expected:

The reason amplify/backend/ is not wiped, is because we want to give customers the opportunity to clone their existing environment. This is similar to creating a new git branch from an existing branch.

---

Moving on to "Bug" 2, which you asked about in your most recent comment:

$ amplify env pull --restore

$ amplify status
    Current Environment: envtwo
    
┌──────────┬───────────────┬───────────┬─────────────────┐
│ Category │ Resource name │ Operation │ Provider plugin │
└──────────┴───────────────┴───────────┴─────────────────┘

$ git status
	modified:   amplify/#current-cloud-backend/amplify-meta.json
	modified:   amplify/#current-cloud-backend/awscloudformation/build/root-cloudformation-stack.json
	modified:   amplify/.config/local-aws-info.json
	modified:   amplify/.config/local-env-info.json
	modified:   amplify/backend/amplify-meta.json
	modified:   amplify/backend/awscloudformation/build/root-cloudformation-stack.json
	modified:   amplify/backend/types/amplify-dependent-resources-ref.d.ts
	modified:   amplify/team-provider-info.json

$ ls amplify/backend
amplify-meta.json  api  auth  awscloudformation  backend-config.json  function  tags.json  types

$ ls amplify/#current-cloud-backend
amplify-meta.json  api  auth  awscloudformation  backend-config.json  function  tags.json

Given the above dump and our (shared) expectations from the previous posts:

amplify env pull --restore // Shouldn't this fully update the backend/ directory to match prod??

This sounds correct to me given the steps you laid out. Can you share the git status of the changes so we can see which files are not in sync?

I think we can conclude that these operations do not match our expectations?

I did subsequently run amplify pull --restore (no 'env'). No difference in results.

My observations:

• amplify status always yields the expected result. AFAICT amplify status is powered by diffing the two amplify-meta.json files, which we can see are also appropriately updated through these steps.
• It seems like amplify is "lazily removing" the backend categories or something- they are removed the from the cloudformation stack files and the amplify-meta.json, so they wouldn't be pushed back to the cloud or show up in amplify status.
• ... but, they are not actually purged from the local filesystem, meaning that they ARE retained by git. I don't understand why this would be desired and instead believe it is a bug (but defer to the experts).

Thank you for your continued troubleshooting.

[jhockett]

@eettaa thanks so much for the detailed write-up, this is super useful! I will share with the team and get back to you.

[eettaa]

@jhockett Great, thank you! Please do let me know what they say as this is blocking our (mult-dev) org from adopting amplify environments.
The thought crossed my mind that perhaps amplify was deleting files but not the directories (leading to misleading ls results), but I verified that this also isn't the case- it does indeed seem like amplify env pull --restore is truly not cleaning the backend/ directory as we both would expect.
Thanks!

[eettaa]

@danielleadams happy new year, just want to circle back here in case there was any new guidance.
thank you for your team's continued support!

[danielleadams]

@eettaa apologies for the late response here - is this still an issue for you?

[eettaa]

Hi @danielleadams, thanks for circling back!
Yes, as far as I can tell this is still an unresolved problem (or I just don't understand what Amplify is doing but it is actually works as intended?).

Would love any insight the amplify team can provide :)

[josefaidt]

Hey @eettaa 👋 apologies for the delay here! Thank you for including the details of the issues you're experiencing with the multi-environment workflow. Currently this workflow relies heavily on git, and as such the CLI will not retain state of the filesystem between environments. Some of these issues are documented separately as bugs/features to ultimately improve the multi-env workflow

• New created Lambda is not recognized in the next Env #10592
• amplify env checkout causes local changes to lambda layer #8216
• Merging new api and lambda layer between branches does not show resources in amplify status. #9790
• Consistent and extreme amount of git conflicts on merge #7938
• Amplify does not recognize function created in a separate git branch #8090
• team-provider-info.json shouldn't be treated as a canonical source of environment information. #6298
• Feature Request: More Robust Multienv push #594

I'll close this issue for now in favor of tracking the existing items noted above, however if you feel these issues do not quite cover the case you're actively experiencing, please reply back to this thread or file a new issue.