RFC: Combine and organize metadata file locations across lifecycle

[sambhav]

Readable

It's my first time creating an rfc so not sure if I did everything right here.

[sambhav]

I updated the rfc with the sub-directory proposal instead of the prefix one as discussed in the last WG meeting.

[jromero]

I updated the rfc with the sub-directory proposal instead of the prefix one as discussed in the last WG meeting.

This makes it sound like the sub-directory structure would be the primary idea but I'm still seeing prefixes as the primary proposal. Am I missing something?

[sambhav]

I updated the rfc with the sub-directory proposal instead of the prefix one as discussed in the last WG meeting.

This makes it sound like the sub-directory structure would be the primary idea but I'm still seeing prefixes as the primary proposal. Am I missing something?

Urgh - I think I forgot to push. Should be fixed now.

[jromero]

Really like the directory restructure. The term buildpack-workspace is a bit of a mouthful so I may bikeshed on it a bit but it totally makes sense IMO.

[jabrown85]

I like the structure change.

[natalieparellano]

Really like the directory restructure. The term buildpack-workspace is a bit of a mouthful so I may bikeshed on it a bit but it totally makes sense IMO.

buildspace?

[ekcasey]

Capturing a discussion from WG, Stephen proposed:

1. Moving "top-level" files to a different "config" directory so that the layers dir provided to the buildpack is exclusively for layers and their metadata.
2. Providing the location of both the layers and config directories as environment variables, rather than positional arguments

Elaborating on this idea and digging the details we would need to work out:

current directory structure:

`/layers` (configurable with `CNB_LAYERS_DIR` or -layers) 
 ├── group.toml
 ├── plan.toml
 └── <escaped_buildpack_id> (buildpack arg `$1` )
    ├── build.toml
    ├── launch.toml
    ├── store.toml
    ├── <layer>.toml
    └── <layer>
  └── config
    └── metadata.toml

Assuming we want to reusing the existing layers volume for the new files instead of introducing a new required volume, I think these are the possible future directory structures?
Option 1:

`/layers` (configurable with `CNB_LAYERS_DIR` or -layers) 
 ├── group.toml
 ├── plan.toml
 └── <escaped_buildpack_id> (`CNB_BP_LAYERS_DIR` )
    ├── <layer>.toml
    └── <layer>
  └── config
     └── metadata.toml
     └── <escaped_buildpack_id> (`CNB_BP_CONFIG_DIR` )
          ├── build.toml
          ├── launch.toml
          └── store.toml

pros:

• doesn't break layer reuse
• same-length for layer path names
  cons:
• calling the top-level layers even though it is layers + config

Option 2:

`/build` (configurable with `CNB_BUILD_DIR` or -build-dir) 
 ├── group.toml
 ├── plan.toml
 └── l ( or `layers?`)
      └── <escaped_buildpack_id> (`CNB_BP_LAYERS_DIR` )
          ├── <layer>.toml
          └── <layer>
 └── c ( or `config`?)
      ├── metadata.toml
      └── <escaped_buildpack_id> (`CNB_BP_CONFIG_DIR`  )
          ├── build.toml
          ├── launch.toml
          └── store.toml

pros:

• doesn't incorrectly call the entire top level directory layers (it's really too bad we used workspace for the app dir).
  cons:
• breaks layer reuse
• layers/layers? layers/l? (both are bad, one is shorter)
• longer path names
• right now we export layers/config, if we implement this before eliminating metadata.toml we would need to partially export /layers/config.

In the above examples names of specific directories or env vars could change. E.g:

1. Do we want to reuse CNB_LAYERS_DIR to mean something different in the buildpack context instead of a creating a new CNB_BP_LAYERS_DIR?
2. build is just a placeholder. workspace actually seems the most logical to me but we already used it for the app dir.

But I think the overall structures represent the two obvious options.

Other possibilities that I won't take as much time to elaborate on:

1. add a layers and config dir under each buildpack ID
2. unite layers and the app dir in a single mega workspace (e.g. workspace/app and workspace/<bp_id> or workspace/config, and workspace/layers) -- this one would be nice but it would also require a lot of changes...

[jkutner]

I'm +1 on @ekcasey's "option 1". I'm also fine with /layers/config. I don't think it's exceptionally weird.

[sclevine]

Bikeshed

[sclevine]

Since CNB_BP_LAYERS_DIR is used more often than CNB_LAYERS_DIR, could we pick a shorter name for the former?

Maybe something like:

• CNB_ALL_LAYERS_DIR for /workspace/layers
• CNB_ALL_CONFIG_DIR for /workspace/layers-config
• CNB_LAYERS_DIR for /workspace/layers/<buildpack-id>
• CNB_CONFIG_DIR for /workspace/layers-config/<buildpack-id>?

[jabrown85]

I have a minor concern with using CNB_LAYERS_DIR for CNB_BP_LAYERS_DIR since the former is already a config var for /layers. We would be moving CNB_LAYERS_DIR from a platform env var to a buildpack env var, which feels a bit weird.

[sclevine]

Ah, good point. Maybe CNB_ALL_LAYER_DIR and CNB_LAYER_DIR?

[natalieparellano]

I still like CNB_BP_LAYERS_DIR for its obviousness.

[ekcasey]

Right now the buildpack's layers dir is provided to it as a positional arg

/bin/build <layers[EIC]> <platform[AR]> <plan[ER]>, Working Dir: <app[AI]>

Are we proposing replacing all positional args in the buildpack API with environment variables as part of that RFC? J I think that is a good idea but it's a big enough change that I think we should call this out explicitly in a separate section and provide the names for the env vars that will replace the plan and platform arguments as well.

[ekcasey]

If we go with CNB_BP_LAYERS_DIR should we switch from CNB_BUILDPACK_DIR to CNB_BP_DIR for consistency?

[jabrown85]

I decided to change them to _BUILDPACK_ to match CNB_BUILDPACK_DIR. They are long, but explicit and match our current standards that buildpacks are already built against.

[jabrown85]

@ekcasey I would prefer to not tackle the positional argument changes in this RFC.

[ekcasey]

If we aren't switching from position arguments to environment variables then I am unsure what CNB_BUILDPACK_LAYERS_DIR is referring to above. The layers dir is currently a positional argument and nobody would be setting or reading this environment variable.

[ekcasey]

Also, if we are introducing at least one environment variable CNB_BUILDPACK_CONFIG_DIR it does seem to me like an appropriate time to convert the rest (just for the sake of internal consistency).

[natalieparellano]

For new buildpacks running on the old platform, do we set CNB_BP_CONFIG_DIR equal to CNB_BP_LAYERS_DIR?

[jabrown85]

I think we would need to.

[ekcasey]

If we go with CNB_BP_LAYERS_DIR should we switch from CNB_BUILDPACK_DIR to CNB_BP_DIR for consistency?

[natalieparellano]

Would this make sense to go in the platform directory? In talking through buildpacks/pack#1221 it seems like it could be helpful for the platform to mount a directory where it could provide configuration and receive output.

[natalieparellano]

Should this be a child of config?

[hone]

Putting on the backburner for now due to other active RFCs.

[buildpack-bot]

Maintainers,

As you review this RFC please queue up issues to be created using the following commands:

/queue-issue <repo> "<title>" [labels]...
/unqueue-issue <uid>

Issues

(none)

[hone]

@natalieparellano suggested of pairing this with deprecation of other APIs.

[edmorley]

I would love to change the default location for the app from /workspace to /app. Could we incorporate that change into this RFC too?