Deprecate "select latest by build time" for image automation

[squaremo]

Flux v1 goes to some trouble to download metadata and extract the build time for each image, then (by default) uses that to order the images and select the most recent. This is an unreliable* and expensive feature: it requires complicated caching infrastructure to keep accurate image metadata locally, since requesting it every time it's needed is not viable. Getting each image manifest and config file needs a fetch per image, rather than a fetch per repo. Aggressive rate limiting by DockerHub means that it's now difficult to keep an acceptably current cache for image repos hosted there (without a paid account).

(*Why unreliable? It doesn't always pick the image you expect, since a "new" image can refer to existing image layers. Using labels helps with this, since it forces new images to have a distinct config layer.)

An alternative is to put the build time in the image tag, and sort alphabetically. If you can choose the labels used for an image, you can likely choose the tags; and if not, you can at least re-tag images into another repo where you do have that control.

Given a reasonable alternative, and having developed the caching for Flux v1, I am of the opinion that it's better to avoid the complexity and rate limiting of fetching manifests, and only deal with sorting and selecting tags.

[kingdonb]

Does the image tag have its own create time attribute, or does the proposal work without that? Would an image tag have to be capable of being sorted by another attribute, like string sort or semver order, for this proposal to work?

I have noticed that Flux v2 docs in places favor the use of build number over the "8-char short commit sha" which I have grown fond of, but unfortunately short hash does not have any property of string sortability; if this shift has been done so that a proposal like this one can take advantage of innate string (or eg. calver/semver) sort order then it helps show the use of this proposal, definitely make sense to me.

[squaremo]

Does the image tag have its own create time attribute, or does the proposal work without that?

No to the first bit: in the image registry API tags are like pointers, rather than values -- they don't have any data attached to them, they just point at an image manifest. The aim of the proposal is to be able to work without needing to fetch the manifest for any image, since that is subject to rate limiting (and more expensive).

Would an image tag have to be capable of being sorted by another attribute, like string sort or semver order, for this proposal to work?

Yes, exactly that.

[squaremo]

Tagging images with git-commit-sha is pretty common though, especially in the development lifecycle (we do use semver in prod mind).

Right, we do this as well (in fact <branch>-<sha1[:8]>). The proposal lets people continue putting a SHA1 in the tag -- so long as they also put something sortable there too.

nearly all the tooling out there supports something like CI_JOB_ID (I'm hoping they are sequential!)

They do, but I wouldn't bet on them being strictly sequential unless documentation says so. You may be better off using a timestamp -- down to the second might be good enough, if builds take minutes.

[nabadger]

Is the proposal saying to just rely on the ImagePolicy definition like this:

kind: ImagePolicy
spec:
  filterTags:
    pattern: '^dev-(?P<id>.*)'
    extract: '$id'
  policy:
    alphabetical:
      order: asc

i.e. you extract the $id here and use that as the sort key? This is really flexible and works well - since you need an ImagPolicy regardless, it's no extra effort really.

[kingdonb]

I still have a bad taste about this, I've never had to rely on image build processes to produce a number ordering my releases before. Build numbers are hard. My CI creates a new job for each branch, if my CI is getting nuked periodically, it's going to start counting over at 1, and that means I must also nuke my image repo contents (at least nuke any tags that match the ImagePolicy - the tags with the build numbers.) There is a new coupling introduced that tightly binds the lifecycle of the image repository and the build job. That's definitely going to cause some people problems.

The pathological response to this change from a user whose CI cannot order builds is going to be tagging their images with a yyyymmddhhmmss timestamp, (and those image tags are gonna be ugly as sin.) I don't have an acceptable solution right now but I'm going to have to keep looking for one. This change breaks a lot of workflows that were very flexible before.

Thanks @nabadger for showing concretely how this strategy can be used, I do like this approach! I just think it still has some problems.

[squaremo]

I've never had to rely on image build processes to produce a number ordering my releases before.

I wouldn't advise relying on a CI build system's build numbers unless it guarantees monotonicity. You are better off using a timestamp, though you still run the risk of builds completing out of order. Another possibility is the count of commits on the branch.

[scottrigby]

Should we keep that functionality as an option (since we have the code already), but just change the flag to false by default? Removing it altogether sounds like a breaking change?

[jacekn]

Rewriting build systems to include timestamp might not be an option in some situations. For example some users might be syncing 3rd party images and they won't be able to change the way images are tagged upstream.

I think it's also worth noting that flux v1 supports --registry-use-labels which allows it to use standard OCI labels to detect image build timestamp. If flux v2 explicitly removes this functionality I think it should be made quite clear in the roadmap. Currently Flux image update feature parity says "100%" which is incorrect.

Do the maintainers think OCI annotation support could be implemented at some point? If so I could file a bug to track this work

[squaremo]

"Feature parity" does not mean it works exactly the same; it means you can accomplish the same results, while accounting for the fact that it's a system with a substantially different design.

I get that you probably feel stuck between the old system that works for you, and having to make a big adjustment for the new system. It's a pain that it's not really a viable approach any more, but on the other hand, it was pretty complicated to maintain the code and to operate, and not always very reliable.

Do the maintainers think OCI annotation support could be implemented at some point?

I would strongly prefer not to, for the reasons set out in the opening comment. Relying on image labels means fetching the container config for every single image, and that is very strictly rate-limited these days, especially by DockerHub. At the least it means you can't have instantaneous results, since that requires a full and up-to-date scan of the image repo, with any failures holding the whole process up indefinitely.

Rewriting build systems to include timestamp might not be an option in some situations. For example some users might be syncing 3rd party images and they won't be able to change the way images are tagged upstream.

You can retag upstream images into your own repo -- I grant you this is not going to be viable for everyone though.

But I would question whether you really want to be relying on the timestamps of images that you aren't building yourself. I would expect semver or calver to be more apt (both for the org publishing the images, and the people using them).

[squaremo]

If flux v2 explicitly removes this functionality I think it should be made quite clear in the roadmap.

Good suggestion -- it's now clearer up-front in the roadmap that ordering by build timestamp is not a goal for feature parity.

I'm still interested in arguments for and against this feature, for GA.

[ohkinozomu]

For some people depending on build time, keeping this feature as an option will ease migration from flux v1 to v2.
And it would be nice if we could config whether to use build time in each ImageRepository resource.
ECR and GCR don't have the extremely low rate limit that Docker Hub does.
https://docs.aws.amazon.com/AmazonECR/latest/userguide/service-quotas.html
https://cloud.google.com/container-registry/quotas

[kkoralsky]

I would strongly prefer not to, for the reasons set out in the opening comment. Relying on image labels means fetching the container config for every single image, and that is very strictly rate-limited these days, especially by DockerHub.

AFAIU setting sane imagerepositories.image.toolkit.fluxcd.io.spec.interval is an option to prevent being rate-limited by the registry

At the least it means you can't have instantaneous results, since that requires a full and up-to-date scan of the image repo, with any failures holding the whole process up indefinitely.

I agree this requires some more development effort, but I think this is actually the most intuitive from the user perspective, how automatic image update should work.

[stefanprodan]

In Flux1 we told users to tag their images with ${GIT_BRANCH}-${GIT_SHA:0:7}, migrating to Flux2 they'll have to change the format to ${GIT_BRANCH}-${GIT_SHA:0:7}-$(date +%s).

I have added an example to the image-reflector-controller API docs on how to write image policies that filter tags based on branch name and select the latest build fluxcd/image-reflector-controller#97.

[jacekn]

The org.opencontainers.artifact.created annotation is part of the OCI spec. It would be convenient to support it to avoid forcing users to conform to a certain tag format (some organization might have rules outside of flux that they need to adhere to making it hard to change tags).

Is there a technical reason why support for the org.opencontainers.artifact.created can't be added? Or is it just a matter of limited development time?

[stefanprodan]

In Flux v2 we decided to no longer download images but only list tags which is a different API. Flux v1 was banned from Quay due to having 1000+ clusters downloading images just to get some timestamp, also Docker Hub rate limits would make this impossible.

[jacekn]

Thanks for your reply!
Yes I can completely understand that using the flux v1 method with public registries can cause problems and get people rate limited. It's for sure something that at very least should be disabled by default.
At the same time this is a non-issue for many private registries.

However there may be a way to have both - avoid rate limiting while also avoiding forcing people to change their build systems.
Looking at the docker registry API v2 there are fields returned by the tag list that should allow you to sort by time without making any other API request. Specifically there are the last_updated and tag_last_pushed fields that contain timestamp.
Is there a technical reason why those fields can't (optionally) be used to improve image update usability?

[stefanprodan]

Is there a technical reason why those fields can't (optionally) be used to improve image update usability?

This would require a major refactoring of Flux API, controller, its internal storage structure and of the library we use to query the registry API. I'm not saying NO to this, anything is possible with enough contributors and maintainers.