"pack build --publish" to Github Package Registry fails during export

[making]

pack build --publish to Github Package Registry fails in ANALYZING phase with the following message. Tested with v0.5.0 (git sha: c9cfac75b49609524e1ea33f809c12071406547c)

===> ANALYZING
[analyzer] Error: failed to access previous image: connect to repo store 'docker.pkg.github.com/<owner>/<repo>/<image>:latest': NAME_UNKNOWN: docker package "<image>" does not exist under owner "<image>/<repo>"
ERROR: failed with status code: 1

Actually it works without --publish option + docker push.

[making]

ANALYZING worked after the first docker push. but then EXPORTING fails

===> EXPORTING
[exporter] Reusing layers from image 'docker.pkg.github.com/<owner>/<repo>/<image>@sha256:8c92e099807d70c930d310202f1712fe6d6b9308d9f59402764492a3f9d7303b'
[exporter] Exporting layer 'app' with SHA sha256:9a19b36e215658b551d2873efcc52d4f72fedc1aac2f33b94b5532c368343de8
[exporter] Reusing layer 'config' with SHA sha256:abf8021bc25156cdc1b3c9ff702f7ee04974eb4c72e0bd8fcb719b2d051c9d6f
[exporter] Error: failed to export: exporting config layer: previous image did not have layer with sha 'sha256:abf8021bc25156cdc1b3c9ff702f7ee04974eb4c72e0bd8fcb719b2d051c9d6f'
ERROR: failed with status code: 1

[jromero]

Verified by @dfreilich. This is likely to require a change to imgutil and a new version of lifecycle.

@ekcasey / @natalieparellano correct me if I'm wrong.

[khaledavarteq]

I've had a very similar issue with a GitHub docker registry whereas the same exporter step in a Tekton taskrun worked with an ECR registry instance. So it seems to be tied to GitHub registries.

[ekcasey]

There are two issues here:

Analyze
The analyzer issue was already fixed in lifecycle v0.8.0 (we added more permissive error in imgutil to catch a wider range of repo not found errors coming from different registries) - #303

Export
The problems exporting to the github package registry are caused by this issue - https://ghp.lvmhsh.cnmunity/t/cannot-pull-docker-image-from-github-package-registry-by-digest/14342

The github registry doesn't support fetching manifests by digest. We explicitly record the digest of the previous image during analyze and use the digest reference when reusing layers in export (this helps avoid race conditions and other errors). The lack of support for digest reference makes the github package registry uncompliant with the OCI Distribution spec.

We could try to do something special to support the github package registry despite it's non-compliance:

1. Always use tag reference and double check the digest when reading. This creates potential failure scenarios that don't exist for other registries (the build will fail if the tag has been updated between the analyze and export phases).
2. Never fetch the manifest during export (save everything we need during analyze). No race conditions here, but this would require massive changes and still has some limitations (you wouldn't be able to give analyzer a digest reference, which the lifecycle supports generally).

I would prefer that github simply add support for digest references... not sure how realistic that it on a reasonable time table.
IMHO a registry that doesn't support content-addressable images is not a registry worth using

[dfreilich]

Apparently GH Registry is opening a private beta with support for fetching manifests by digest (see containerd/containerd#3291 (comment) for details). Hopefully they'll roll that out and we'll be able to close this.

[dfreilich]

With the release of Github Container Registry (https://github.blog/2020-09-01-introducing-github-container-registry/), I was successfully able to publish an image with pack.

$ pack build ghcr.io/dfreilich/goflake-server:latest --publish
...
===> EXPORTING
Reusing layers from image 'ghcr.io/dfreilich/goflake-server@sha256:5cc11b3794216aa23e3b1fad52994ccfb34fb52309e99ec96aebba57193294c3'
Reusing layer 'paketo-buildpacks/go-build:targets'
Reusing 1/1 app layer(s)
Reusing layer 'launcher'
Reusing layer 'config'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
*** Images (sha256:5cc11b3794216aa23e3b1fad52994ccfb34fb52309e99ec96aebba57193294c3):
      ghcr.io/dfreilich/goflake-server:latest
Reusing cache layer 'paketo-buildpacks/go-dist:go'
Reusing cache layer 'paketo-buildpacks/go-build:gocache'
Successfully built image ghcr.io/dfreilich/goflake-server:latest

This issue can be closed.