Troubleshooting Tips
This page contains tips for troubleshooting problems that one may encounter when working with Chainguard Enforce.
Installation
If you’re encountering an issue with installation, we have a few recommendations to check, and guidance on using Rancher.
Verify cluster is installed
The remoteId
in chainctl cluster install
is the UUID of the Gulfstream namespace; this will help identify whether a cluster is installed or not. You can learn more about installation in our Installing the Chainguard Enforce Agent tutorial.
You can get the ID for Gulfstream with kubectl
, as in the next command.
kubectl get namespace gulfstream -o json | jq .metadata.uid
You’ll receive the ID.
"B501db05-3702-4403-81b5-c74a55976597"
With chainctl cluster list
, you can also get this ID by passing the group ID.
chainctl cluster list --group $GROUP_ID -o json | jq -r .items[0].remoteId
The output here will also be the remote ID.
b501db05-3702-4403-81b5-c74a55976597
You can review more options in the chainctl
docs.
Cluster OIDC configuration exit status
You may encounter an exit status due to being unable to fetch the OIDC configuration. This can occur when running the chainctl cluster install
command.
chainctl cluster install --group=$GROUP --private --context demo
The error will be similar to the following.
...
Installing Chainguard agent...
couldn't fetch cluster OIDC configuration exit status 1
The cause of this may be due to the openid-configuration
and jwks
being inaccessible within the Kubernetes API. You can determine this with kubectl
.
First, check whether the issue is with the openid-configuration
.
kubectl get --raw /.well-known/openid-configuration
If this is the issue, you’ll receive an error as output.
Error from server (NotFound): the server could not find the requested resource
Next, you can check if the issue is related to jwks
.
kubectl get --raw /openid/v1/jwks
If jwks
being inaccessible is the issue, you’ll receive an error here.
Error from server (NotFound): the server could not find the requested resource
This output reveals that the endpoints are not available to chainctl
, whcih may be due to an auth proxy between chainctl
and the control plane.
To solve for this issue, you’ll need an OpenID Connect Well-Known Configuration Endpoint, which should be available at the /.well-known/openid-configuration
path, this will enable you to use chainctl
for installation and to register an organization. You can check that the file is there with kubectl
.
kubectl get --raw /.well-known/openid-configuration
If you receive an error that the requested resource was not found, make sure you are using the --private
option on installation. You can review guidance on this option in the Configuring Enforcer Options guide.
If you are using Rancher, you’ll be able to generate “direct access” kubeconfigs that bypass the auth proxy and go directly to the cluster. Please follow the Rancher documentation to create a new kubeconfig file.
Installing Enforce with Rancher
If you’re using Rancher for Kubernetes, you’ll need to point directly to your kubeconfig file:
KUBECONFIG=/etc/rancher/rke2/rke2.yaml chainctl cluster install --group=$GROUP --private
If you cannot reach the OIDC endpoint, this will cause issues for a cluster install, review the previous section for guidance on troubleshooting this.
Additionally, you’ll need direct access to the authorized endpoint and not the Rancher proxy URL. When you run kubectl config view
, you should receive something like the output below, without the Rancher URL.
clusters:
- cluster:
server: <direct-url-to-cluster>
Issues logging in with chainctl
Ensure that you’re pointing to production. You can check the chainctl
config.
chainctl config view
The config should be pointing to the production environment, as in the next output.
platform:
api: https://console-api.enforce.dev
audience: https://console-api.enforce.dev
console: https://console.enforce.dev
issuer: https://issuer.enforce.dev
registry: https://cgr.dev
timestamp-authority: https://tsa.enforce.dev
For more guidance on working with the chainctl
config, review our guide on How to Manage chainctl
Configuration.
Policies
If you’re running into issues with policies, this section has some tips for recovery.
Policy does not cover a given image as expected
When working with the ClusterImagePolicy
, note that the glob wildcard *
does not cover the /
character. For example, a glob pattern like gcr.io/apple/*
will cover paths like gcr.io/apple/yak
or gcr.io/apple/zebra
, but not gcr.io/apple/banana/cashew
.
To match everything — including the /
character — use the **
wildcard instead.
Check that the cluster image policy is in the ConfigMap
You can ensure that a cluster image policy is in the ConfigMap with the following kubectl
command:
kubectl get cm config-image-policies -n cosign-system
A describe
of the ConfigMap will show the full policy.
kubectl describe cm config-image-policies -n cosign-system
Name: config-image-policies
Namespace: cosign-system
Labels: <none>
Annotations: gulfstream.dev/resync: 10s
Data
====
demo-custom-metric-usage-dev:
----
{"uid":"b98f9309-01df-4590-82a0-afb05314b828",REDACTED}
rego-hostnetworking:
----
{"uid":"02d60b97-5d22-48c2-bd92-684025377e53",REDACTED}
BinaryData
====
Events: <none>
Check that the cluster image policy is deployed to cluster
You can use the following kubectl
command to make sure that the cluster image policy is deployed to your cluster.
kubectl get cip
You should get output similar to the following.
NAME AGE
demo-custom-metric-usage-dev 7h24m
rego-hostnetworking 5h45m
Disable admission control
There may be urgent situations where having admission control enabled is not desirable. For example, it could be preventing the cluster from functioning correctly or may be getting in the way of the operator.
In cases like this, you can completely disable the Chainguard Enforce admission webhook with the following commands.
kubectl delete validatingwebhookconfiguration enforcer
kubectl delete mutatingwebhookconfiguration enforcer
After your urgent situation is over, reinstall Enforce in your cluster to restore the webhook to its normal operation.
Verify that namespace is labeled for enforcement
You can check that the namespace is labeled for enforcement with the following command:
kubectl label namespaces $NAME_SPACE policy.sigstore.dev/include=true
To label an individual namespace, or set the option to on install to label all namespaces.
chainctl cluster install --opt=namespace_enforcement_mode=opt-in
You can read more information about installation options in Chainguard Enforce in our guide.
Unable to parse an SBOM
One issue that may come up when working with Chainguard Enforce is that it won’t ingest an SBOM as expected. This section outlines several potential causes for this issue and how you can address them.
Check your permissions
Chainguard Enforce needs access to your images in order to parse the associated SBOMs. If your image is in a private repository, check out our guide on setting up cloud account associations to grant Enforce read access to the image.
SBOMs using older versions of CycloneDX
For CycloneDX, Chainguard Enforce currently only supports version 1.4.
SBOM included as an in-toto attestation
We are in the process of updating Chainguard Enforce so it can readily parse SBOM attestations out of the box. In the meantime, you can parse SBOMs through implementing specific policies that cover this use case.
I don’t have any SBOMs
We recently updated Chainguard Enforce so it will generate SBOMs for any container images it finds that don’t already have one. It can now also ingest SBOMs automatically out of the box.
Check out our guide on Generating and Filtering SBOMs with Chainguard Enforce to learn more about these features.
Enforce does not block Pod creation as expected
The first thing to check is whether you labeled your namespace with policy.sigstore.dev/include=true
. You can double check whether the default
namespace is correctly labeled with the following command.
kubectl get ns -l policy.sigstore.dev/include=true
If it is indeed labeled like this, you’ll receive output like the following.
NAME STATUS AGE
default Active 24s
If you need to label your namespace, you can do so with the following command. Note that this example labels the default
namespace.
kubectl label ns default policy.sigstore.dev/include=true --overwrite
Be sure to use the exact string, variations like included
won’t work.
Sometimes Chainguard Enforce is installed using the observer
profile. Essentially, this means that Enforce just has read-only permission for workload discovery, and it cannot actually enforce policies. You can run chainctl cluster ls
to find what profile the cluster is using.
Debugging with chainctl
You can run chainctl
in debug mode using the verbose flag -v
or --v
and passing an integer value of 2
or larger, as in -v=2
.
Using this flag, you can log all requests and responses or errors.
For example, to grab all logs related to the clusters listed for a group, you can run:
chainctl cluster ls --group $DEMO_GROUP -v=2
The output will return log data to support you in debugging.
2023/03/28 22:23:58.418086 "level"=1 "msg"="Commandline flags" "active-within"="168h0m0s" "api"="api" "audience"="audience" "config"="" "console"="api" "group"="$DEMO_GROUP" "help"="false" "issuer"="oidc" "name"="" "output"="" "registry"="registry" "timestamp-authority"="timestamp" "v"="2"
2023/03/28 22:23:59.000042 gRPC: "level"=2 "msg"="iam.Groups.List" "request"="name:'$DEMO_GROUP' "
2023/03/28 22:23:59.089459 gRPC: "level"=2 "msg"="iam.Groups.List" "response"="items:<id:'' name:'$DEMO_GROUP' description:'Root of $DEMO_GROUP ' > "
2023/03/28 22:23:59.090213 gRPC: "level"=2 "msg"="tenant.Clusters.List" "request"="active_since:<seconds:1677651909 nanos:89511000 > uidp:<descendants_of:'' > "
2023/03/28 22:23:59.177413 gRPC: "level"=2 "msg"="tenant.Clusters.List" "response"=""
NAME | GROUP | REMOTEID | REGISTERED | K8S VERSION | AGENT VERSION | LAST SEEN | ACTIVITY
-------+-------+----------+------------+-------------+---------------+-----------+-----------
By default, your next command passed without the -v
flag will revert to the standard chainctl
experience. Alternatively, you can pass -v=0
to your chainctl
command for the default experience, without logging.
Last updated: 2023-08-09 15:22