diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index 82ce9650cd..4aff98c90f 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -132,16 +132,22 @@ Common findings: helm -n openshell status openshell helm -n openshell get values openshell kubectl -n openshell get deployment,statefulset,pod,svc,pvc -kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200 -kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200 -kubectl -n openshell rollout status deployment/openshell -kubectl -n openshell rollout status statefulset/openshell +if kubectl -n openshell get deployment openshell >/dev/null 2>&1; then + GATEWAY_WORKLOAD="deployment/openshell" +elif kubectl -n openshell get statefulset openshell >/dev/null 2>&1; then + GATEWAY_WORKLOAD="statefulset/openshell" +else + echo "ERROR: neither deployment/openshell nor statefulset/openshell exists in namespace openshell" >&2 + exit 1 +fi +kubectl -n openshell logs "${GATEWAY_WORKLOAD}" -c openshell-gateway --tail=200 +kubectl -n openshell rollout status "${GATEWAY_WORKLOAD}" ``` -Use the log and rollout commands for the workload kind that exists in the -release. Look for failed installs, unexpected values, missing namespace, wrong -image tag, TLS settings that do not match the registered endpoint, and -scheduling failures. +The workload discovery fails explicitly if the release has neither supported +gateway controller. Look for failed installs, unexpected values, missing +namespace, wrong image tag, TLS settings that do not match the registered +endpoint, and scheduling failures. For HA or PostgreSQL-backed installs, also check the external database Secret referenced by `server.externalDbSecret` and the PostgreSQL workload if the test @@ -196,11 +202,18 @@ label, supervisor env vars `OPENSHELL_K8S_SA_TOKEN_FILE` and `OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET`, plus both the projected `openshell-sa-token` volume and the `spiffe-workload-api` CSI volume. -Check the image references currently used by the gateway deployment: +Check the image references currently used by the gateway workload: ```bash -kubectl -n openshell get deployment openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}" -kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}" +if kubectl -n openshell get deployment openshell >/dev/null 2>&1; then + GATEWAY_WORKLOAD="deployment/openshell" +elif kubectl -n openshell get statefulset openshell >/dev/null 2>&1; then + GATEWAY_WORKLOAD="statefulset/openshell" +else + echo "ERROR: neither deployment/openshell nor statefulset/openshell exists in namespace openshell" >&2 + exit 1 +fi +kubectl -n openshell get "${GATEWAY_WORKLOAD}" -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}" helm -n openshell get values openshell | grep -E 'repository|tag|supervisorImage|workload' ``` @@ -244,8 +257,15 @@ If the gateway is healthy but sandbox creation fails: ```bash kubectl -n openshell get pods kubectl -n openshell get events --sort-by=.lastTimestamp | tail -n 50 -kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200 -kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200 +if kubectl -n openshell get deployment openshell >/dev/null 2>&1; then + GATEWAY_WORKLOAD="deployment/openshell" +elif kubectl -n openshell get statefulset openshell >/dev/null 2>&1; then + GATEWAY_WORKLOAD="statefulset/openshell" +else + echo "ERROR: neither deployment/openshell nor statefulset/openshell exists in namespace openshell" >&2 + exit 1 +fi +kubectl -n openshell logs "${GATEWAY_WORKLOAD}" -c openshell-gateway --tail=200 ``` Check the configured sandbox namespace: @@ -344,7 +364,7 @@ openshell logs | Docker or Podman sandbox never registers | Wrong callback endpoint or supervisor startup failure | Gateway logs and sandbox container logs | | Docker GPU e2e fails before GPU sandbox comparison | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` | | Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod ` | -| Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` | +| Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | Set `GATEWAY_WORKLOAD` as shown in Step 5, then run `kubectl -n openshell logs "${GATEWAY_WORKLOAD}" -c openshell-gateway` | | CLI TLS error | Local mTLS bundle does not match server cert/CA | Check `~/.config/openshell/gateways//mtls/` | | Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials | | `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled |