Troubleshooting Guide¶
Start Here¶
This page covers problems on a running Orion platform. For other issues:
| If you're... | See |
|---|---|
| Installing Orion, or can't reach the web interface after install | Installation Troubleshooting |
| Setting up GPU passthrough for VM workloads | GPU Passthrough Troubleshooting |
| Still stuck | Getting Support |
Common Issues¶
"WebSocket disconnected" on Helios or VM consoles (Traefik)¶
Symptom: on clusters using Traefik, the first connection works, but after any workload is launched or deleted, Helios and VM consoles show "WebSocket disconnected". The browser developer console shows net::ERR_CERT_AUTHORITY_INVALID.
Cause: Traefik has no default certificate configured, so it generates a new self-signed certificate every time its configuration reloads, and launching or deleting a workload causes a reload. Browsers reject a WebSocket with an untrusted certificate without showing a warning, and clicking through the "not secure" page only trusted the previous certificate. This doesn't happen with ingress-nginx.
Fix: give Traefik a fixed default certificate through a TLSStore named default, and make sure browsers trust it.
-
Find Traefik's namespace. This is usually
traefik, orkube-systemfor the Traefik bundled with k3s. Use it in place of<traefik-namespace>below. -
Get a certificate for your Orion domain and its subdomains (
<domain>and*.<domain>), stored as a TLS secret namedtraefik-default-certin Traefik's namespace. The certificate must come from a CA that your users' browsers trust.- Recommended: cert-manager. Issue a wildcard
Certificatefrom your issuer withsecretName: traefik-default-certin<traefik-namespace>. cert-manager renews it automatically. -
Existing certificate files from your CA:
- Recommended: cert-manager. Issue a wildcard
-
Create the default TLSStore. It must be named
default, and must be in the same namespace as the secret. Traefik versions older than v2.10 useapiVersion: traefik.containo.us/v1alpha1. -
Verify. Run the fingerprint command from Networking and Ingress, launch or delete a workload, then run it again. The fingerprint must not change, and an open Helios or VM connection should stay up.
Diagnostic Commands¶
Useful kubectl commands for general cluster troubleshooting.
Connect to Argo CD¶
# Port forward argocd-server. Providing access to the argoCD GUI.
# View logs, sync status, deployed services etc.
# example connection: localhost:8080/argocd
kubectl port-forward svc/argocd-server -n argocd 8080:443
General Cluster Checks¶
Nodes¶
# Get list of all nodes
kubectl get nodes
# describes the given node showing individual resource requests, limits, and age
kubectl describe node <node-name>
# describes all nodes showing individual resource requests, limits, and age
kubectl describe nodes
# Verify node health
kubectl describe nodes | grep -i "ready\|schedulable"
# Check system resource usage
kubectl top nodes
Pods¶
# Get all pods in all namespaces
kubectl get pods -A
# Get all pods in a specific name space
kubectl get pods -n <namespace>
# describe a specific pod
kubectl describe pod <pod-name> -n <namespace>
# Get a pods logs
kubectl logs <pod-name> -n <namespace>
Jobs¶
# Get all jobs in all namespaces
kubectl get jobs -A
# Get a specific job
kubectl get job <job-name>
# Describe a specific job
kubectl describe job <job-name>
# Delete a job
kubectl delete job <job-name>
Deployments¶
# Get deployments in a specific namespace
kubectl get deploy -n <namespace>
# Get deployments in all namespaces
kubectl get deploy -A
# describe a specific deployment
kubectl describe deploy <deployment-name> -n <namespace>
# Force restart all Orion deployments
kubectl rollout restart deployment -n argocd
# Force restart a specific deployment
kubectl rollout restart deployment/<deployment-name> -n <namespace>
Storage¶
# Check persistent volumes
kubectl get pv
kubectl get pvc -A
# Verify storage class
kubectl get storageclass
Networking and Ingress¶
# Check ingress controller status (ingress-nginx or Traefik)
kubectl get pods -A -l app.kubernetes.io/name=ingress-nginx
kubectl get pods -A -l app.kubernetes.io/name=traefik
# Get ingress from specific namespace
kubectl get ingress -n <namespace> -o yaml
# Get all network policies
kubectl get networkpolicy -A
# describe a specific network policy
kubectl describe networkpolicy <policy> -n <namespace>
# Traefik: check the default certificate (must exist, and be named "default")
kubectl get tlsstore -A
# Show the fingerprint of the certificate the ingress is serving
openssl s_client -connect <domain>:443 -servername <domain> </dev/null 2>/dev/null | openssl x509 -noout -fingerprint -sha256