Skip to content

Troubleshooting Guide

Orion Logo Orion Logo


Start Here

This page covers problems on a running Orion platform. For other issues:

If you're... See
Installing Orion, or can't reach the web interface after install Installation Troubleshooting
Setting up GPU passthrough for VM workloads GPU Passthrough Troubleshooting
Still stuck Getting Support

Common Issues

"WebSocket disconnected" on Helios or VM consoles (Traefik)

Symptom: on clusters using Traefik, the first connection works, but after any workload is launched or deleted, Helios and VM consoles show "WebSocket disconnected". The browser developer console shows net::ERR_CERT_AUTHORITY_INVALID.

Cause: Traefik has no default certificate configured, so it generates a new self-signed certificate every time its configuration reloads, and launching or deleting a workload causes a reload. Browsers reject a WebSocket with an untrusted certificate without showing a warning, and clicking through the "not secure" page only trusted the previous certificate. This doesn't happen with ingress-nginx.

Fix: give Traefik a fixed default certificate through a TLSStore named default, and make sure browsers trust it.

  1. Find Traefik's namespace. This is usually traefik, or kube-system for the Traefik bundled with k3s. Use it in place of <traefik-namespace> below.

    kubectl get pods -A -l app.kubernetes.io/name=traefik
    
  2. Get a certificate for your Orion domain and its subdomains (<domain> and *.<domain>), stored as a TLS secret named traefik-default-cert in Traefik's namespace. The certificate must come from a CA that your users' browsers trust.

    • Recommended: cert-manager. Issue a wildcard Certificate from your issuer with secretName: traefik-default-cert in <traefik-namespace>. cert-manager renews it automatically.
    • Existing certificate files from your CA:

      kubectl create secret tls traefik-default-cert \
        --cert=tls.crt \
        --key=tls.key \
        -n <traefik-namespace>
      
  3. Create the default TLSStore. It must be named default, and must be in the same namespace as the secret. Traefik versions older than v2.10 use apiVersion: traefik.containo.us/v1alpha1.

    kubectl apply -f - <<'EOF'
    apiVersion: traefik.io/v1alpha1
    kind: TLSStore
    metadata:
      name: default
      namespace: <traefik-namespace>
    spec:
      defaultCertificate:
        secretName: traefik-default-cert
    EOF
    
  4. Verify. Run the fingerprint command from Networking and Ingress, launch or delete a workload, then run it again. The fingerprint must not change, and an open Helios or VM connection should stay up.

Diagnostic Commands

Useful kubectl commands for general cluster troubleshooting.

Connect to Argo CD

# Port forward argocd-server. Providing access to the argoCD GUI.
# View logs, sync status, deployed services etc.
# example connection: localhost:8080/argocd
kubectl port-forward svc/argocd-server -n argocd 8080:443

General Cluster Checks

# Quick cluster health check
kubectl cluster-info

# API version
kubectl version --short

Nodes

# Get list of all nodes
kubectl get nodes

# describes the given node showing individual resource requests, limits, and age
kubectl describe node <node-name>

# describes all nodes showing individual resource requests, limits, and age
kubectl describe nodes

# Verify node health
kubectl describe nodes | grep -i "ready\|schedulable"

# Check system resource usage
kubectl top nodes

Pods

# Get all pods in all namespaces
kubectl get pods -A

# Get all pods in a specific name space
kubectl get pods -n <namespace>

# describe a specific pod
kubectl describe pod <pod-name> -n <namespace>

# Get a pods logs
kubectl logs <pod-name> -n <namespace>

Jobs

# Get all jobs in all namespaces
kubectl get jobs -A

# Get a specific job
kubectl get job <job-name>

# Describe a specific job
kubectl describe job <job-name>

# Delete a job
kubectl delete job <job-name>

Deployments

# Get deployments in a specific namespace
kubectl get deploy -n <namespace>

# Get deployments in all namespaces
kubectl get deploy -A

# describe a specific deployment
kubectl describe deploy <deployment-name> -n <namespace>

# Force restart all Orion deployments
kubectl rollout restart deployment -n argocd

# Force restart a specific deployment
kubectl rollout restart deployment/<deployment-name> -n <namespace>

Storage

# Check persistent volumes
kubectl get pv
kubectl get pvc -A

# Verify storage class
kubectl get storageclass

Networking and Ingress

# Check ingress controller status (ingress-nginx or Traefik)
kubectl get pods -A -l app.kubernetes.io/name=ingress-nginx
kubectl get pods -A -l app.kubernetes.io/name=traefik

# Get ingress from specific namespace
kubectl get ingress -n <namespace> -o yaml

# Get all network policies
kubectl get networkpolicy -A

# describe a specific network policy
kubectl describe networkpolicy <policy> -n <namespace>

# Traefik: check the default certificate (must exist, and be named "default")
kubectl get tlsstore -A

# Show the fingerprint of the certificate the ingress is serving
openssl s_client -connect <domain>:443 -servername <domain> </dev/null 2>/dev/null | openssl x509 -noout -fingerprint -sha256

Events

# Get all events from a specific namespace
kubectl get events -n <namespace>