Troubleshooting
Comprehensive troubleshooting guide for pods, network, databases, Keycloak, backend services, and frontend issues
Troubleshooting
Common issues and solutions for ORISO Platform v3.0.0 deployed on Kubernetes with Helm charts.
Pods Stuck in Pending
Check pod events and resource availability:
# Check pod events
kubectl describe pod <pod-name> -n caritas
# Common causes:
# - Insufficient resources
# - Persistent volume issues
# - Image pull errors
# Check node resources
kubectl describe nodes
# Check available resources
kubectl top nodesPods CrashLoopBackOff
Investigate application errors and configuration:
# Check logs
kubectl logs <pod-name> -n caritas
kubectl logs <pod-name> -n caritas --previous
# Common causes:
# - Application errors
# - Database connection issues
# - Missing configuration
# - Resource limits too low
# Check events
kubectl get events -n caritas --sort-by='.lastTimestamp'Image Pull Errors
Resolve image pull and registry issues:
# Check image pull secrets
kubectl get secrets -n caritas
# Check pod image
kubectl get pod <pod-name> -n caritas -o jsonpath='{.spec.containers[0].image}'
# Pull image manually
docker pull <image>
# Import to k3s
docker save <image> | sudo k3s ctr images import -Service Not Accessible
Verify service configuration and pod selection:
# Check service (use oriso-platform-* prefix)
kubectl get svc -n caritas oriso-platform-<service>
# Check endpoints
kubectl get endpoints -n caritas oriso-platform-<service>
# Check if pods are selected
kubectl get pods -n caritas -l app=oriso-platform-<service>
# Test from within cluster using Kubernetes DNS
kubectl run test-pod --rm -it --image=busybox -n caritas -- /bin/sh
wget -O- http://oriso-platform-<service>.caritas.svc.cluster.local:<port>
# Test service communication
kubectl exec -n caritas deployment/oriso-platform-userservice -- \
curl http://oriso-platform-tenantservice.caritas.svc.cluster.local:8081/actuator/healthPort Not Responding
Check firewall and network configuration:
# Check if port is listening
sudo netstat -tulpn | grep <port>
# Check UFW
sudo ufw status verbose
# Allow port if needed
sudo ufw allow <port>/tcp
# Check iptables
sudo iptables -L -nCannot Connect to MariaDB
Verify MariaDB pod and service:
# Check MariaDB pod
kubectl get pod -n caritas | grep mariadb
# Check logs
kubectl logs deployment/oriso-platform-mariadb -n caritas
# Test connection (use service DNS name)
kubectl exec -it -n caritas deployment/oriso-platform-mariadb -- \
mysql -h oriso-platform-mariadb.caritas.svc.cluster.local -u root -p${MYSQL_ROOT_PASSWORD} -e "SHOW DATABASES;"
# Check service
kubectl get svc -n caritas oriso-platform-mariadb
# Test from another pod
kubectl exec -n caritas deployment/oriso-platform-userservice -- \
curl http://oriso-platform-mariadb.caritas.svc.cluster.local:3306Cannot Connect to MongoDB
Verify MongoDB pod and service:
# Check MongoDB pod
kubectl get pod -n caritas | grep mongodb
# Check logs
kubectl logs deployment/mongodb -n caritas
# Test connection
MONGODB_POD=$(kubectl get pods -n caritas -l app=mongodb -o jsonpath="{.items[0].metadata.name}")
kubectl exec -it -n caritas $MONGODB_POD -- mongosh --eval "show dbs"
# Check service
kubectl get svc -n caritas mongodbHTTPS Required Error
This means HTTP access was not configured. This is a critical configuration step.
# Run the configuration script
cd ~/online-beratung/caritas-workspace/ORISO-Keycloak
chmod +x configure-http-access.sh
./configure-http-access.sh
# Or configure manually
KEYCLOAK_POD=$(kubectl get pods -n caritas -l app=keycloak -o jsonpath="{.items[0].metadata.name}")
kubectl exec -n caritas $KEYCLOAK_POD -- \
/opt/keycloak/bin/kcadm.sh update realms/master -s sslRequired=NONESee Configure Keycloak for detailed instructions.
Cannot Login to Keycloak
Check Keycloak logs and accessibility:
# Check Keycloak logs
kubectl logs deployment/oriso-platform-keycloak -n caritas
# Verify Keycloak is accessible (internal)
kubectl exec -n caritas deployment/oriso-platform-keycloak -- \
curl -I http://localhost:8080
# Verify Keycloak is accessible (external via Ingress)
curl -I https://auth.oriso.org
# Check Keycloak service
kubectl get svc -n caritas oriso-platform-keycloak
# Reset admin password if needed (see Security Hardening)Realm Not Found
Verify realm exists and re-import if necessary:
# Check if realm exists
KEYCLOAK_POD=$(kubectl get pods -n caritas -l app=keycloak -o jsonpath="{.items[0].metadata.name}")
kubectl exec -n caritas $KEYCLOAK_POD -- \
/opt/keycloak/bin/kcadm.sh config credentials \
--server http://localhost:8080 \
--realm master \
--user admin \
--password admin
kubectl exec -n caritas $KEYCLOAK_POD -- \
/opt/keycloak/bin/kcadm.sh get realms --fields realm
# If realm missing, re-import (see Configure Keycloak)Service Returns 500 Error
Check service logs and configuration:
# Check service logs (use oriso-platform-* prefix)
kubectl logs deployment/oriso-platform-<service> -n caritas --tail=100
# Common issues:
# - Database connection failed (check DNS: oriso-platform-mariadb.caritas.svc.cluster.local)
# - Keycloak not accessible (check DNS: oriso-platform-keycloak.caritas.svc.cluster.local)
# - Missing environment variables
# - Application errors
# Check environment variables
kubectl get deployment oriso-platform-<service> -n caritas -o yaml | grep -A 20 env:
# Check service communication
kubectl exec -n caritas deployment/oriso-platform-<service> -- \
curl http://oriso-platform-mariadb.caritas.svc.cluster.local:3306Service Health Check Fails
Verify health endpoints and dependencies:
# Check actuator health endpoint
curl http://127.0.0.1:<port>/actuator/health | jq .
# Check detailed health
curl http://127.0.0.1:<port>/actuator/health/db | jq .
curl http://127.0.0.1:<port>/actuator/health/redis | jq .
# Restart service
kubectl rollout restart deployment/<service-name> -n caritasFrontend Shows White Screen
Check frontend logs and environment:
# Check frontend logs
kubectl logs deployment/frontend -n caritas
# Common causes:
# - Build errors
# - Missing environment variables
# - API URL incorrect
# Check frontend environment
kubectl exec -it deployment/frontend -n caritas -- env | grep VITEFrontend 403 Errors
Check Ingress configuration and CORS:
# Check Ingress Controller logs
kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller
# Check Ingress resources
kubectl get ingress -n caritas
kubectl describe ingress -n caritas
# Check CORS configuration in Ingress
kubectl get ingress -n caritas -o yaml | grep -A 5 cors
# Verify backend is accessible
curl -I https://api.oriso.org/actuator/healthIngress Not Working
# Check Ingress Controller
kubectl get pods -n ingress-nginx
# Check Ingress resources
kubectl get ingress -n caritas
# Check Ingress Controller logs
kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller --tail=100
# Test Ingress routing
curl -I https://app.oriso.orgTLS Certificate Issues
# Check cert-manager
kubectl get pods -n cert-manager
# Check certificates
kubectl get certificate -n caritas
kubectl describe certificate <cert-name> -n caritas
# Check certificate requests
kubectl get certificaterequest -n caritas
kubectl describe certificaterequest <name> -n caritas
# Check ClusterIssuer
kubectl get clusterissuer letsencrypt-prod
kubectl describe clusterissuer letsencrypt-prodHelm Issues
# Check Helm release
helm list -n caritas
# Check release status
helm status oriso-platform -n caritas
# View release values
helm get values oriso-platform -n caritas
# Check for errors in manifest
helm get manifest oriso-platform -n caritas | grep -i error
# View release history
helm history oriso-platform -n caritas
# Rollback if needed
helm rollback oriso-platform -n caritasGeneral Debugging
# Check all pods status
kubectl get pods -n caritas
# Check all services (use oriso-platform-* prefix)
kubectl get svc -n caritas
# Check all deployments
kubectl get deployments -n caritas
# Check Helm release
helm list -n caritas
helm status oriso-platform -n caritas
# Check Ingress
kubectl get ingress -n caritas
# Check TLS certificates
kubectl get certificate -n caritas
# Check events (last 1 hour)
kubectl get events -n caritas --sort-by='.lastTimestamp' | tail -50
# Check node resources
kubectl top nodes
kubectl top pods -n caritas
# Get all logs for a pod (use oriso-platform-* prefix)
kubectl logs deployment/oriso-platform-<name> -n caritas --tail=200
# Follow logs in real-time
kubectl logs deployment/oriso-platform-<name> -n caritas -f
# Execute command in pod
kubectl exec -it deployment/oriso-platform-<name> -n caritas -- /bin/sh
# Port-forward for local access
kubectl port-forward -n caritas svc/oriso-platform-<service> 8080:8080
# Describe resource (shows events and details)
kubectl describe pod <pod-name> -n caritas
kubectl describe deployment oriso-platform-<name> -n caritas
kubectl describe svc oriso-platform-<service> -n caritas
# Check resource usage
kubectl top pod <pod-name> -n caritas
# Restart deployment
kubectl rollout restart deployment/oriso-platform-<name> -n caritas
# Check rollout status
kubectl rollout status deployment/oriso-platform-<name> -n caritas
# Scale deployment
kubectl scale deployment/oriso-platform-<name> --replicas=2 -n caritas
# Test service DNS resolution
kubectl run test-pod --rm -it --image=busybox -n caritas -- \
nslookup oriso-platform-userservice.caritas.svc.cluster.localRelated Documentation
- Quick Reference - Essential commands
- Post-Deployment Configuration - Verification steps
- Configure Keycloak - Keycloak configuration