Troubleshooting¶

Day-2 reference for diagnosing a NextcloudInstance, Nextcloud, or NextcloudPool that isn't behaving as expected. Start with State inspection, then Logs, then match your symptom in the Common errors table.

State inspection — always start here¶

NS=nextcloud-demo
NAME=demo

# 1. CRD status + events (events often contain the real reason)
kubectl describe nci $NAME -n $NS

# 2. Conditions (ready / failed / why)
kubectl get nci $NAME -n $NS -o jsonpath='{.status.conditions}' | jq

# 3. Resolved Helm chart version
kubectl get nci $NAME -n $NS -o jsonpath='{.status.versionResolution}' | jq

# 4. Downstream HelmRelease
kubectl describe helmrelease -n $NS

# 5. Pods (anything CrashLoopBackOff / ImagePullBackOff / Pending?)
kubectl get pods -n $NS
kubectl describe pod -n $NS <pod-name>

# 6. Managed database (only if spec.database.managed=true)
kubectl get perconapgcluster -n $NS
kubectl describe perconapgcluster -n $NS

# 7. Secrets created by the operator
kubectl get secret -n $NS -l app.kubernetes.io/managed-by=nextcloud-operator

For the logical Nextcloud tenant resource:

kubectl describe nc $NAME -n $NS
kubectl get nc $NAME -n $NS -o jsonpath='{.status.instanceRef}' | jq

Where are the logs?¶

Component	Command
Operator (kopf handlers)	`kubectl logs -n nextcloud-operator-system -l app.kubernetes.io/name=nextcloud-operator --tail=500`
Operator — specific handler errors	Same, then `grep -i "error\\|PermanentError\\|TemporaryError"`
Flux HelmController (who actually installs the chart)	`kubectl logs -n flux-system -l app=helm-controller --tail=200`
Flux SourceController (resolves `HelmRepository`)	`kubectl logs -n flux-system -l app=source-controller --tail=200`
Nextcloud PHP	`kubectl logs -n $NS deploy/<release>-nextcloud -c nextcloud --tail=200`
Nextcloud `occ` output (on-demand)	`NextcloudCommand` CRD; result in `.status.results`
Percona PG operator	`kubectl logs -n pgo -l app.kubernetes.io/name=percona-postgresql-operator --tail=200`
PostgreSQL itself	`kubectl logs -n $NS <pgcluster>-instance1-xxxx-0 -c database --tail=200`

Make the operator verbose

The operator is started with kopf run --verbose by default in our chart. If you run it differently, increase verbosity with --verbose or --debug to see handler-level reconciliation traces.

Force reconciliation¶

When state looks wrong but no error is visible, nudge the operator to re-run its handlers:

# Force reconcile (operator re-runs all field handlers)
kubectl annotate nci $NAME -n $NS k8s.bnerd.com/reconcile=$(date +%s) --overwrite

# Force maintenance tasks on demand
kubectl annotate nci $NAME -n $NS k8s.bnerd.com/run-maintenance=$(date +%s) --overwrite

See Operations & Annotations for all supported annotations, including k8s.bnerd.com/force-delete for bypassing deletion protection.

Common errors¶

Instance stuck in `Pending` or `Creating`¶

Symptom: kubectl get nci shows Pending or Creating for more than 5 minutes.

Check in order:

Operator logs for validation errors — a PermanentError means the spec is wrong and no amount of retrying will help. Example: missing spec.admin, invalid spec.database.type, unknown spec.version.
HelmRepository + HelmRelease exist — kubectl get helmrelease,helmrepository -n $NS. If missing, Flux isn't installed or the operator couldn't reach the K8s API.
Pods Pending — usually means no default StorageClass or the cluster is out of resources. kubectl describe pod surfaces the scheduler's reason.
Pods ImagePullBackOff — the registry isn't reachable or the image/tag doesn't exist. Check spec.image / resolved chart version.

`Database not ready yet` — managed PG never becomes ready¶

Symptom: Operator logs repeat TemporaryError: Database not ready yet until the 20-minute timeout, then the instance goes Failed.

Causes:

Percona PG Operator not installed — kubectl get crd perconapgclusters.pgv2.percona.com. Install via helm install pgo percona/pg-operator -n pgo --create-namespace.
Percona operator is running but has no RBAC in the target namespace — see its logs: kubectl logs -n pgo -l app.kubernetes.io/name=percona-postgresql-operator.
No StorageClass — the PG cluster can't provision PVCs.
Resources too tight — PG instance pods pending because no node has enough CPU/memory.

To recover after fixing the underlying issue, annotate the instance to force reconcile:

kubectl annotate nci $NAME -n $NS k8s.bnerd.com/reconcile=$(date +%s) --overwrite

`HelmRelease` stuck or in `Failed` state¶

Symptom: kubectl get helmrelease -n $NS shows Ready=False.

Diagnose:

kubectl describe helmrelease -n $NS
kubectl logs -n flux-system -l app=helm-controller --tail=200 | grep $NAME

Common HelmRelease failures:

chart "nextcloud" version "x.y.z" not found — the resolved chart version doesn't exist in the repository. Check status.versionResolution on the instance; if you pinned spec.helm.version, verify that tag exists in the upstream Helm repo.
values don't validate against schema — usually from custom spec.helm.values. Test your values locally with helm template.
timed out waiting for the condition — chart installed but pods never went ready. Inspect the pods directly.

To force Flux to retry: flux reconcile helmrelease <release-name> -n $NS --with-source.

Pool instance never gets assigned¶

Symptom: Nextcloud (logical) stays in Assigning phase; NextcloudPool.status.unassigned is 0.

Causes:

Pool is drained — all instances already assigned. Increase spec.replicas on the pool or wait for the pool reconciler to replenish.
Labels don't match — spec.poolSelector.matchLabels on the Nextcloud doesn't match template.metadata.labels on the pool. Compare with kubectl get nci -A --show-labels | grep pool.
Pool instances stuck Pending — the pool is creating replacements but they can't become ready (see "Instance stuck in Pending" above).
Instances exist but aren't Ready yet — only a fully installed instance (phase: Ready, status.installed: true) is assignable. The operator will not hand a partially-provisioned instance to a tenant. Check kubectl get nci -n $NS for the pool's instances; if they sit in Deploying with reason WaitingForInstall, see the next section. Once they reach Ready, the pending Nextcloud is assigned automatically on the next reconcile.

Instance stuck in `Deploying` with the `Installed` condition `False`¶

Symptom: Pods are up and HelmRelease/workload are ready, but the instance never reaches Ready. The Installed condition is False and status.installed is false.

The operator treats "Ready" as installed, not just "pods are running". After the HelmRelease and workload come up it runs occ status once and only advances to Ready when Nextcloud reports installed: true. While it waits, the Installed condition's reason tells you what is happening:

Already-Ready instances are also checked once (backfill)

Instances that reached phase: Ready under an older operator version — before the install check existed — were never verified (status.installCheckedAt is unset, status.installed is empty). The operator now runs the install check once for these too. This backfill is read-only and flag-only: if such an instance turns out to be not installed it is flagged with the WedgedNeedsManualInstall reason (below), never auto-installed. The check fires only while installCheckedAt is unset and stops once the result is cached.

kubectl get nci $NAME -n $NS -o jsonpath='{.status.conditions[?(@.type=="Installed")]}' | jq
kubectl get nci $NAME -n $NS -o jsonpath='{.status.installed}{"\n"}'

`Installed` reason	Meaning	What to do
`WaitingForInstall`	Pods are up; Nextcloud is finishing its first install.	Normal for the first 1–3 min after pods become ready. No action.
`HealInstallTriggered`	The first install had wedged (started but never finished) with an empty database, so the operator re-ran `occ maintenance:install` in place to un-wedge it.	None — the next reconcile re-checks and should flip to `Ready`. The operator did this only after proving the database was empty.
`DbProbePending`	The DB-empty safety probe could not complete on the last few reconciles (up to 10 consecutive). No install was attempted.	Usually transient — the Postgres pod may still be starting. Check `kubectl get pods -n $NS`. Clears automatically once the probe succeeds.
`DbProbeUnreachable`	The DB-empty safety probe has been UNDETERMINED for 10 or more consecutive reconciles (~5 min at the default 30 s timer). The operator cannot confirm the database is empty and will not auto-install (fail-closed by design).	Alertable — see the DbProbeUnreachable runbook below.
`WedgedWithData`	Nextcloud reports not installed, but its database already contains tables. The operator will never auto-install over existing data (it would destroy it).	Manual investigation required — see below.
`RegressedAfterInstall`	The instance previously reported installed and now reports not-installed (a regression, not a first-install failure). The operator will not auto-install.	Manual investigation required — see below.
`HealAttemptsExhausted`	The operator tried the safe self-heal install the bounded number of times and it still isn't installed.	Manual investigation required — see below.
`WedgedNeedsManualInstall`	The install-check backfill found a pre-existing instance (already `phase: Ready` under an older operator version) that reports not installed. Because the instance was long-Ready its database state is unknown, so the operator flags it rather than auto-installing.	Confirm the database state, then run `occ maintenance:install` manually — see below.

The operator never auto-installs over a non-empty database

The automatic self-heal runs occ maintenance:install only when it can prove there is nothing to lose: the instance is not installed, has never been installed before, and the database is verifiably empty of tables. If any of those is not true — or the operator cannot verify the database is empty — it stops and surfaces a Warning condition instead of touching the instance. This is deliberate: re-installing over a data-bearing instance would wipe it.

Manual path for WedgedWithData / RegressedAfterInstall / HealAttemptsExhausted / WedgedNeedsManualInstall:

Look at what the instance's database actually contains:

kubectl logs -n $NS deploy/<release>-nextcloud -c nextcloud --tail=200
# If managed PG, exec into the DB and inspect tables:
kubectl exec -n $NS <pgcluster>-instance1-xxxx-0 -c database -- \
  psql -U nextcloud -d nextcloud -c '\dt'

If the database is genuinely empty and you want the operator to (re)install, it is safe to clear the wedged state — recreate the instance (delete the NextcloudInstance; the pool reconciler creates a fresh one) so it provisions cleanly. For WedgedNeedsManualInstall specifically (an already-Ready instance flagged by the backfill), prefer installing in place once you have confirmed the database is empty:
```
kubectl exec -n $NS deploy/<release>-nextcloud -c nextcloud -- \
  php occ maintenance:install --help   # review options, then run the real install
```
If the database has real tenant data, do not re-install. This is a recovery case: restore from backup or repair the Nextcloud install in place with the appropriate occ commands. Escalate to b'nerd if you are unsure — re-installing here loses data.

Runbook: DbProbeUnreachable — instance stuck not-installed (#7832)¶

Symptom: kubectl get nci $NAME -n $NS -o jsonpath='{.status.conditions[?(@.type=="Installed")]}' returns reason: DbProbeUnreachable. A Warning event is also recorded on the NextcloudInstance object.

What this means:

The operator runs a DB-empty safety probe before it will auto-install Nextcloud: it connects to the Postgres primary pod and counts tables via psql. This probe returning UNDETERMINED (non-zero exit, no pod, network policy blocked, wrong credentials, …) for 10 or more consecutive reconciles (roughly 5 minutes at the default 30 s timer) escalates from the benign DbProbePending condition to an alertable DbProbeUnreachable condition plus a Warning event.

This is fail-closed by design. The operator will not auto-install while the probe result is unknown — even if it has been unknown for a long time. This is not a data-loss risk: no install command has been run, nothing has been mutated. The instance is simply held in a pre-install holding pattern until the connectivity problem is fixed.

Diagnose the probe failure:

# See the full condition message (includes failure count)
kubectl get nci $NAME -n $NS \
  -o jsonpath='{.status.conditions[?(@.type=="Installed")]}' | jq

# See the Warning event
kubectl get events -n $NS --field-selector reason=DbProbeUnreachable

# Check probe failure counters in status
kubectl get nci $NAME -n $NS \
  -o jsonpath='{.status.dbProbeFailures}{"\n"}{.status.dbProbeFirstFailedAt}{"\n"}'

# Is the Postgres primary pod running?
kubectl get pods -n $NS -l postgres-operator.crunchydata.com/role=master
# or, for Percona-managed clusters:
kubectl get pods -n $NS | grep -i pg

# Operator logs — look for "DB-empty probe" lines
kubectl logs -n nextcloud-operator-system deploy/nextcloud-operator \
  | grep -i "DB-empty probe"

Common causes and remediation:

Cause	Remediation
Postgres primary pod not running / still starting	Wait for the cluster to become healthy (`kubectl get perconapgcluster -n $NS`), or fix the underlying infrastructure issue. The operator retries automatically.
Wrong DB credentials in the `<name>-nextcloud-db` Secret	Correct the credentials. If using `spec.database.credentialsSecret`, verify the referenced secret contains the correct password.
`pg_hba.conf` does not allow the Nextcloud user to log in locally	The DB-empty probe connects to `127.0.0.1` inside the Postgres pod. Ensure `pg_hba.conf` allows the Nextcloud database user to connect locally.
Role / database not yet created	On an external (non-managed) database, ensure the role and database exist before creating the instance.
Network policy blocks pod exec from operator	The probe uses `kubectl exec` (WebSocket) into the Postgres pod. Ensure the Kubernetes API server can exec into pods in the instance namespace.

Recovery:

Once DB connectivity is restored and the probe can run, the operator auto-recovers on the next reconcile — no manual intervention is needed. If the probe returns db_empty=True and all other preconditions hold (never previously installed, attempts remaining), occ maintenance:install runs automatically and the instance proceeds to Ready.

If you want to accelerate recovery rather than waiting for the next 30 s timer tick:

kubectl annotate nci $NAME -n $NS \
  k8s.bnerd.com/reconcile=$(date -u +"%Y-%m-%dT%H:%M:%SZ") \
  --overwrite

PromQL alert hint:

To alert on this condition before it sits unnoticed, query the Kubernetes events API or the operator's structured log. If you use kube-state-metrics with a kube_event_count rule:

# Alert when a DbProbeUnreachable Warning event is newer than 10 minutes
count by (namespace, name) (
  kube_event_created_at{reason="DbProbeUnreachable", type="Warning"} >
  (time() - 600)
) > 0

Alternatively, watch the Installed condition reason via the operator's Prometheus metrics (if you have CRD-status scraping) or your log-based alerting on the operator pod:

# Log line emitted each time the alertable condition fires
kubectl logs -n nextcloud-operator-system deploy/nextcloud-operator \
  | grep "DbProbeUnreachable"

`Cannot load API key for SignalingServer / RecordingServer`¶

Symptom: The operator retries every 60s with this TemporaryError.

The referenced credentialsSecret is missing or lacks the expected key. Check:

kubectl get secret <credentialsSecret-name> -n nextcloud-operator-system -o yaml

The secret must contain the API key under the key specified in the SignalingServer/RecordingServer spec.

`Authentication failed for <api-endpoint>: 401`¶

Symptom: PermanentError in operator logs when registering a backend.

The API key in the credentialsSecret is wrong or the backend API is rejecting it. Verify the key on the backend (signaling or recording server), update the secret, then delete the SignalingServer / RecordingServer CR to re-register cleanly.

`NextcloudCommand` times out¶

Symptom: A NextcloudCommand finishes with phase: Failed and status.results[].stderr shows a timeout.

A single command exceeded spec.perCommandTimeoutSeconds (default 300s). For expensive migrations like occ db:convert-filecache-bigint, raise the per-command timeout to e.g. 3600.
The overall job exceeded spec.timeoutSeconds. Raise it or split the commands across multiple NextcloudCommand resources.
No running Nextcloud pod was available — the instance is not Ready. Check spec.targetRef points to a healthy NextcloudInstance/Nextcloud.

`S3 data backup enabled but no repository configured`¶

Symptom: PermanentError at instance creation time.

spec.backups.data.enabled: true requires either an S3Backup CRD (from the bnerd backup operator) to be installed, or the backup repository to be configured. Install the backup operator, or disable the feature.

CrashLoopBackOff on Nextcloud pod after an upgrade¶

Symptom: After bumping spec.version, pods crash-loop with migration errors.

Post-upgrade migrations should run automatically via utils/maintenance.py, but you can trigger them manually:

kubectl annotate nci $NAME -n $NS k8s.bnerd.com/run-maintenance=$(date +%s) --overwrite

Watch the operator logs to confirm the maintenance tasks ran. If migrations like add-missing-indices or convert-filecache-bigint fail, inspect the output and run them as a dedicated NextcloudCommand with a longer timeout.

Finalizer blocks namespace deletion¶

Symptom: kubectl delete namespace hangs; the namespace stays in Terminating.

A NextcloudInstance still has a finalizer because cleanup is incomplete (typically a managed DB that won't delete). To unblock:

# Check what's blocking
kubectl get nci -n $NS -o jsonpath='{.items[*].metadata.finalizers}'

# Force-delete (skips operator cleanup — only use when you accept losing state)
kubectl annotate nci $NAME -n $NS k8s.bnerd.com/force-delete=true --overwrite
kubectl delete nci $NAME -n $NS

For the full teardown order, audit log, and recreate-safety behaviour, see Deletion & Cleanup.

When to file a bug vs. keep debugging¶

File a bug if:

The operator panics or the pod crash-loops (kubectl logs shows a Python traceback without a clear PermanentError).
A TemporaryError repeats indefinitely even after the underlying cause is fixed.
Status fields contradict reality (e.g. phase: Ready but pods are CrashLoopBackOff). Ready only gets set when the HelmRelease, Deployment, Endpoints, and Ingress all report ready and Nextcloud reports installed: true (status.installed) — so this combination should no longer be reachable. If you see it, file a bug with kubectl get nci $NAME -n $NS -o yaml attached.
Instance is stuck in Deploying: inspect status.workload and the Ready condition's reason. WaitingForHelmRelease → check the Flux HelmRelease (kubectl describe helmrelease ...). WaitingForPods → describe the Nextcloud Deployment; usually a values misconfiguration or PVC problem. WaitingForEndpoints → the Service has no ready backends (pod ready probe failing). WaitingForIngress → the cluster's ingress controller hasn't assigned a load-balancer address yet.
Instance is stuck in Creating with database.managed: true: inspect status.database and the DatabaseReady condition. reason=Initializing is normal — Percona PG cluster startup typically takes 1–5 min. reason=ProvisioningFailed → check kubectl describe perconapgcluster $NAME-pg -n $NS and look at events. reason=Timeout → 20 min have passed and the cluster still isn't ready; phase will transition to Failed but the operator keeps retrying. Common causes: missing StorageClass, pg-operator pod not running, image pull failure. Fix the underlying infra issue and the next 60 s retry should self-heal back to Creating → Deploying → Ready without operator intervention.
Instance is at phase: Failed with status.database.reason=PgOperatorNotFound: the Percona PG operator CRD is not installed in the cluster. This is a PermanentError — install the pg-operator (make install-pg-operator or your usual Flux/Helm flow) and then re-trigger via kubectl annotate nci $NAME -n $NS k8s.bnerd.com/reconcile=$(date +%s) --overwrite.

Keep debugging yourself if:

The error message explicitly says what's wrong (missing secret, invalid field, wrong version). The operator is telling you — believe it.
The HelmRelease is failing — that's a chart/values issue, not an operator bug.
Kubernetes primitives are broken (no StorageClass, no ingress, no DNS). Those aren't the operator's job to fix.