Troubleshooting¶
Organized by what you are looking at. Every install failure prints a line
starting with x install:; find yours below.
Throughout, <kube> stands for the kubectl command that reaches your
cluster — which one depends on how you installed:
| How you installed | <kube> is |
|---|---|
| k3d mode (Mac) | kubectl --context k3d-mesopod-<your-slug> |
| OrbStack mode (Mac) | kubectl --context orbstack |
A cluster you brought yourself (--kube-context CTX) |
kubectl --context CTX |
| Linux server (k3s) | sudo k3s kubectl — the sudo is not optional there, because k3s keeps its kubeconfig at /etc/rancher/k3s/k3s.yaml readable by root only |
State on your Mac¶
Both Mac modes keep three small files under your home directory, unprivileged:
| File | What it holds |
|---|---|
$HOME/.mesopod/outpost-token (0600) |
This cluster's outpost token. Its presence is what lets a re-run resume without a fresh claim. |
$HOME/.mesopod/cluster-slug |
The slug the platform assigned. k3d mode reads this on a resume to know which cluster (mesopod-<slug>) to reuse. |
$HOME/.mesopod/install-mode |
Which mode you installed in (k3d or orbstack), so a re-run resumes where the first one landed instead of asking again. |
(--token-file PATH moves the first; the other two then live next to it. Full
detail: install.md.)
Deleting any of them is safe with respect to your cluster and apps — none of these files is read by anything running there, only by a future run of the install script on this Mac. But it throws away the resume path: the token cannot be recovered from the file system once it's gone, so the next run has no saved token to resume from and needs a fresh claim. Get one from the console's Generate new install command — don't re-paste an old command from your clipboard or shell history; its claim was already spent and generating a new one retires it anyway.
The install command fails¶
unknown claim — check the pasted command for truncation (HTTP 401)¶
The claim string didn't reach the server intact. Copy the command from the console again with the copy button; don't retype it.
… — generate a new install command in the console (HTTP 410)¶
The claim was already used, or it expired (they last 24 hours, and each is single-use). In the console, on the cluster's row, press Generate new install command and run that one. Generating a new command retires any earlier one immediately.
If the button isn't there, look at the tab you created the cluster in: it still shows the original command in its copy well, because the claim is reveal-once and that tab is the only place it ever existed. Reload — or open the console anywhere else — and the well is replaced by the Generate new install command button.
If the earlier run got as far as printing a + token saved to … line, you
don't need a claim at all — just re-run the script with none:
curl -fsSL https://console.mesopod.cloud/api/v1/install.sh | sh
(Linux server: add sudo — see the Linux appendix.)
claim endpoint rate limit — wait a minute and re-run (HTTP 429)¶
Exactly that. Wait a minute.
Kubernetes 1.29 is below Mesopod's 1.30 floor (ValidatingAdmissionPolicy GA) — upgrade the cluster first¶
You are installing onto a cluster older than 1.30. Mesopod's outpost is fenced
in by admission policies that don't exist before 1.30, and it will not install
without them. This should never actually happen with a fresh k3d cluster or
OrbStack's Kubernetes — both ship well above the floor — so seeing this
usually means you pointed --kube-context at an older cluster. Upgrade it, or
let the script create a fresh k3d cluster instead.
On OrbStack mode, a --kube-context install, or the Linux k3s path, this check
runs before the claim exchange, so your claim is not spent — the same command
works again after the upgrade. On k3d mode the exchange happens first (its
response is what names the cluster), so by the time this fires the claim is
already redeemed and its token saved; that's fine — re-running needs no new
claim either way, it just resumes.
unsupported architecture … — Mesopod ships linux/amd64 and linux/arm64¶
Mesopod has no build for that machine (a 32-bit ARM board, most likely).
this platform build has no linux/arm64 outpost image (HTTP 404)¶
The machine's architecture is supported, but this Mesopod deployment has no image for it. Tell us — that's ours to fix, not yours.
the saved token was revoked — generate a new install command in the console¶
Someone generated a new install command for this cluster — that revokes the old token — or the cluster was removed. If the row still offers Generate new install command, use it and run the new one. The console only offers that before a cluster's first successful connect; if this cluster has connected before and its token is now dead, tell us — reissue after first connect isn't in the console yet.
Mac-specific install issues¶
Resuming after any of these¶
Which re-run you want depends on whether your claim was spent, and the failures below split cleanly in two.
Before the claim is spent. These fire while the script is still checking
this Mac — nothing has been exchanged and nothing written: no orbstack
context, docker or k3d missing in k3d mode, no terminal to ask the mode
question on, an unusable answer to that question, and k3s asked for on a Mac.
Fix the problem and paste the same command from the console again; the
claim is untouched and still good (for its 24 hours).
After the claim is spent. Everything else in this section happens once the
claim has become an outpost token on disk (see
State on your Mac above). Re-pasting the original
command then hits the … — generate a new install command in the console
(HTTP 410) failure above, because its claim is gone. Fix the problem and run
the no-claim form instead — it resumes from the saved token, in the mode the
first run used:
curl -fsSL https://console.mesopod.cloud/api/v1/install.sh | sh
Each entry below says which of the two it is.
docker is not on PATH — install OrbStack (it provides docker) or Docker Desktop, then re-run¶
Before the claim is spent. k3d mode needs a Docker engine to create its
cluster in, and looks for one before exchanging anything. Install
OrbStack (it provides docker even if you never turn
on its Kubernetes) or Docker Desktop, make sure it's running, then paste the
same command again (above).
k3d is not on PATH — install it with: brew install k3d¶
Before the claim is spent — this check runs ahead of the exchange too.
Run brew install k3d, then paste the same command again
(above).
no "orbstack" kubectl context found — enable Kubernetes in OrbStack's settings (Settings > Kubernetes) and try again¶
Before the claim is spent. OrbStack mode targets a kubectl context
literally named orbstack, which only exists once you've turned Kubernetes on
inside OrbStack. Open the OrbStack app → Settings → Kubernetes →
enable it, wait for it to come up, then paste the same command again
(above). (A context merely named something
with "orbstack" in it doesn't count — the match is exact.)
Mac port 80 or 443 is already in use¶
Before the claim is spent. k3d mode publishes ports 80 and 443 from the cluster onto your Mac, so other devices on your network reach your apps at the Mac's address. Only one cluster can hold those ports. The script checks them before it exchanges your claim, and asks:
Ports 80 and 443 (k3d cluster 'mesopod-other') already in use.
Publishing 80/443 is what makes this machine answer on those ports, so
other devices on your network reach deployed apps at its address. Only one
cluster can hold them.
1) quit, so you can free the ports (k3d cluster stop NAME)
2) install anyway, without publishing — apps then reachable from this
machine only, at the cluster node's own address
Choice [1]:
Nothing has been spent at this point, so quitting costs nothing — no claim exchanged, no cluster registered. Answer 1, stop whatever holds the ports, and paste the same command again.
The message names the culprit for you. It's usually another k3d cluster from an
earlier install, which k3d cluster stop <name> frees. Note that lsof -i :80
is not a useful check here: on a Mac it reports OrbStack (or Docker) for
any published container port — the port-forwarding proxy, not the cluster behind
it. docker ps --format '{{.Names}}\t{{.Ports}}' | grep ':80->' shows the real
holder.
Answer 2 (or pass --no-port-publish, or set
MESOPOD_INSTALL_PUBLISH_PORTS=0) to install anyway. This is what lets two k3d
clusters share one machine. The trade: your apps answer only at the cluster
node's own address, reachable from this Mac but not from other devices on your
network — so point the cluster's DNS target there, or accept it's a
this-machine-only install.
If neither suits, OrbStack mode doesn't need those ports the same way.
A bind failure can still reach k3d cluster create — something grabbed the port
between the check and the create, or the check couldn't run (no lsof). Then
k3d's own error surfaces verbatim, after the claim is spent; resume the install
(above), nothing is lost.
docker load failed — output above; is OrbStack running? — see the install guide's troubleshooting section¶
After the claim is spent. OrbStack mode side-loads the outpost image with
a plain docker load. This means either OrbStack itself has stopped, or its
Kubernetes was toggled off mid-install. Make sure OrbStack is running and its Kubernetes is enabled, then
resume the install (above).
image import into k3d failed — output above; is the k3d cluster running? check: docker ps — see the install guide's troubleshooting section¶
After the claim is spent. The k3d node container isn't there or isn't
running. docker ps should show a
container named k3d-mesopod-<your-slug>-server-0; if it's missing, the
cluster may have been deleted or Docker/OrbStack restarted and dropped it.
Resuming the install (above) recreates the
cluster if needed (it's idempotent).
no terminal to ask on — say where Mesopod should run with --k3d or --orbstack (or set MESOPOD_INSTALL_MODE=k3d|orbstack)¶
Before the claim is spent. The script wanted to ask the mode question but
had no terminal to ask it on — usually because the command is running inside
another script or a CI job. Pass --k3d or --orbstack explicitly
(sh -s -- --k3d after the pipe, or as a flag if you saved the script), or
set MESOPOD_INSTALL_MODE=k3d / MESOPOD_INSTALL_MODE=orbstack in the
environment, and paste the command again — the claim is still good.
answer 1 or 2 (or re-run with --k3d or --orbstack) — got "…"¶
Before the claim is spent. Something other than 1, 2, k3d,
orbstack, or an empty Enter was typed at the mode prompt. Paste the same
command again and answer with one of those, or skip the prompt entirely with
--k3d/--orbstack.
k3s does not run on macOS — choose --k3d (a local k3d cluster) or --orbstack (OrbStack's built-in Kubernetes)¶
Before the claim is spent. Something asked for the Linux k3s path on a Mac
(k3s just doesn't run there). In practice that is a leftover
MESOPOD_INSTALL_MODE=k3s exported in the shell — an explicit --kube-context
cannot reach this guard, since naming a context resolves to that context.
unset MESOPOD_INSTALL_MODE, or pass --k3d / --orbstack, and paste the
command again — see install.md.
node not Ready after ~5m¶
This is the Linux k3s path only (k3d and OrbStack manage node readiness themselves during cluster creation/startup, surfaced as a different failure if it doesn't happen). k3s installed but its node never came up. Look at it directly:
sudo k3s kubectl get nodes
sudo journalctl -u k3s -n 100 --no-pager
Low memory and a full disk are the usual causes.
The outpost doesn't come up¶
outpost rollout did not complete — pod state above¶
The script prints the pods before it exits. Read the STATUS column:
<kube> -n mesopod-system get pods
ImagePullBackOff/ErrImagePull— the image is not on the machine. Mesopod pulls from no registry, so this means the side-load step didn't leave the image where the cluster looks for it. Re-run the install script; it downloads and loads the image again.CrashLoopBackOff— the outpost starts and exits. Read its log (below).Pending— the cluster has nowhere to put the pod.<kube> -n mesopod-system describe pod <name>names the reason; usually it is disk pressure or memory.
The outpost's logs¶
<kube> -n mesopod-system logs -l app=mesopod-outpost --tail=100
Two replicas run; one holds the lease and does the talking, the other waits. Seeing a quiet second pod is normal.
The console says something is wrong¶
The row sits at "waiting for your server to call home"¶
The cluster exists, but no outpost has ever connected. In order:
- Did the install finish? Re-run it; it resumes from the saved token.
- Are the pods up?
<kube> -n mesopod-system get pods. -
Can the machine reach us? The outpost dials out and nothing dials in. Two endpoints, two probes:
curl -sSf -o /dev/null https://console.mesopod.cloud/api/v1/install.sh && echo reachable openssl s_client -connect bridge.mesopod.cloud:443 -servername bridge.mesopod.cloud </dev/null 2>&1 | grep -m1 'Verify return code'The first must be a
GET— the route answers no other method. The second is the one that matters here: the console being reachable proves nothing about the bridge, and the bridge is what "waiting for your server" is waiting on. AVerify return code: 0 (ok)means the machine got a TLS session; anything else — a hang, a refused connection, a certificate from something that isn't us — is a firewall, a proxy, or DNS. Mesopod needsconsole.mesopod.cloud:443andbridge.mesopod.cloud:443outbound only.On a non-production Mesopod, substitute the origin from your own install command and its matching bridge (
console.staging.mesopod.cloudpairs withbridge.staging.mesopod.cloud). 4. Was the token revoked? Generating a new install command retires the old token, and an outpost holding it can no longer connect. Run the new command.
"Setting up" never becomes "Provisioned", or the row reads "Provision failed"¶
The first time an outpost connects, Mesopod installs the routing pieces the
cluster needs. Until that finishes, the deploy picker greys the cluster out with
still provisioning; if it fails, the row offers a retry and the picker says
provisioning failed. Retry once — and if it fails again, send us the outpost's
log. A failed provision is a bug on our side far more often than a
misconfiguration on yours.
An app reads "render failed"¶
Mesopod failed to turn the app's template and your form inputs into a set of Kubernetes objects, so nothing was ever sent to your server — your cluster is fine and untouched. Deploying again with different inputs may get past it, but a catalog app that won't render is a packaging bug on our side: tell us which app and what you typed.
An app reads "apply failed"¶
Your server refused the deploy and said why; the reason is on the app page. Note that Mesopod will not clear that state on its own — only a new deploy does. Fix the cause, then deploy again.
An app reads "degraded"¶
The app is running and at least one of its services is unhealthy. Open the app and read its logs there. A database that is still starting shows this briefly.
Host directory volumes¶
A registered directory carries a chip with the cluster's own verdict:
| Chip | What to do |
|---|---|
| Verified | Nothing. The directory exists on the node and apps can write to it. |
| Awaiting verification | No probe has run yet. Press Verify — this is the normal state of a directory you just added. |
| Awaiting cluster confirmation | Run the registration command the console shows on the cluster, then verify again. |
| Verifying… | Wait; the check is in flight. |
| Verification failed | Read the remedy line under the chip. |
The remedies, in the console's own words:
-
The registered directory doesn't resolve on the server — check the path exists on the node. Register the real path, not a symlink: the cluster compares paths as the node resolves them. On a k3d cluster, the node is a container, and it sees the Mac's
/Usersand/Volumesonly if the installer gave them to it. Check withdocker exec k3d-mesopod-<slug>-server-0 ls /Users /Volumes: if both are missing, the cluster predates that, and the fix is to re-run the install command on the Mac — no claim needed, nothing is recreated, nothing restarts:curl -fsSL https://console.mesopod.cloud/api/v1/install.sh | shIt resumes from the saved token, notices the node lacks the mounts, adds them in place (the node mounts the Mac's filesystem share itself and binds just those two paths), and keeps them across
k3d cluster stop/startand Docker restarts. Then press Verify on the directory. This is verified on OrbStack. If the script instead reports that this Docker exposes no share to the node, the node cannot reach the Mac's filesystem through Docker's sharing: on Docker Desktop, use the VirtioFS file-sharing implementation with/Usersand/Volumesin the shared paths (Settings → Resources → File sharing), then re-run; that path is not verified. The cluster must be running for the re-run to add the mounts; a stopped one is left alone and the script says so. - The directory doesn't exist under its root — create it on the server, then verify again. - The directory isn't writable by apps — fix its ownership or permissions, then verify again. - The check itself failed to run — retry, and if it persists read the outpost's log.
An app cannot attach an unverified host directory. That gate is deliberate: it is the difference between "your data is mounted" and "your app quietly started with an empty disk".
Your app's address doesn't resolve¶
At home this is nearly always the router's DNS rebind protection blocking a public name that answers with a private address. The fix, by router: install.md.
If you're on OrbStack mode, note that its LoadBalancer addresses are reachable from the Mac itself only today — reaching an app from another device on your LAN needs k3d mode instead (see install.md).
Check what your machine resolves:
dig +short photos.basement-nuc.mesopod.app
When the console shows an address for the app and this comes back empty, your
resolver dropped the private answer; the record is there. Ask another resolver
to confirm: dig +short photos.basement-nuc.mesopod.app @8.8.8.8.
Windows (WSL2)¶
The one-line installer names its own fix at every stop, so start with what it printed. The cases that are not stops:
- "Restart required. Restart Windows, then run the install command again." Not an error: WSL or the Virtual Machine Platform was just enabled and needs the restart. The command is still valid.
- The cluster is offline after a restart. The startup task could not start
WSL without a stored password on this PC. Run
& "$env:LOCALAPPDATA\Mesopod\install.ps1" -StartupTaskPassword; it asks for your Windows password once and Task Scheduler keeps it. - An app opens from your phone but not from this PC.
hostAddressLoopbackis off. The installer writes it to%UserProfile%\.wslconfig; if the file was edited since, add[experimental]/hostAddressLoopback=true, thenwsl --shutdownand start the cluster again. - An app opens from this PC but not from your phone. The Hyper-V firewall
rule
Mesopodis missing (Get-NetFirewallHyperVRule -Name Mesopod), or the console's DNS target is not the PC's address — re-run the one line (without its claim once that has been used), then check the cluster row. - "could not hold ports 80/443 in this distro". Something already listens
on 80 or 443: the message names a listener in the distro when there is one,
otherwise a Windows program (
netstat -ano | findstr ":443 "). Stop it or move it, then re-run. - A shell in the distro.
wsl -d mesopodopens one, as root. There is no other user in it by design.
Install Mesopod on Windows (WSL2) explains each piece the installer sets up and how to check it, and covers the problems that turn up after the install.
Still stuck¶
Send us the failing command's output, <kube> -n mesopod-system get pods,
and the outpost log. During alpha, every friction — however small — is worth
reporting the same day.
One thing to strip first: redact the mpc_… claim wherever it appears in
what you paste (the install command itself, a shell prompt line, an error that
echoes it back). It is a credential for enrolling this cluster, and a support
thread is not a place to leave one — replace it with mpc_…. The outpost
token never appears in the script's output, so there is nothing else to
scrub.