One section per failure mode. If your container is labeled but unreachable,
start with Container not routed. To have proximo look
for you, run proximo doctor: it reports every check
on this host, and each failure carries the command that clears it.
Docker is the one mandatory prerequisite: without it there is no stack, no
routes, and nothing for proximo to read. Every command that needs it stops with
cannot reach the Docker daemon (is Docker running?).
docker version
Its own output names the cause — Docker Desktop not started, a DOCKER_HOST
pointing at a context that is gone, or a user not in the docker group. Start
Docker (or point DOCKER_HOST at a context that exists) and re-run.
Trust and DNS are host changes, and they are made by one verb the developer types. Until then the local CA is not on disk and nothing points the TLD at proximo's DNS server, so names do not resolve and certificates are not trusted — however healthy the stack is.
proximo install
proximo doctor reports this as one failure and skips everything that
depends on it, rather than repeating the same cause a dozen times. The
pre-install checks still run, so a machine that is also missing Docker says so
before the install that would fail on it. What install changes is listed in
What install changes on your host.
https://<name>.test does not resolve — confirm the stack is up
(proximo status) and the resolver is wired: on macOS check
/etc/resolver/<tld>; on Linux check
/etc/systemd/resolved.conf.d/proximo-<tld>.conf and that systemd-resolved is
active (resolvectl status). Only systemd-resolved is supported in v1.
To test the DNS server directly, bypassing the host resolver:
dig @127.0.0.1 -p 5354 whoami.test
On macOS, a name looked up before proximo install wired the resolver can sit
in the system cache as a failure. Flush it and ask again:
sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder
scutil --dns | grep -A 3 "domain.*test" # the resolver is in effect
If that answers 127.0.0.1 but the browser still fails, the host resolver is
the problem — see also
VPN or corporate DNS overrides the resolver.
proximo install and proximo up abort before changing anything if
127.0.0.1:5354/udp is held by something other than proximo. (Port 5353 is
deliberately not used: macOS mDNSResponder/Bonjour binds it.) The port held by
proximo's own DNS service is the healthy case and stops nothing.
The check names the holder it can name, and hands over the question for the one it cannot:
docker ps --filter publish=5354 # another container publishes it
sudo lsof -nP -iUDP:5354 # a process on this host holds it
Traefik publishes 443 and 80 on 127.0.0.1. If another service already
listens there (another reverse proxy, a local web server, an old Traefik), the
stack fails to start with a Docker error like
Bind for 127.0.0.1:443 failed: port is already allocated. On Linux a listener
on every interface (0.0.0.0:443) holds the loopback address too.
proximo install and proximo up check this before touching anything, and
proximo doctor reports it any time. All three ask who holds the port rather
than whether it is free — a healthy machine has :443 bound, by proximo — so
the answer also picks the command that names the holder:
docker ps --filter publish=443 # another container publishes it
sudo lsof -nP -iTCP:443 -sTCP:LISTEN # a process on this host holds it
Stop the conflicting listener, then bring the stack back up with proximo up.
127.0.0.1:5354/udp relies on the Docker VM forwarding UDP. It works on
current Docker Desktop; if a setup proves unreliable, that is the first thing
to check.
A browser that flags https://<name>.<tld> as untrusted
(ERR_CERT_AUTHORITY_INVALID / "issuer not trusted") almost always means the
local CA is not in the store the browser reads — Firefox and Chrome on Linux use
NSS, not the system store; Chrome and Safari on macOS use the system keychain.
First fully restart the browser (NSS loads CAs only at process start). If the warning persists, re-trust the CA:
proximo trust
proximo checks both stores separately, so
proximo doctor says which one is missing the CA —
and it compares the certificate, not its name, so a CA that was regenerated
while an older namesake is still in the store is reported as untrusted rather
than as fine.
The NSS store needs certutil. Where it is absent, proximo installs it through
Homebrew or apt; on a host with neither, install says so before changing
anything, and the remedy is to install the NSS tooling with your own package
manager (libnss3-tools on Debian-family distributions, nss-tools elsewhere).
proximo trust re-adds the CA to the system and NSS
stores via certutil (installing nss-tools if needed). It touches neither DNS
nor the stack, so it works while proximo is up, without a down/up cycle.
Restart the browser again afterwards. To confirm the CA landed in both stores,
run proximo doctor.
proximo refuses to run under sudo (why). A
version that did not refuse used root's home on Ubuntu: it created a local CA in
/root/.proximo, trusted it in place of yours, and never installed a team root
configured for you. On Linux, as yourself:
proximo trust # puts your local CA back in the system store
sudo rm -rf /root/.proximo # root's CA and key, no longer trusted
Older versions logged a recurring, harmless error on every container restart:
ERR Unable to parse certificate .../certs/<name>.crt error="... tls: failed to find any PEM data in certificate input"
It came from a race: the watcher rewrote a .crt with a plain truncate-then-write
while Traefik's file provider reloaded mid-write, briefly reading an empty file.
Certificate materialization is now atomic (write-temp-then-rename), so the proxy
only ever sees a complete file and this error no longer appears on reissue.
If you still see it, the cert is genuinely malformed — a corrupted CA or a
hand-edited file under certs/. Re-run proximo install to regenerate the CA,
then restart the stack.
Gatekeeper's "is damaged" dialog is handled by the cask. For a manually
downloaded binary: xattr -dr com.apple.quarantine ./proximo.
The watcher logs every routing decision it skips (invalid hostnames, ambiguous ports, duplicate hosts across label schemes). When a doc says "a warning is logged", this is where to look:
cd ~/.proximo/stack && docker compose logs watcher
With observability enabled (proximo up --observability),
the same logs are browsable at https://logs.<tld> (Dozzle) — select the
watcher container.
Every Incident the
watcher records is logged there too, prefixed Incident: — the same facts
proximo errors lists, in the order they arrived:
cd ~/.proximo/stack && docker compose logs watcher | grep Incident:
A container with proximo labels is not reachable. Check in order:
proximo.hosts must be present (its presence is
the opt-in) and contain valid hostnames under the configured TLD,
comma-separated. Invalid entries are rejected and logged.proximo.enable=false/0/no parks the container.
Remove the label or set it truthy.EXPOSEs zero or several ports and no
proximo.port is set, the container is skipped; proximo status shows it
flagged (⚠ set proximo.port). Add an explicit proximo.port.proximo status lists it with the
reason and the winner: see
A host collision is reported.The full label contract is in Routing.
Two containers claim one host — two worktrees of a repository, or a stale
container nobody stopped. proximo does not pick a winner quietly: the container
that did not get the host is listed by proximo status with the reason and the
name of the container serving it.
CONTAINER URL
shop-api-1 https://api.test + api.shop.test
work-api-1 ⚠ api.test is served by shop-api-1; this container answers at api.work.test
work-api-1 https://admin.test + admin.work.test
Read it as three facts:
admin.test above) and
still answers at its own qualified host: a collision costs a bare host, never a
service. Two containers of one project share the qualified host too, so
there the loser is left with nothing and the note says so instead — give one of
them a different proximo.hosts.proximo.hosts, or simply use the qualified host and
leave both running.Two variants of the same line say proximo stood down rather than arbitrated — one of the two claims was its own:
qualified host api.shop.test withdrawn: <name> serves it — someone declared
by hand the name proximo would have generated. The hand-written declaration wins.api.test is matched by a traefik.* rule on <name>; proximo withdrew its router — a native Traefik label already routes that host, whether on that
container or on another. The host stays served, and status lists it under
the container carrying the rule. Use one scheme per host
(native Traefik labels).A container outside a Compose project has no Namespace, so it has no qualified host to fall back on and the note says so. Put it in a Compose project and it gets one.
A container that is up but still booting (DB migrations, JIT warmup, slow start)
gets traffic the instant it is running and answers 502/503 until it is
actually ready — most visible on a frequently-restarting container, which
repeats the failed-request window on every cycle.
Declare a Docker HEALTHCHECK on the image (or healthcheck: in Compose).
proximo then gates the route on health: it publishes the router + certificate
only once the container reports healthy and withdraws them when it turns
unhealthy, so requests never reach a backend that cannot serve. While the
container is starting, proximo status lists it as starting (waiting for healthy) rather than as a working URL.
proximo.health=false opts out and routes on running. See
proximo.health.It never will, and nothing is broken. An expression that throws in Chrome's
console is caught by DevTools itself, so window.onerror never fires — and that
is what the agent listens to. The same is true of anything you try/catch
yourself: a handled error is not an uncaught one.
Throw from a real task instead, which is what a broken app does:
setTimeout(function(){ null.foo }, 0)
Every route produces an Exchange, whether or not it is inspected, because
Traefik records an access log. A stack brought up before that existed records
none, and proximo errors is then silent for a reason that has nothing to do
with your code — it says so inline, and proximo doctor reports it as The
running stack records access logs.
$ proximo update
Traefik reads its static configuration once, at startup, so a stack keeps
running without the access log however new the file on disk is. proximo update
restarts it; proximo doctor confirms.
If the check passes and a route you called still shows nothing, the request
never reached proximo. proximo doctor tells apart "nothing called it" from
"the name does not resolve here", which are the two causes and look identical
from the listing.
An Incident is what the
runtime declared about a container: a non-zero exit, a restart, an OOM kill. The
watcher records them and publishes them on a loopback port; a stack brought up
before that existed publishes nothing, and proximo errors is then silent about
a container that keeps dying for a reason that has nothing to do with your code.
It says so inline, and proximo doctor reports it as The running stack records
Incidents.
$ proximo update
If the Check passes and a container you expected still produces no Incident, work through these in order:
proximo.transcript=true. proximo status lists
it as no route — observed for Incidents once the label takes effect, which
is how you verify the label at all.--since defaults to 15 minutes, and the
watcher keeps a bounded number of Incidents per service, in memory: they are
lost when the stack is recreated.proximo errors --service <service> answers with the
readings —
running since when, what the healthcheck says, how many restarts, when its
output last moved — and stops there rather than concluding, because an idle
consumer and a stuck one read identically. Read the output for the window
yourself: proximo errors transcript --service <service> --since 30m.start_period is what keeps a slow first job from being reported unhealthy
on the way there.A Transcript is quoted from the container's own output at the moment you ask, never stored. Five outcomes are not failures, and each says which it is:
"wrote nothing while this request was live" — the container was up and quiet. Nothing to fix.
"has written nothing at all since it started, so it probably logs elsewhere" — the project writes to a file inside the container, or to a collector. Only stdout and stderr can be quoted. Point the application's logger at stdout and the transcripts fill in.
"log driver cannot be read back" — the container runs a logging driver
Docker cannot replay (syslog, fluentd, gelf). Docker's own json-file
and local drivers can; the rest cannot, by design.
"the container that served this request is gone" — it was removed
since, or the address now answers for a container started after the request.
A container that merely stopped is still quoted, because Docker still answers
docker logs for it. proximo says so rather than quoting whichever container
holds the address now: attributing another container's stack trace to your
request is the one failure worth being silent about.
"proximo remembers this Incident, not what … wrote" — the Incident is runtime metadata proximo kept; the output it would quote is the project's, and proximo holds none of it. The container is gone, or the one answering to that name now started after the Incident, so there is nothing left to read back. This is the declared price of remembering only what the runtime says, and proximo says it rather than quoting whichever container holds the name today.
A transcript can also say N other request(s) overlapped this one. The cut is
temporal — it is what the container wrote in this window, never what this request
caused — so when two requests to one container overlap, their lines interleave
and nothing after the fact can separate them. Reproduce the problem on its own,
or read the whole transcript with proximo errors transcript <id>.
If you were testing by hand from the console, read the entry above first — that accounts for most of it. Otherwise work down this list; each step is visible without guessing.
This entry is about the Client reports half — what the browser saw. The backend half needs no label: read the Transcript of any route.
proximo status lists the route;
proximo.inspect is ignored on a TCP (SNI) route and on a
replica set. The watcher logs why —
see Where to read watcher warnings.</head>. The agent is inserted before the
closing head tag. A response without one is left untouched, and the Exchange
for it carries a warning saying so.inspector container, and an inspected route then 502s. proximo update./.proximo/agent.js. If it is in the HTML but the browser did not run it, the
page's Content-Security-Policy blocked it — the Exchange will say proximo
relaxed the policy, and if it does not, the policy came from somewhere proximo
cannot rewrite (a <meta http-equiv> tag in your own markup).proximo errors says so when the hop came up in the last ten minutes.--since defaults to 15 minutes and
follows the report, not the page load, so a page opened an hour ago that threw
a moment ago still appears. --all adds the Exchanges with nothing wrong, which
is worth a look when you want to confirm the route is being served through the
hop at all.On a shared route:
proximo reserves the path prefix /.proximo/ on the origin of an inspected route
— that is where the page reports to, same-origin. If your project serves anything
under that prefix, it is unreachable for as long as the route carries the label.
Remove proximo.inspect and the prefix is yours again.
VPN clients and corporate DNS setups often install their own resolver
configuration with higher precedence, so .test queries never reach proximo's
DNS server even though the stack is healthy.
scutil --dns: the
domain : test entry must point at 127.0.0.1 port 5354. Some VPN clients
install scoped resolvers that shadow /etc/resolver/<tld>.resolvectl status: the VPN interface may claim a routing
domain of ~. (everything), which outranks the proximo drop-in. Restrict the
VPN's DNS scope, or re-run proximo install after connecting.In both cases dig @127.0.0.1 -p 5354 <name>.test answering correctly proves
the problem is resolver precedence, not proximo's DNS server.
A mesh nameserver configured as the machine's primary resolver is one more
such cause. It captures every query no rule matches, .test included, and it
breaks every local route on the machine. The mesh's nameserver configuration must
never be primary (constraint 6).
A connection from inside a container to an address the host reaches fine — a service behind a VPN, a remote Docker network over a tunnel — resolves, then hangs until it times out. On Docker Desktop the host's own terminal is unaffected, which is what makes it hard to attribute: the stack network's bridge lives inside the Linux VM, so only containers lose the route.
The usual cause is the stack network
occupying a subnet the host already routes: packets for that subnet match the
stack network's route and never leave the host
(ADR 0009).
proximo doctor states the stack network's subnet on every run:
✔ The stack network is on the declared subnet — 10.101.0.0/24
It reports a different subnet — typically a /16 in 172.17.0.0/16–
172.31.0.0/16, inherited from Docker's default pool by a stack created
before proximo declared its own. proximo up recreates the network on the
declared subnet; the stack's services restart once.
The declared subnet itself overlaps the route — compare it with the
address you cannot reach, or with your VPN's routes (netstat -rn on macOS,
ip route on Linux). Move the stack elsewhere:
proximo config subnet 10.102.0.0/24
The value must be a private IPv4 CIDR between /16 and /30, outside
Docker's default pool. If Docker refuses it (Pool overlaps with other one on this address space), the stack goes back to the subnet it had and the config
is unchanged — pick another.
proximo never reads the host routing table: it cannot tell you which route collides, only which subnet the stack holds.
Routes stop updating (new containers are not picked up, removed containers still resolve) when the watcher is not running — it is the reconcile loop, so without it the routing files and certificates go stale.
proximo doctor # names the service that is missing
docker ps --filter "label=proximo.role" # traefik, dns and watcher should be up
proximo up # converges the stack back to healthy
Stale routes left behind by a crash are cleaned up on the next reconcile after
the watcher restarts. If proximo doctor reports a version skew between the
CLI and the stack, run proximo update — see Updating.
--image is sticky: it is recorded in the stack's .env and
survives until an up or update without it. A stack running an overridden ref
is running one thing while the CLI pins another — fine while you are testing a
build, and a surprise weeks later.
✘ The stack runs the image this CLI pins
the stack runs proximo:src, this CLI pins ghcr.io/filippolmt/proximo:v0.4.0
Remedy: proximo up
proximo up (without --image) restores the pinned image. To keep the
override, keep passing the flag — see
Running a different image.
A converge that leaves the stack image absent ends with a Remedy naming it:
the stack image ghcr.io/filippolmt/proximo:v0.4.0 is not on this host.
Remedy: docker pull ghcr.io/filippolmt/proximo:v0.4.0
Run that pull. proximo does not claim the pull is what broke — other services can fail the same converge — but the pull's own output settles it, and it is almost always one of three:
ghcr.io (offline, VPN, proxy). Then the pinned Traefik and
dashboard images cannot be fetched either; nothing else is wrong, and the next
proximo up converges once the registry is reachable.denied / unauthorized. The image tag for this CLI version has not been
published yet (a release still building), or the package is no longer public.
A published version tag is never deleted, so a tag that used to work and now
denies is a package-visibility problem, not a missing image.manifest unknown. You are on a CLI built from an unreleased ref. Use a
published version, or point the stack at an image you have:
proximo up --image <ref> — see
Running a different image.proximo never falls back to building the image on your host: that retry needs the same network that just failed, and on the rare success it would leave you running an image nobody else has.
proximo doctor reports the copies of the agent skill proximo
installed and found behind the binary, or edited since:
✘ The agent skill matches the installed CLI
~/.claude/skills/proximo was written by another version of proximo
Remedy: proximo skill install --scope global
See: https://github.com/filippolmt/proximo/blob/main/docs/troubleshooting.md#the-agent-skill-is-out-of-date
install, up and update already refresh every copy they can see, so a copy
this Check names is one they could not reach:
--scope project copy only while proximo runs inside that
repository, so a repository left alone holds a stale copy until someone
returns to it. Run proximo skill install there.skill install
defaults to project scope, so the Remedy names --scope global when the copy
it is about lives under your home. A failure spanning both scopes takes two
runs, and the next report names what is left.proximo skill install --force, which discards the local edits — the only
command that does.Two answers are not failures. A copy with no .proximo-skill.json beside it is
unmanaged — installed from a marketplace or by an agent's own installer — so
proximo neither updates nor removes it, and the Check names it rather than
claiming a version it cannot know:
– The agent skill matches the installed CLI — proximo installed no agent skill
on this host (~/.claude/skills/proximo came from another channel, and proximo
neither updates nor removes it)
And a host with no copy at all skips the Check with nothing to name: a developer who uses no coding agent should never see a red line about one.
After any of these, restart the agent session — a skill is read when the session starts.
A timeout with no page is what a machine that has not admitted you looks like:
nothing listens below TLS. The causes, none of them a proximo fault and none of
them affecting the sharing machine's own .test routes:
On the sharing machine, proximo doctor reports mesh.
.test address that does not resolveThe browser shows DNS_PROBE_FINISHED_NXDOMAIN for a .test host. The app
built an absolute URL — a redirect, usually after login — from a configured base
URL instead of from the request. See property 1 of
what an app needs to be shareable.
The login appears to succeed and the next page is anonymous. The app pinned its
session cookie to Domain=<name>.test, which the browser silently refuses on the
peer name. In the other direction, a session started on the .test host never
follows onto the peer name: one login per name. A colleague's app that pins a
cookie to the Peer suffix can also overwrite your session, because the whole mesh
is one site. See property 2 of
what an app needs to be shareable.
The route is shared and the request reached it:
proximo.path prefix is shared, so the root answers 404, and
proximo status does not print paths;404;proximo.inspect on, /.proximo/ is reserved on the peer names too;An unshared route or an invented name does not answer 404; it fails at the
certificate (next section).
Read what the certificate names:
proximo.share; the label is set and a value is missing (the sharing machine's
proximo status names it); the declared host is outside the TLD; the container
is not running.CSSMERR_TP_NOT_TRUSTED,
Chrome shows a warning that cannot be clicked through, Firefox
SEC_ERROR_UNKNOWN_ISSUER. Install it
(enrolling a colleague). On Linux a Chrome
that had never started leaves it out with no error: its certificate store did
not exist yet. Run proximo trust once Chrome has started once. A
sudo proximo trust from before proximo refused sudo also left it out
(cleaning up).peer-intermediate fails a month ahead.doctor Check reports it: the browser that matters is on the colleague's
machine.-2147409643, the same
for a DNS and an IP violation), SecureTransport flattens it to unable to get local issuer certificate, Chrome shows the generic ERR_CERT_INVALID, Firefox
SEC_ERROR_CERT_NOT_IN_NAME_SPACE — even before the root is installed — and
OpenSSL permitted subtree violation or excluded subtree violation. Any
diagnostic that matches one alert string is wrong on some client. It means a
certificate was issued outside the machine's subtree, which the configuration
refusals exist to prevent.The path between the two machines is relayed — behind a phone hotspot or carrier-grade NAT it usually is, and stays so. That is a normal operating condition and there is no per-network fix to apply. On a relayed path a new connection costs from a few hundred milliseconds up to about a second, with visible jitter. Later requests reuse the HTTP/2 connection and skip the handshake, so it feels like a distant server, not like localhost.
peer-routes failed: a label on a shared route cannot take effect. Either a
declared host is outside the TLD, so it has no peer names, or the route is a TCP
route, which is never shared. docker inspect <container> shows the labels; drop
proximo.share or the TCP label, or move the host under the TLD.
peer-dns or peer-relay failed. The peer DNS service is not answering on
<address>:5354, or the peer relay does not
complete a TLS handshake on <address>:443. The causes are the same for both:
proximo up retries now.proximo up.config address is not this machine's address on the mesh.dig @<address> -p 5354 proximo-doctor.<machine>.<suffix>
for peer-dns, openssl s_client -connect <address>:443 -servername proximo-doctor.<machine>.<suffix> for peer-relay.Until this passes, the nameserver group that routes the subtree to this machine resolves nothing — which is why it is created only after this passes.
mesh failed while peer-dns passed: proximo answers, and the system resolver
does not send it the query. The Remedy shows which resolver handles the Peer
suffix and where it points. The causes are the mesh's: its client is down or
logged out, the nameserver group for <machine>.<suffix> does not exist or
forwards to the wrong port, or this machine is not in the group it is
distributed to — see
a publishing machine is not a resolving peer.
mesh resolves the machine's own names, so a publishing machine outside the
resolving peers fails it even while colleagues reach it. Being a resolving peer
is a requirement for publishing: add the machine to the set the subtrees are
distributed to.
peer-intermediate failed: the intermediate expires in under 30 days, or has.
Run the ceremony again — proximo config csr, have it signed, proximo config intermediate <file>. The machine label does not change, so every bookmark keeps
working.