Troubleshoot Nest

INFO

To learn about Nest, see Nest.

Installation

Console Access

Press Ctrl+S during the installer to drop to a console.

To investigate the system state:

# check Nest service logs
journalctl -u nest-*

# check Kubernetes pod status
sudo -i
kubectl get pods -A

Cannot Resolve *.d.meltcloud.io

Some routers (e.g. Sunrise, Orange) block public domains like d.meltcloud.io from resolving to private IP addresses. This is called DNS rebinding protection. The Nest installer's DNS validation will fail with an error like "Your ISP may block public domains resolving to private IPs". Go back to the Foundry Domain screen and enable Write static DNS entries on all hosts. This makes the Nest and all enrolled Machines resolve the required subdomains internally via /etc/hosts or CoreDNS, bypassing the ISP's DNS entirely.

Your browser (and tools like Terraform) will also be unable to resolve the Foundry URL unless you use DNS-over-HTTPS. Access Foundry directly via its VIP address (e.g. https://192.168.1.252) or add a local /etc/hosts entry for app.<domain> pointing to the Foundry VIP.

Kubeadm Fails to Reach Its Own IP After Reboot

If the Foundry won't start after Nest has rebooted because kubeadm cannot reach its own IP, the hypervisor's virtual network likely moved it to a different subnet. See the platform-specific troubleshooting for workarounds:

• Nest Breaks After Host Reboot (Hyper-V)
• Nest Breaks After Reboot (macOS)

Accessing Foundry from a Cloud VM

If you're running the Nest inside a cloud VM (e.g. Azure, AWS, GCP), the Foundry URL resolves to a private IP that isn't reachable from your local machine. Use an SSH SOCKS proxy to access it through the VM:

ssh -D 1080 user@<cloud-vm-public-ip>

Then configure a browser to use localhost:1080 as a SOCKS v5 proxy. To avoid changing your default browser's settings, use a separate browser profile:

• Firefox (recommended): go to about:profiles, click Create a New Profile, and launch it. In that profile, go to Settings → General → Proxy settings, select Manual proxy configuration, set SOCKS Host to localhost, Port to 1080, select SOCKS v5, and check Proxy DNS when using SOCKS v5. Firefox has its own proxy settings, so this won't affect your other browsing.

TIP

Chrome and Edge delegate proxy configuration to the operating system on both macOS and Windows, so they cannot be configured per-profile. Use Firefox for the SOCKS proxy approach.

Navigate to the Foundry URL in the proxied browser profile. The traffic is tunneled through the cloud VM and reaches the private IP.

Install Logs of an Added Instance

The installs of added Nest Instances send their logs to Foundry: on the Nest Instance, open tab Installer Images → select the image → tab Logs.

Joining

An added Nest Instance that fails its boot shows the status Boot failed; hover over it for the reason.

• Another instance is joining: Nest Instances join one at a time. Reboot the host once the other instance is Ready.
• A Nest Instance is being reset: reboot the host once the reset is done.
• The join never finishes: reinstall the instance.

Operations

Foundry runs every change to the Nest as an operation: updates, resets, enabling high availability and rebalancing. The Last Operation on the Nest, Nest Instance and Foundry pages links to the latest one, and each of them has an Operations tab.

Refused Actions

Foundry starts resets, rebalances, updates and enabling high availability only from a healthy Nest. The following messages are shown when Foundry refuses one:

Message	What to do
Nest Instance … is not Ready	Wait until the instance comes back, or reset it
etcd member … does not answer	Wait until the instance comes back, or reset it
etcd learner … is still joining	Wait for the join to finish
… is joining / … is being reset	Wait for that instance to finish
the Nest needs at least 2 other joined Nest Instances	Create another Nest Instance before the reset
Postgres is not ready / garage is …	Wait for the Foundry to recover
… holds the only copy of …	The running instance holds the only copy of the Foundry's database or object storage (Foundry without high availability), or of a Cluster's etcd (Cluster with one control plane member). Enable high availability on the Foundry, or make the Cluster highly available, first. An instance that is down skips this check

Cluster updates are refused with The Nest is not healthy: … for the same reasons.

Warnings

• This Nest has two Nest Instances: if one fails, etcd loses quorum and the Nest fails. Create a third.
• Runs on two Nest Instances until … is installed again: a Nest Instance was reset. Install it again soon.
• This Multi-zone Foundry runs in one zone: losing that Zone takes it down. Create a Nest Instance in another Zone, then rebalance.

Nest Kubernetes API

The Nest's own Kubernetes cluster runs Foundry and the control planes. To investigate it, download a kubeconfig from Admin settings → Nest → tab Info → Kubeconfigs: Admin or Read-only. Then:

kubectl get pods -A

Terminal

Every Nest Instance has a browser terminal: Admin settings → Nest → tab Instances → select the Nest Instance → tab Terminal.

# Nest service logs
journalctl -u nest-*

# Nest Kubernetes API
sudo -i
kubectl get pods -A

# Containers
crictl ps

Serial Console

Use the serial or VGA console only when Foundry is down and the Terminal is not available. Log in with the Emergency User core and the Emergency Password from the Nest Instance's Info tab, under Linux. Note the password down while Foundry is still reachable.