Homelabs fail in specific, dateable ways. This is a running record of what broke, how it was actually diagnosed, and the exact fix — not generic best-practice advice.
On the Pi's 6.18 kernel, pwm-ir-tx sends every edge of an IR code within microseconds, and ir-ctl takes the right amount of time to report success
With dtoverlay=pwm-ir-tx on Raspberry Pi OS's 6.18 kernel, a 100 ms pulse, 100 ms space, 100 ms pulse came out as four PWM state changes within 7 microseconds, while ir-ctl exited 0 after about 350 ms, the length of t...
On an AppArmor host, rootful Podman restarts a --uidmap container once and then leaves it stopped, logging that AppArmor is disabled
On Debian 13 with Debian's podman 5.4.2 and AppArmor enabled, a rootful container with --uidmap/--gidmap and --restart always came back once and then stayed exited, while the same container without the mapping, or wit...
llama-server's RAM prompt cache brings a prompt back with KV computed under a different LoRA scale, and answers from it
llama-server's RAM prompt cache, on by default, stores a slot's KV without recording which LoRA scale it was computed with, and the check for a LoRA change compares against the slot's previous request, not the entry i...
Ollama picks the Gemma 4 prompt template from the model's name, so a copy of gemma4:12b is prompted as an e2b
Ollama 0.35.1 resolves RENDERER gemma4 to the small (e2b/e4b) or large (12b and up) template by looking for a size in the model's name, then by parameter count, and the 12B counts as 11.9B. So ollama cp gemma4:12b myg...
With speculative decoding on, llama-server's logprobs are placeholders: 0.0 and no alternatives for every token after the first
llama-server emits every token that comes out of its speculative decoding loop with logprob 0.0 and an empty top_logprobs list, and never fills them in. With a draft model that is every token after the first. On b1143...
With think enabled, Ollama only enforces your JSON schema after the model has thought, so a direct answer comes back as plain text
Since 0.34.4, Ollama applies format on a thinking model as one grammar: free text until the thinking block closes, then the schema. The grammar lets the response end before the closing, so when Gemma 4 skips thinking ...
vLLM's weight cache daemon gives an engine another checkpoint's weights when the two have the same tensor layout
vLLM 0.30.0's weight cache (load_format ipc_cache) decides whether the daemon's weights match the engine's checkpoint by hashing safetensors headers: names, shapes, dtypes, offsets, no values. With the daemon holding ...
vLLM serves a LoRA adapter that uses rank_pattern or alpha_pattern at the wrong scale, and says nothing
PEFT scales each LoRA module by its own alpha / r, which rank_pattern and alpha_pattern can set per module. vLLM 0.30.0 reads one r and one lora_alpha and applies lora_alpha / r to everything. On an RTX 2070, an adapt...
On one of Ollama's two chat paths, your JSON schema's keys are re-sorted alphabetically, and the model fills the fields in the wrong order
Ollama 0.35 sends models without a usable Go template through a native llama-server chat path that decodes the response schema into a Go map and re-encodes it, which sorts the keys. With OLLAMA_GO_TEMPLATE=false, a sc...
llama-server's sleep mode loses, or crashes on, a request that arrives in the last few dozen milliseconds before it sleeps
With --sleep-idle-seconds, a request handler checks that the server is awake, tokenizes the prompt, then queues the task. If the idle timer fires in between, the server unloads the model and sleeps. A 17,653-token pro...
A Docker swarm service failing on a taken host port cut every container on the node off its overlay network, and docker ps still said running
The overlay driver destroys a network's sandbox on a node when its join count reaches zero, and every container start that fails after the driver's Join takes one too many off that count. A swarm service publishing a ...
Rootless `podman export` of a `--userns=keep-id` container shifts every file owner in the tarball, and exits 0
A rootless container created with --userns=keep-id stores its files under the intermediate namespace's IDs, and podman export tars that storage without mapping them back. On a Pi with podman 5.4.2, the user's home cam...
Ollama's OpenAI-compatible API replaces your model's temperature and top_p with 1.0. `ollama show` keeps printing the model's values.
A request to /v1/chat/completions that doesn't set temperature or top_p gets 1.0 for both, and that overrides the model's PARAMETER lines. qwen3:0.6b ships temperature 0.6 and top_p 0.95; llama-server ran it at 0.600/...
llama.cpp picks stop tokens partly by their spelling. On PLaMo the closing strikethrough tag is ordinary text, and output stops at it.
llama.cpp adds any vocab entry spelled like an end-of-sequence marker to its stop list and forces it to a control token, even when the model marks it NORMAL. In PLaMo-2 and PLaMo-3 the closing strikethrough tag is exa...
Ollama 0.34 logs your OLLAMA_GPU_OVERHEAD reservation as applied. The llama-server that places the layers never receives it.
Ollama 0.34 runs GGUF models through a bundled llama-server, and with the default num_gpu the layer placement is llama-server's --fit, which keeps its own 1 GiB margin. OLLAMA_GPU_OVERHEAD is read, printed, and subtra...
podman cp on a stopped container ignores the volume's subpath: it reads a different file, writes outside the container's view, and overwrote another container's config. Exit 0 every time.
A named volume mounted with subpath= shows the container one directory of the volume. While the container runs, podman cp respects that. When it is created or stopped, podman resolves the path against the volume's roo...
The Laya checkpoint still ships a 0.1 temperature. laya clamps it when it loads; an ONNX port that reads the file does not, and its 15-option answers come back 0.88 sure when they are wrong.
Laya's English checkpoint carries a softmax temperature of 0.1006 for choice questions with 11 or more options, which sharpens the logits about ten times. laya fixed it on 2026-09-21 by clamping temperatures to [0.5, ...
llama-server's `--cache-ram -1` is documented as "no limit". On a dense model it keeps less than the default; on a hybrid model it has no memory limit at all.
The prompt cache has two limits, bytes and tokens. -1 removes the byte limit, and the token limit, which a positive setting raises to fit the bytes, stays pinned at n_ctx. With 16 distinct prompts on a 4096-token cont...
One failed WPA handshake and a headless Pi on Trixie stays off WiFi until someone types a command — its NetworkManager has no retry for PSK, asks for a key nobody can give, and blocks the profile.
NetworkManager 1.52.1 on Raspberry Pi OS Trixie: fail one 4-way handshake on a profile that has worked for weeks, and 3.6 seconds later NM is 'asking for new key' — the only retry setting is documented as 802.1X-only ...
Send Podman's Docker-compatible API an update that only changes memory, and the container's restart policy is now 'no'. HTTP 200, no warning, and the container stays down after the next crash or reboot.
Podman 5.4.2 (Debian 13), compat API v1.41: POST /containers/{id}/update with any body that omits RestartPolicy — a memory limit, a CpuShares value, an empty {} — returns 200 {"Warnings":null} and quietly sets the res...
Ollama's gemma4 renderer never shows the model a tool parameter named type, description, required, properties or nullable — it stays in the required list, and the model makes a value up.
Same tool schema, temperature 0, only the parameter renamed. Called kind, the model returns the enum value urgent_A7 — a string that exists only in the parameter's definition. Called type, it returns "urgent": a plaus...
llama.cpp reads past your pos array on every embedding batch for an M-RoPE model — the header says n_tokens, the batch code reads four times that, and the logits change with the heap.
Same batch, same context, memory cleared, one thread: token ids in, bitwise identical every run; embedding vectors in, Qwen3.5-0.8B, five runs, five different logits and the argmax moves. It looked like the weight typ...
llama.cpp's batch API says pass pos as NULL and positions are tracked automatically. For M-RoPE embeddings it then reads past its own buffer, and nothing reports it.
An embeddings batch with pos = NULL on a Qwen2.5-VL-style model gets a position vector sized n_tokens, and the batch splitter reads n_tokens × 4 from it. AddressSanitizer on today's master: heap-buffer-overflow, READ ...
Ollama's Responses API accepts previous_response_id, returns 200 and "completed", and starts every turn from nothing.
Plant a secret word in turn 1, ask for it in turn 2 with previous_response_id, and the model guesses. The request costs exactly as many input tokens as a request with no history at all — 41 and 41 — while sending the ...
llama-server accepts the json_schema form its own README documents, returns 200, and generates as if you had sent no response_format at all.
Send {"type":"json_schema","schema":{...}} — the example on the server README — and the reply is byte-identical to sending no response_format. The parser's json_schema branch only ever reads json_schema.schema, the 20...
docker logs stops at the first NUL byte in the file, returns exit 0, and --since after a crash tells you nothing happened.
One record of a json-file log overwritten with NUL bytes — what an abrupt power loss leaves behind — and the full read returns 17 of 50 lines with exit 0 and an empty stderr. --tail works only if its window starts aft...
Raspberry Pi OS ships a meta-data file whose instance-id key cloud-init never reads. Every user-data you write after the first boot is parsed, merged, and then skipped.
The stock /boot/firmware/meta-data sets instance_id with an underscore. cloud-init reads instance-id with a hyphen, so the value is never seen and the id falls back to the constant nocloud. One boot with the stock all...
vLLM accepts a LoRA adapter it has already decided not to apply anywhere, and answers every request with the base model instead.
Restrict a deployment with --lora-target-modules and load an adapter whose modules fall outside that list, and nothing objects. The adapter loads, the LoRA kernels compile, and every response is the unmodified base mo...
An Ollama library tag that pulls, runs, and streams fluent text with no working code in it. The same model at the same quantisation, converted by someone else, is fine.
qwen2.5-coder:3b-instruct-q3_K_M downloads cleanly, loads at normal speed and answers every prompt. It scored 0/3 on trivial coding tasks here. The q4_K_M sibling scored 3/3, and so did the official Qwen GGUF at the s...
llama-cli reports a commit hash that is not in llama.cpp. It belongs to whatever repository you unpacked the source inside.
A release tarball extracted anywhere inside an unrelated git work tree makes CMake stamp that repository's HEAD into the binary. Configure exits 0 with no warning, the dirty flag stays silent because untracked files a...
llama-cli prints the error and exits 0. The same binary exits 1 when the model is missing.
A missing --image or --audio path is printed as an error and then reported as success. It is not that the exit status is unreliable in general: the same binary propagates a missing model correctly. One path prints and...
The same ssh_import_id works in one cloud-init file and does nothing in the next. The difference is one letter.
A top-level ssh_import_id is read only for a user marked default, and a nested one only for a user that is not. Raspberry Pi's documented users: example and Imager's generated user: block sit on opposite sides of that...
iptables says your kernel needs upgrading. Upgrading the kernel is what broke it.
The Pi's 6.18 kernels no longer build the legacy ip_tables modules, but CONFIG_IP_NF_IPTABLES=m is still set — because the symbol that builds them is now called something else. The module directory still looks full, a...
tar backed up your Pi's WiFi config and exited 0. The archive holds one empty directory.
On a stock Raspberry Pi OS Trixie image, /etc/NetworkManager/system-connections is empty — and it is the directory every guide names. The connection you actually care about, WiFi password included, lives in /etc/netpl...
Podman publishes the same port and your firewall still holds. Even as root.
A commenter said rootless Podman avoids the Docker-bypasses-ufw problem because it has no root to write firewall rules with. The conclusion is right and the reason isn't: running Podman as root still leaves ufw in con...
UFW says the port is closed. Docker published it to your whole network anyway.
A stock Raspberry Pi with ufw enabled blocks an unlisted port exactly as advertised. Publish the same port from a container and it answers from anywhere on the LAN, while ufw status keeps reporting deny (incoming). Do...
Your Pi's 2 GB swap file isn't swap, and no swap command will tell you that
Trixie replaced dphys-swapfile with rpi-swap, which defaults to a zram+file hybrid. /var/swap still exists and still costs you 2 GB of disk, but it is zram's writeback target, not a swap area — so swapon, free and /pr...
Your Pi accepts every memory limit you set and enforces none of them
The memory cgroup controller is off by default on Raspberry Pi OS, so docker --memory, systemd MemoryMax and k3s limits are all accepted and silently ignored. The firmware injects cgroup_disable=memory ahead of your c...
A new NetworkManager property broke a decade of headless WiFi scripts
nmcli 1.52 added autoconnect-ports, which made the long-standing abbreviation autoconnect-p ambiguous. Provisioning scripts using it now abort with exit 2, and because nmcli connection modify is all-or-nothing, the SS...
sysctl -p says it worked. The next reboot disagrees.
On Raspberry Pi OS Trixie, /etc/sysctl.conf does not exist and is not read at boot — systemd-sysctl only reads the .d directories. The procps sysctl CLI still reads it, so sysctl -p applies your change and reports suc...
Raspberry Pi OS Trixie throws away your logs on reboot, and /var/log/journal exists anyway to reassure you it doesn't
Stock Trixie ships a vendor drop-in setting Storage=volatile, so the journal lives in RAM and dies on reboot. The empty /var/log/journal directory sitting there makes it look like persistence is already on. And the fi...
Stripe retries a failed webhook for three days, and I found out by getting the same email three times
A webhook that failed once during setup kept being retried in the background. Once the underlying bug was fixed, every one of those retries succeeded — and every one of them sent a real customer the same email again.
Undervoltage doesn't look like a power problem. It looks like bad Wi-Fi.
A Raspberry Pi 4B refused to hold a Wi-Fi connection, wouldn't finish a login, and behaved like it had a driver fault. The actual cause was a power supply that couldn't hold 5V — and the firmware had been saying so in...
Your Discord bot is ignoring you, and the status output is lying about why
A self-hosted agent bot connected fine, posted fine, and silently discarded every message aimed at it. The status line blamed a Discord intent. The actual cause was an unset applicationId — and the logs said so all al...
The rpi-clone PARTUUID trap: why your Pi 4B silently refuses to boot from a cloned SSD
The unmaintained rpi-clone tries to fix the root=PARTUUID reference in the destination's cmdline.txt — but looks for it at the pre-Bookworm path, finds a placeholder file, and silently does nothing. Flip the boot orde...