Mesh Map Live: Implementation Notes

This document captures the state of the project and the key changes made so far, so a new Codex session can pick up without losing context. Current version: 1.8.4 (see VERSIONS.md).

Overview

This project renders live MeshCore traffic on a Leaflet + OpenStreetMap map. A FastAPI backend subscribes to MQTT (WSS/TLS or TCP), decodes MeshCore packets using the official @michaelhart/meshcore-decoder, and broadcasts device updates and routes over WebSockets to the frontend. Core logic is split into config/state/decoder/LOS/history modules so changes are localized. The UI includes heatmap, LOS tools, map mode toggles, and a 24-hour route history layer.

Versioning

VERSION.txt holds the current version string (1.8.4).
VERSIONS.md is an append-only changelog by version.

Key Paths

backend/app.py: FastAPI server + MQTT lifecycle and websocket broadcasting.
backend/config.py: environment/config constants (shared across backend modules).
backend/state.py: shared runtime state (devices/routes/history) + dataclasses.
backend/decoder.py: payload parsing, official decoder integration, route helpers.
backend/los.py: LOS math + elevation sampling.
backend/history.py: route history persistence + pruning.
backend/weather.py: weather radar country-bounds lookup router.
backend/static/index.html: HTML shell + template placeholders.
backend/static/styles.css: UI styles.
backend/static/app.js: Leaflet UI, markers, legends, routes, tools.
backend/static/sw.js: PWA service worker.
docker-compose.yaml: runtime configuration.
data/state.json: persisted device/trail/roles/names (loaded at startup).
data/route_history.jsonl: rolling 24h route history segments (lines).
data/neighbor_overrides.json: optional neighbor override pairs for route disambiguation.
.env: dev configuration (mirrors template variables).

Runtime Commands (Typical Workflow)

docker compose up -d --build (run after any file changes).
docker compose logs -f meshmap-live (watch MQTT + decode logs).
curl -s http://localhost:8080/snapshot (current device map).
curl -s http://localhost:8080/stats (counters, route types).
curl -s http://localhost:8080/debug/last (recent MQTT decode/debug entries).
curl -s http://localhost:8080/peers/<device_id> (peer counts for a node; uses route history).

Env Notes (Recent Additions)

CUSTOM_LINK_URL adds a HUD link button; blank hides it.
PACKET_ANALYZER_URL adds an external link on Route Details hashes; set it to a base such as https://analyzer.letsmesh.net/packets?packet_hash=.
QR_CODE_BUTTON_ENABLED shows a Generate QR Code button in node popups; it opens a theme-aware MeshCore-compatible contact QR modal and defaults to false.
PEERS_DEFAULT_LIMIT controls the default number of peers returned by /peers/{device_id} when no ?limit= is passed; default 8.
MAP_BOUNDARY_MODE switches geographic filtering between radius and polygon; default is radius.
MAP_BOUNDARY_FILE points at the polygon JSON file used when MAP_BOUNDARY_MODE=polygon.
MAP_BOUNDARY_SHOW controls whether the active radius/polygon boundary is drawn on the map.
MQTT_ONLINE_FORCE_NAMES forces named nodes to show MQTT online and skips them in peers.
MQTT_ONLINE_STATUS_TTL_SECONDS controls MQTT connected TTL from /status.
MQTT_ONLINE_INTERNAL_TTL_SECONDS controls MQTT connected TTL from /internal.
MQTT_ACTIVITY_PACKETS_TTL_SECONDS controls feed/activity TTL from /packets.
MQTT_STATUS_OFFLINE_VALUES forces offline when recent status payloads report those values.
MQTT_ONLINE_TOPIC_SUFFIXES remains for legacy compatibility; primary connected logic uses status/internal TTLs.
LOS_CURVATURE_ENABLED controls whether LOS includes Earth curvature; default is true.
LOS_CURVATURE_FACTOR controls the LOS effective Earth radius multiplier; default is 1.333333.
GIT_CHECK_ENABLED, GIT_CHECK_FETCH, GIT_CHECK_PATH enable update checks.
GIT_CHECK_INTERVAL_SECONDS controls how often the server re-checks for updates.
ROUTE_MAX_HOP_DISTANCE prunes hops longer than the configured km distance.
ROUTE_INFRA_ONLY limits route lines to repeaters/rooms (companions excluded from routes).
ROUTE_ALLOW_AMBIGUOUS_ONE_BYTE_FALLBACK restores the legacy route fallback for colliding 1-byte prefixes when conservative routing is too strict; default is false.
DEVICE_TTL_HOURS controls advert/device staleness (default 96 hours).
PATH_TTL_SECONDS controls path staleness (default 172800 seconds / 48h).
DEVICE_COORDS_FILE points to optional coordinate overrides (/data/device_coords.json by default).
NEIGHBOR_OVERRIDES_FILE points at an optional JSON file with neighbor pairs to resolve hash collisions.
CHANNEL_SECRETS_FILE points at an optional JSON file of channel secrets used to decrypt sender names from compatible MeshCore group-text packets.
BACKUP_ENABLED enables automatic .tar.gz backups of runtime state files.
BACKUP_INTERVAL_SECONDS controls backup frequency; default 43200 seconds (12 hours).
BACKUP_DIR controls where backup archives are written; default /backup.
BACKUP_RETENTION_DAYS controls archive pruning; set 0 to disable retention pruning.
Weather overlay settings:
- WEATHER_RADAR_ENABLED: master switch for radar layer support.
- WEATHER_RADAR_COUNTRY_BOUNDS_ENABLED: set true to clip radar tiles to resolved country bounds.
- WEATHER_RADAR_COUNTRY_LOOKUP_URL: keep /weather/radar/country-bounds unless using a custom endpoint.
- WEATHER_WIND_ENABLED: set false to disable wind in the Weather panel.
- WEATHER_WIND_API_URL: wind data endpoint (Open-Meteo format by default).
- WEATHER_WIND_GRID_SIZE: wind sample density per map view (1-5).
- WEATHER_WIND_REFRESH_SECONDS: wind refresh interval in seconds (minimum 30).
Turnstile protection is gated by PROD_MODE=true and controlled by: TURNSTILE_ENABLED, TURNSTILE_SITE_KEY, TURNSTILE_SECRET_KEY, TURNSTILE_API_URL, and TURNSTILE_TOKEN_TTL_SECONDS.
PROD_MODE/PROD_TOKEN must be passed into the container (compose now forwards them).
Turnstile auth is used for the map page + WebSocket session flow (meshmap_auth cookie or ?auth= on /ws), while protected API routes still require PROD_TOKEN.
Discord/social embeds can be preserved under Turnstile with TURNSTILE_BOT_BYPASS and TURNSTILE_BOT_ALLOWLIST.

MQTT + Decoder

MQTT supports WebSockets + TLS or plain TCP. Typical deployments use MQTT_TRANSPORT=websockets, MQTT_TLS=true, and MQTT_WS_PATH=/ or /mqtt.
MQTT_TOPIC accepts a comma-separated list, so one map can subscribe to multiple topic trees such as meshcore/BOS/#,meshcore/CON/#.
Against meshcore-mqtt-broker, the map should normally use a broker SUBSCRIBER_N username/password pair. Recommended role is 2 (full_access); role 1 is only needed when you explicitly want /internal topics or admin-only broker visibility.
The map does not generate or rotate MeshCore JWT auth tokens on its own, so node-style publisher auth (v1_<PUBLIC_KEY>) is not the normal credential flow for this app.
Decoder uses Node + the official @michaelhart/meshcore-decoder installed in the container.
The official package now handles 1-byte, 2-byte, and 3-byte repeater prefixes used by the map.
backend/decoder.py writes a small Node helper and calls it to decode MeshCore packets.
Route path decoding now accepts mixed hop prefixes: 1-byte (AB), 2-byte (ABCD), and 3-byte (ABCDEF) values in the same path during upgrade rollouts.
Ambiguous 1-byte prefixes are now handled conservatively: if multiple nodes share the same first byte, the map skips broad closest/time-based guesses and only keeps the hop when there is stronger evidence such as a unique match or neighbor/manual adjacency.

Frontend UI

Header includes a GitHub link icon and HUD summary (stats, feed note).
Base map toggle: Light/Dark/Topo; persisted to localStorage.
Dark map also darkens node popups for readability.
Node popups do not auto-pan; dragging the map won’t snap back to keep a popup in view.
Node popups now let users click the short key under the node name to copy the full public key, and can optionally expose a MeshCore-compatible contact QR modal that shows the node name plus a clickable truncated key that still copies the full public key.
Legend is collapsible and persisted to localStorage.
HUD is capped to 90vh and scrolls to avoid running off-screen.
Map start position is configurable with MAP_START_LAT, MAP_START_LON, MAP_START_ZOOM.
Radius filter: MAP_RADIUS_KM=0 disables filtering; .env.example uses 241.4 km (150mi). MAP_RADIUS_SHOW=true draws a debug circle.
Default base layer can be set with MAP_DEFAULT_LAYER (localStorage overrides).
Units toggle (km/mi) is site-wide; default from DISTANCE_UNITS and stored in localStorage.
Node size slider defaults from NODE_MARKER_RADIUS and persists in localStorage.
History link size slider defaults from HISTORY_LINK_SCALE and persists in localStorage.
Node search (name or key) and a labels toggle (persisted to localStorage).
History tool defaults off and opens a right-side panel with a heat filter slider (visibility is not persisted).
History panel can be dismissed with the X button while keeping history lines visible (toggle History tool to show it again).
History slider modes: 0 = All, 1 = Blue only, 2 = Yellow only, 3 = Yellow + Red, 4 = Red only.
History legend swatch is hidden unless the History tool is active.
Peers tool shows incoming/outgoing neighbors for a selected node, with counts and percentages pulled from dedicated rolling peer-history buckets instead of raw route-history segments.
Peer-history buckets are also updated from adjacent route point_ids when a hop cannot be rendered as a visible map segment, so peer counts do not drop to zero just because a route endpoint lacked usable coordinates.
Peers tool skips nodes listed in MQTT_ONLINE_FORCE_NAMES (observer listeners).
Peers panel legend clarifies line colors (incoming = blue, outgoing = purple).
Coverage tool only appears when COVERAGE_API_URL is set; it supports both the legacy /get-samples format and MeshMapper coverage.php grid-square responses.
Polygon boundary mode uses the same filter for devices, routes, and history, so the visible map and stored history window stay consistent.
Use map_boundary.example.json for the file format and either tools/map-boundary-builder.html or https://yellowcooln.com/map-boundary-builder/ to generate a polygon JSON file outside the live app.
MeshMapper-only coverage envs are COVERAGE_API_KEY, COVERAGE_MAX_AGE_DAYS, COVERAGE_RATE_LIMIT_COOLDOWN_SECONDS, COVERAGE_CACHE_FILE, and COVERAGE_SYNC_INTERVAL_SECONDS; the legacy coverage map does not use them.
MeshMapper coverage is synced server-side into a local cache file and served from that file to users; it also uses a cooldown after HTTP 429 rate-limit responses.
MeshMapper viewport refresh now reuses the cached expanded rectangles and only attaches or detaches visible ones, which avoids the old lag from rebuilding the full visible coverage set on every map move.
Coverage responses are filtered by COVERAGE_MAX_AGE_DAYS before they reach the map; default is 30 days, while MeshMapper can still keep the full downloaded dataset in its local cache file.
Weather is a right-side tool panel with per-layer toggles:
- Radar toggle controls RainViewer tile layer visibility.
- Wind toggle controls arrow sampling/rendering and refresh polling.
- If both radar and wind are disabled by env (WEATHER_RADAR_ENABLED=false and WEATHER_WIND_ENABLED=false), the Weather button is hidden.
Trail text in the HUD is only shown when TRAIL_LEN > 0; TRAIL_LEN=0 disables trails entirely.
Hide Nodes toggle hides markers, trails, heat, routes, and history layers.
Heat toggle can hide the heatmap; it defaults on and the button turns green when heat is off.
HUD logo uses SITE_ICON; if unset or broken it falls back to a small “Map” badge so the toggle still works.
History line weight was reduced for improved readability.
Propagation overlay keeps heat/routes/trails/markers above it after render; the panel lives on the right and retains the last render until you generate a new one.
Propagation origin markers can be removed individually by clicking them.
Propagation now supports adjustable TX antenna gain (dBi), and defaults Rx AGL to 1m.
Heatmap includes all route payload types (adverts are no longer skipped).
MQTT online status shows as a green marker outline and popup status; connected state is derived from /status and /internal timestamps, while /packets is feed activity only. Role detection is now conservative and only trusts explicit role fields and numeric role codes.
Devices that remain MQTT-online keep their last known map position until MQTT presence expires, even if fresh location packets stop arriving.
MQTT_ONLINE_FORCE_NAMES can force named nodes to show as MQTT online regardless of last seen.
PWA install support is enabled via /manifest.webmanifest and a service worker at /sw.js.
Preview image (/preview.png) renders in-bounds device dots for shared links.
Route lines prefer known neighbor pairs (including overrides) before falling back to closest-hop selection.
Clicking the HUD logo hides/shows the left panel while tool panels stay open.
Share button copies a URL with the current view + toggles (including HUD visibility).
Optional custom HUD link appears when CUSTOM_LINK_URL is set.
Update banner shows when GIT_CHECK_ENABLED=true and the repo is behind; users can dismiss it per remote SHA.
Update banner dismissal relies on .hud-update[hidden] to ensure the banner actually disappears.
URL params override stored settings: lat, lon/lng/long, zoom, layer, history, heat, coverage, weather, weather_radar, weather_wind, labels, nodes, legend, menu, units, history_filter.
Service worker uses no-store for navigation requests so env-driven UI toggles (like the radius ring) update without clearing site data.
HUD scrollbars are custom styled in Chromium for a cleaner look.

LOS (Line of Sight) Tool

LOS elevations are fetched via /los/elevations (proxy for LOS_ELEVATION_URL) and the LOS/relay/profile math runs client-side for realtime updates (fallbacks still use /los).
UI draws an LOS line (green clear / red blocked), renders an elevation profile, and marks peaks.
LOS now applies Earth curvature by default in both the frontend realtime path and the backend /los fallback, using an effective Earth radius factor of 1.333333 unless overridden by env.
When blocked, a relay suggestion marker (amber/green) highlights a potential mid-point.
Peak markers show coords + elevation and copy coords on click.
Hovering the profile or the LOS line syncs a cursor tooltip on the profile.
Hovering the LOS profile also tracks a cursor on the map and highlights nearby peaks.
LOS legend items (clear/blocked/peaks/relay) are hidden unless the LOS tool is active.
Shift+click nodes (or long‑press on mobile) or click the map repeatedly to add LOS pins and build a chained path segment-by-segment.
Drag endpoints to update LOS in realtime. Click a point marker to select it, then click the map to reposition that specific point. LOS heights are stored per pin, not as one shared A/B pair.

Device Names + Roles

Names come from advert payloads or status messages when available.
Roles are only accepted from explicit decoder fields:
- deviceRole/deviceRoleName (MeshCore advert flags), or role fields in payload.
- Name-based role heuristics were removed to avoid mislabels.
Roles are not assigned to the receiver of a packet. For decoded packets, the role applies to the advertised pubkey (decoded location.pubkey or decoded_pubkey).
Roles persist to data/state.json with device_role_sources. Only explicit/override roles are restored on load.
Optional overrides: data/device_roles.json can force roles per device_id.

Routes / Message Paths

Routes are drawn when a packet contains a path list (decoder pathHashes or path). Hop prefixes may be 1-byte, 2-byte, or 3-byte, and mixed lists are supported. When a hop hash collides, the backend prefers neighbor pairs (or overrides) before falling back to closest-hop selection; oversized path lists are ignored via ROUTE_PATH_MAX_LEN. All route hops enforce ROUTE_MAX_HOP_DISTANCE to prevent cross‑region jumps.

24h History Layer

Every route segment is persisted to data/route_history.jsonl and kept for the last ROUTE_HISTORY_HOURS.
History lines are color‑coded by volume (blue = low, orange = mid, red = high) and weight scales with counts.
History is hidden by default; the History tool opens a right panel with a slider to filter by heat band.
The History tool also includes a link size slider; it scales line weight without changing counts.
History records route modes from ROUTE_HISTORY_ALLOWED_MODES (default: path).

If routes aren’t visible:

The packet may only include a single hop (path: ["24"]).
Other repeaters might not be publishing Path/Trace packets, so a message may not include hops.
Routes and trails drop any 0,0 coordinates (including string values) and will purge bad entries on load.
Route styling uses payload type: 2/5 = Message (blue), 8/9 = Trace (orange), 4 = Advert (green).
If history is empty but routes show, confirm ROUTE_HISTORY_ALLOWED_MODES includes the active route mode.

Frontend Map UI

Legend includes Trace/Message/Advert line styles and Repeater/Companion/Room/Unknown dot colors.
Unknown markers were made more visible (larger, higher contrast gray).
Zoom control moved to bottom-right.
Route lines are thicker/bolder for large screens.
LOS + Propagation panels appear on the right; on mobile they stack to avoid overlap.

Persistence

Devices, trails, names, and roles are saved to data/state.json.
On restart, devices should stay visible if state.json exists.
Route history is persisted separately to data/route_history.jsonl (rolling window).
If stale/mis-labeled roles appear, delete data/state.json or remove role entries.
State load now removes any 0,0 coordinates from devices/trails (including string values).
When TRAIL_LEN=0, stored trails are cleared on load and no new trails are written.

Troubleshooting Notes

If map is empty but MQTT is connected, check /debug/last for decoded payloads and payloadType.
If markers appear in the wrong place, inspect decoder_meta and location fields.
If roles flip incorrectly, verify role_target_id in /debug/last.
If routes don’t show, verify message hashes appear under multiple receivers in MQTT.
If MQTT online looks wrong, verify /status and /internal topics are flowing and tune MQTT_ONLINE_STATUS_TTL_SECONDS / MQTT_ONLINE_INTERNAL_TTL_SECONDS.

Recent Fixes / Changes Summary

Added full WSS support and TLS options.
Integrated meshcore-decoder for advert/location + role parsing.
Added /stats, /snapshot, /debug/last, /debug/status endpoints.
Added persistence and state reload logic; safer role restore rules.
Added route drawing for traces/paths/messages with TTL cleanup.
UI: route legend, role legend, and improved marker styles.
Roles now apply to advertised pubkey, not receiver.
Docker restarts are required after file changes (always run docker compose up -d --build).
LOS elevations are proxied via /los/elevations and LOS/relay computations run client-side (with /los fallback).
MQTT online presence now separates connected state (/status + /internal) from feed activity (/packets) with dedicated TTL envs.
Filters out 0,0 GPS points from devices, trails, and routes (including string values).
Added 24h route history storage + history toggle with volume-based colors.
Weather backend logic moved out of app.py into backend/weather.py and mounted as a router.
Weather tool now uses its own right-side panel with separate Radar and Wind layer toggles.
Hide nodes now hides heat/routes/history along with markers/trails.
Fixed MQTT disconnect callback signature so broker drops don’t crash the MQTT loop.
Route hash collisions prefer known neighbors (or overrides) before closest-hop selection; long path lists are skipped (ROUTE_PATH_MAX_LEN).
First-hop hash collisions now prefer the closest node to the origin to avoid cross-city mis-picks (Issue #11).
Show Hops now displays plain prefix values (Prefix: AB / Prefix: ABCD / Prefix: ABCDEF) and follows backend route order directly instead of frontend reversal heuristics.
Dev-only route debugging: clicking a route line logs hop-by-hop metadata (distances, hashes, origin/receiver) to the browser console when PROD_MODE=false (PR #14, credit: https://github.com/sefator).
Trails can be disabled by setting TRAIL_LEN=0 (HUD trail text is removed).
Node marker size can be tuned via NODE_MARKER_RADIUS (users can override locally).
Units toggle defaults from DISTANCE_UNITS and persists in localStorage.
Mobile LOS selection supports long-press on nodes.
History tool visibility no longer persists (always off unless history=on in the URL).
WebSocket auth accepts ?auth=<turnstile_token> for Turnstile-gated sessions.
Protected API routes keep requiring PROD_TOKEN even when Turnstile auth is active.
/api/nodes now defaults to flat data arrays with a top-level nodes alias for client compatibility.
/api/nodes now applies updated_since automatically (use mode=full to force full snapshots).
Route IDs are observer-aware (message_hash:receiver_id) so multi-observer receptions do not overwrite each other.
ROUTE_INFRA_ONLY direct-route checks now allow rendering when at least one endpoint is infrastructure.
Propagation range math now uses a user-set TX antenna gain field; Rx AGL default lowered to 1m (credit: C2D).
Device staleness now supports a dual TTL model using both DEVICE_TTL_HOURS and PATH_TTL_SECONDS together.
Multibyte repeater-prefix support (1/2/3-byte) is in place and expected to work, but is not fully field-tested yet.

22 KiB Raw Permalink Blame History Unescape Escape