diff options
| -rw-r--r-- | doc/synoptic-morph-buildout.md | 152 |
1 files changed, 152 insertions, 0 deletions
diff --git a/doc/synoptic-morph-buildout.md b/doc/synoptic-morph-buildout.md new file mode 100644 index 0000000..77a6756 --- /dev/null +++ b/doc/synoptic-morph-buildout.md @@ -0,0 +1,152 @@ +--- +title: "Synoptic morph timelapse — PoC to production" +date-created: "2026-06-10" +type: "memoir" +status: "complete" +author: + - "st33v" + - "claude-opus-4-7" +model: + - "claude-opus-4-7" +tldr: "st33v's long-held fantasy of a slowly-morphing synoptic-chart timelapse went from idea to a wired-up production pipeline in one session. Three PoC iterations tuned framerate, dwell, fade, and pause controls; storage reality on cremonde forced a workspace migration to /mnt/enclave; final shape is a rolling 30-day cache of pair morphs at /mnt/enclave/synoptic/pairs/, triggered automatically off synoptic.service via OnSuccess=, published at pestrel.com/morph.html." +chat-url: "https://claude.ai/chat/..." +session-kind: "creative" +side-quests: 1 +reader-targets: + - "st33v" + - "claude-code" + - "scrivener" +related: + entities: + - "cremonde" + - "stan" + - "pestrel.com" + concepts: + - "rolling pair-morph cache" + - "fade-in across the loop boundary" + songs: [] +tags: + - "bom" + - "synoptic" + - "morph" + - "imagemagick" + - "ffmpeg" + - "systemd" +provenance: + - "doc/radar-four-cities-and-nav.md (earlier today)" + - "doc/bom-radar-rollout.md (previous day)" +--- + +# Synoptic morph timelapse — PoC to production + +## TL;DR + +A "just one more thing" at the end of the radar follow-up session turned into the most aesthetic piece of work in this whole arc. st33v has fantasised for years about a slowly-morphing animation of the synoptic charts; ImageMagick's `-morph` plus ffmpeg's `apng`/h264 made it almost trivial once we stopped hitting resource limits and started caching the pair morphs. The output now lives at pestrel.com/morph.html, regenerates automatically after each new chart, and uses ~1 minute of CPU per 6-hour cycle in steady state. + +## Context + +Followed straight on from the radar-four-cities-and-nav session. st33v asked, almost as an afterthought: are we discarding the synoptic charts after refresh? We weren't — 304 archived PNGs going back 75 days, plus the source PDFs. He surfaced the long-held vision: a beautiful slowly-morphing animation, mostly static (continent outline, lat/lon grid) but with fronts and pressure systems sliding across the chart between 6-hour observations. + +## The PoC arc + +Three iterations, each driven by st33v's feedback after watching the previous one. + +### PoC 1 — proving the technique + +8 charts (2 days), `-morph 11`, 12 fps. ImageMagick wrote it cleanly in 79 seconds. 7-second output. Confirmed the aesthetic instinct: continent and grid sit still, isobars slide convincingly, H/L labels smear a bit but in a way that reads as aesthetic rather than broken. st33v's "transformative work" framing dispatched the §10 copyright concern from the original radar spec — derivative enough, less worrying. + +### PoC 2 — slowing down + +> "it's hard to take it in" — st33v + +Halved the framerate to 6 fps and added a 4-frame linger on each source chart so the "real" frames dwell ≈0.8 s before morphing. Confirmed working, length doubled to ~19 s for 2 days. Added an autoplay-muted-loop HTML wrapper at `/morph-poc.html`. + +### PoC 3 — three days, smoother, fades + +st33v's framing: *"the three-day duration is quite good for putting the current frame into context."* + +Bumped to 12 fps with `-morph 23` for smoother motion within the same wall-clock duration. Hit ImageMagick's resource policy: `magick *.png -morph 23` on 12 inputs aborted with SIGABRT. Fix was structural — process pair-by-pair (`magick chartN.png chartN+1.png -morph 23 …` per gap), then concat. Same output, well under per-process limits, and *naturally cacheable* since each pair is independent. That accidental discovery shaped the production design. + +Added: 1-second fade-in from black at the start, 2-second fade-out at the end, 3-second dwell on the latest chart. The fade-in deliberately re-applies at the loop boundary so each cycle "resets" rather than jump-cuts. + +### PoC controls + +- **Pause/resume**: mouse-click on the video works; Shift-key handler is there but appears to be eaten by i3wm or Firefox in st33v's setup. Click is the reliable path; ⏸ overlay confirms the paused state. +- **Cross-page buttons**: Radars (top: 60%) and Static Chart (top: 75%) on the morph page; Radars and Monthly history on the synoptic page. All bottom-left-edge, dark translucent, dashed inner border for visual lightness. + +## The storage reckoning + +st33v mentioned "I have 34GB free at present" while we discussed PDF retention. `df` told a different story: cremonde root was at **977 MB free** (90% full), `/tmp` was tmpfs at 88% with 1.7 GB of PoC frames. The 34 GB was actually on `/mnt/enclave/` — a separate 65 GB ext4 partition, root-owned (`drwxr-xr-x 6 root root`). + +This was a tractable kind of confusion, but a useful one: storage planning for the production morph (≈7 GB of pair cache for 30 days) had to land on enclave, not root. We had st33v sudo-create `/mnt/enclave/synoptic/{pairs,frames,out}` owned by him, then moved the cached pairs out of tmpfs: + +``` +mv /tmp/morph-poc/pairs/* /mnt/enclave/synoptic/pairs/ +rm -rf /tmp/morph-poc +``` + +`/tmp` recovered to 9 MB used. The bubble was diagnosed as one-off — production writes to `/mnt/enclave/synoptic/` natively, so tmpfs won't get filled again. + +## Production shape + +Written and shipped as `e433a99`: + +- **`synopticMorph.sh`** — content-addressed pair cache at `pairs/<chartA>__<chartB>/p_NNNN.png`. Each run picks the last 120 charts, fills any missing pair (steady-state: one), evicts any pair whose slug isn't in the current window, symlinks frames into a temp `seq/` with linger/dwell, encodes via ffmpeg. `flock` guards against concurrent runs. +- **`systemd/synoptic-morph.service`** — oneshot, `TimeoutStartSec=2h` to cover the ~50-minute bootstrap, `Nice=10` and `IOSchedulingClass=idle` so it doesn't fight the radar timer or anything else for resources. +- **`synoptic.service`** gains `OnSuccess=synoptic-morph.service` — so every successful chart fetch triggers a morph re-render, asynchronously. +- **`morph.html`** at pestrel.com/morph.html → /morph.mp4; **index.html** "Monthly history" button at `top: 75%`, "Radars" at `top: 60%`. + +Bootstrap kicked off manually at end of session. ~50 min expected. + +## Resolutions + +- **Per-pair morph over bulk morph.** Forced by the policy abort, but kept because it naturally enables caching. Decisive structural win. +- **Rolling cache, not delete-after-encode.** Only the oldest pair needs to be evicted each cycle; the rest stays. Trades ~7 GB of disk on enclave for ~24 min/cycle of CPU. +- **OnSuccess= chain, not separate timer.** Morph runs strictly after a successful chart, never on a stale archive. +- **APNG/mp4 split.** Radar loops use APNG (small files, no audio track). Synoptic morph uses mp4/h264 (variable-rate would have ballooned with 4000-frame APNG; h264 keeps it small and works with native `<video>` controls). +- **Click-to-pause as the primary control.** Shift handler retained as best-effort; window manager interception is treated as the user's responsibility, not the page's. +- **`/mnt/enclave/synoptic/`** as the canonical workspace. `/tmp` reverts to tmpfs scratch. + +## Cheat sheet — debugging a future morph cycle + +```bash +# Bootstrap or refresh manually +sudo systemctl start --no-block synoptic-morph.service +journalctl -u synoptic-morph.service -f + +# Inspect cache state +ls /mnt/enclave/synoptic/pairs/ | wc -l # ~119 dirs in steady state +du -sh /mnt/enclave/synoptic/pairs/ # ~7 GB +ls /mnt/enclave/synoptic/pairs/ | sort | tail -3 # newest pairs + +# Verify output +ffprobe /srv/www/pestrel/morph.mp4 2>&1 | grep -E "Duration|Stream" + +# Tune +sudo systemctl edit synoptic-morph.service # add Environment=FPS=8 etc +``` + +The script reads `WINDOW_DAYS`, `MORPH_N`, `FPS`, `LINGER`, `DWELL`, `FADE_IN`, `FADE_OUT`, `PUBLISH_PATH` from the environment, so tuning doesn't require editing the script. + +## Operational learnings + +- **`-morph N` on a sequence has a hidden resource ceiling.** Arch's `/etc/ImageMagick-7/policy.xml` caps in-memory pixel area; pair-by-pair invocation dodges it cleanly. The error mode is SIGABRT with a one-line abort message, not a graceful failure — easy to miss in journal output. +- **The pair-morph idiom is the right primitive for caching.** Once we had it, the rolling-window design followed naturally; eviction is just "remove dirs whose slug isn't in needed_set". +- **`OnSuccess=` for chaining oneshots is the underused systemd primitive.** Synoptic→morph is a perfect fit: morph is *strictly downstream* of synoptic, runs on different timing semantics (CPU-bound rather than network-bound), and should never run on a failed chart fetch. +- **`Nice=10` + `IOSchedulingClass=idle` for low-priority background work.** Doesn't slow the visible work; only uses cycles that would otherwise sit idle. +- **Fade-in re-applies at the loop boundary in browsers.** This is exactly what we want for an autoplay-muted-loop video — gives each cycle a "breath" rather than a jump-cut. Worth designing for explicitly. +- **tmpfs is the right default for `/tmp`.** Anything that wants to persist or grow large should write to a real filesystem; the PoC putting 1.7 GB in tmpfs was diagnosed and fixed by relocating the workspace, not by changing tmpfs. + +## Appendix A — side-quest: a "latest transition" mp4 + +*Surfaced from st33v's "Also can you add the fade-in/fade out to the raw a-b morph" remark, then deferred under "leave the monthly stitch and controls for now".* + +The cleanest read: produce a small (~3 s) standalone mp4 of just the most recent 6-hour transition, fades baked in, served alongside `morph.mp4`. Cheap to add — one extra pair-dir → seq → encode pass on the most recent pair only, using the existing cached PNGs. + +Useful for: +- a "what just changed in the last 6h" view on the synoptic page, in-line below the static chart; +- embedding in social-media-style cards; +- generating the music-bed cue points later, since each transition has a stable duration. + +Don't build unless a use case crystallises. The infrastructure (cached per-pair PNGs) supports it for free. + |