From 15f3388e5ec0e524486a6e42dc2aed03e143e937 Mon Sep 17 00:00:00 2001 From: Muhammad Aizat Date: Sun, 13 Sep 2026 01:58:45 +0800 Subject: [PATCH] v0.6: cross-platform feed on the dashboard, home units, napkin MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit dashboard.html — Feed and Timeline swap places; Feed is every platform's capture now, not Twitter intel, so rows carry their own platform glyph and an unknown platform renders neutral rather than being dropped. A row whose ts is really first_seen reads "seen X ago": saying "just now" for a post of unknown age is a lie that sorts it to the top. dashboard.json — regenerated by the refresher; _yt_caveat dropped, the activate fix made it stale. deploy/home — dash, openbridge, refresh service and timer for this desktop. Refresh every 30min with --skip-foreground, not kdmirah's 10min full pass: there is a human at this box and the metrics table only needs enough resolution to watch engagement move. No Restart= on the oneshot — a down browser is a waiting state the refresher records and exits 0 for. NAPKIN.md — the whiteboard: measured delay budget, the background-render table, the three landmines earned (a DOM signal is a rendering question, %z cannot come back out of struct_time, a pure time sort is not a feed), and the open calls. Co-Authored-By: Claude Opus 5 (1M context) --- NAPKIN.md | 429 +++++++++++++++++++++++ dashboard.html | 101 +++--- dashboard.json | 386 +++++++++++++++++--- deploy/home/socialmag-dash.service | 18 + deploy/home/socialmag-openbridge.service | 19 + deploy/home/socialmag-refresh.service | 18 + deploy/home/socialmag-refresh.timer | 15 + 7 files changed, 883 insertions(+), 103 deletions(-) create mode 100644 NAPKIN.md create mode 100644 deploy/home/socialmag-dash.service create mode 100644 deploy/home/socialmag-openbridge.service create mode 100644 deploy/home/socialmag-refresh.service create mode 100644 deploy/home/socialmag-refresh.timer diff --git a/NAPKIN.md b/NAPKIN.md new file mode 100644 index 0000000..99bb28b --- /dev/null +++ b/NAPKIN.md @@ -0,0 +1,429 @@ +# NAPKIN — socialmag, going programmatic + +Whiteboard for the couch talk, 2026-09-12 ~03:50. Numbers here were measured +tonight on this box against abang's live Brave (`:9336`), not estimated. +Nothing in step 1–3 below is built yet. + +--- + +## Where we are tonight + +Running at home right now, all by hand: + +| Piece | State | +|---|---| +| dashboard `:8321` | up — `http://127.0.0.1:8321/socialmag/dashboard.html` | +| openbridge `:8322` | up (started 03:24; it was **down**, which is why notification clicks opened new tabs instead of reusing the platform tab) | +| brave-abang CDP `:9336` | up — not a service, abang launches it | +| `refresher.py` | works, one-shot, **pressed by hand** | + +Shipped tonight, unplanned: + +- `facebook.check_login` and `youtube.check_login` were **lying OUT on a signed-in + account**. Both were DOM-signal checks. Swapped to JS state (`c_user` cookie, + `ytcfg.get('LOGGED_IN')`). 4-in/2-out → **6-in/0-out**. +- `refresher.fetch_content()` added — `notifications` and `feed` now come from a + real scrape instead of being hand-pasted. Boris Cherny's 18h-old tweet gone; + feed reads 22m/54m/1h. +- Backups in `Archived/` (`facebook.py`, `youtube.py`, `refresher.py`, + `dashboard.json`, all `20260912-03*`). + +--- + +## The delay budget — where 97 seconds goes + +One full refresher pass, 8 jobs, measured 03:35:01 → 03:36:38. + +| Source | Cost | Kind | +|---|---|---| +| `PostQueue._space_out()` | 3.0–6.0s **between every job** → 7 gaps ≈ 21–42s | deliberate anti-bot pacing — **keep** | +| `check_login` × 6 | navigate + `asyncio.sleep(4)` (fb/threads/twitter) or `sleep(5)` (ig/tiktok/yt) | blind sleep | +| `get_feed` | `sleep(4)` + click Following + `sleep(3)` + scroll + `sleep(2)` + `sleep(1)` = **10s fixed** | blind sleep | +| `get_notifications` | navigate + `sleep(4)` | blind sleep | +| CDP attach + activate + real navigate | ~3–8s per platform | real work | + +**~55 of 97 seconds is unconditional `sleep()`.** Not a slow network — the code +asks for it. + +X measured alone, warm tab, 3 runs: `get_feed` **15.0 / 11.2 / 11.1s**, +`check_login` **4.8 / 5.2 / 5.1s**. Of the 11.1s, **10.0s is fixed sleep** — the +actual DOM scrape of 10 articles takes ~1.1s. + +The post flows already poll properly (`threads._wait_for_composer`, +`tiktok._wait_for_js`, `youtube._wait_for_editor_empty`). Only the **read** flows +sleep blind. History, not a technical reason. + +--- + +## Background rendering — measured, not assumed + +Question: can it work without stealing Brave's focus? Parked focus elsewhere, +attached without `activate()`: + +| Platform | `visibilityState` | DOM signal | JS-state signal | +|---|---|---|---| +| twitter | `hidden` | ✅ True | ✅ True — **full scrape works**, 10 tweets, 11 articles at t=2s, flat to t=12s | +| ig | `hidden` | ✅ True | ✅ cookie `ds_user_id` | +| threads | `hidden` | 🔴 **False** | ✅ cookie `ds_user_id` | +| fb | — | 🔴 False | ✅ cookie `c_user` | +| yt | — | 🔴 False | ✅ `ytcfg.LOGGED_IN` | +| tiktok | **unproven** | True | ❌ `sessionid` is httpOnly, invisible to `document.cookie` | + +🔴 **Threads is the third liar**, same shape as fb and yt: the "What's new?" +composer never mounts in a hidden tab, so a signed-in account reads as signed out. + +⚠️ **TikTok could not be backgrounded.** Activated a sibling tab in the *same* +window (`Browser.getWindowForTarget` → both `windowId` 915141824) and TikTok still +reported `visible`. Either it lives in its own window, or it overrides +`visibilityState` to keep video playing. Its DOM signal has never lied — but it has +never been tested in a genuine background either. **Three states, not two: keep +`activate()` on tiktok until there is evidence.** + +The headline: X renders in **2 seconds** in a hidden tab while the code sleeps +**10**. Dropping focus is not the only prize — the sleeps go with it. + +--- + +## Plan — three steps, each stands alone + +### 1 · Background pass *(smallest, highest value; nothing else is tolerable without it)* + +- `threads.check_login` → cookie. **Required** — it lies today. +- `get_or_open_tab(url, contains, activate=True)` — add the parameter, don't + remove the behaviour. **Post flows still need foreground** for file dialogs. +- Read paths pass `activate=False`: five `check_login`s, `get_notifications`, + `get_feed`. TikTok stays `True`. +- Fixed sleeps → `wait_for` in the three read flows. + +Estimate: **97s → ~25-30s, zero tab jumps.** + +### 2 · SQLite behind the refresher + +Not for size — 4.2KB today, JSON holds that forever. JSON breaks because **every +refresh replaces `feed` and `notifications` wholesale**: + +- No history. *"How often does a session drop?"* — the exact question README says + we need answered before judging the VPS version — is unanswerable. +- No concept of new. Tonight's three Threads rows are yesterday's three rows with + `11h` aged to `1d`. Nothing knows they are the same. +- 🔴 The derived `ts` decays. A row sinks an hour every hour. **`first_seen` is a + timestamp we own** — exact, fixed, never guessed. Threads' missing clock stops + being a problem instead of being patched. + +Shape: stdlib `sqlite3`, one file, `id = sha1(platform|text|url)`, +`first_seen` / `last_seen` / `times_seen`. Refresher writes the DB, then emits +`dashboard.json` as a derived snapshot — **`dashboard.html` does not change**. + +Not proposed: artifact `db` mode. The page supports it (`onSnapshot` over four +collections) but it moves the data out of the house and fixes none of the three. + +### 3 · The timer + +`deploy/kdmirah/socialmag-refresh.timer` already exists — +`OnUnitActiveSec=10min`, `Persistent=true`, `OnBootSec=2min`. Written for +`User=ijat` on kdmirah, **never installed here**. Adapt paths and user. + +Blocked on step 1 only. Installing it today means tabs jumping every 10 minutes +all day, on the desktop abang is working at. + +--- + +# ✅ ALL THREE STEPS BUILT — 2026-09-12 04:10-06:00 + +The plan above is history now. What actually shipped, and where it differs: + +| Step | State | +|---|---| +| 1 · background pass | ✅ `activate=False` param on `get_or_open_tab`; threads/ig/twitter moved to JS-state signals; tiktok deliberately left foreground | +| 2 · SQLite | ✅ `store.py` — `platform_checks` / `items` / `metrics`. `dashboard.json` is a derived view; `dashboard.html` unchanged for it | +| 3 · timer | ✅ three units in `deploy/home/`, installed and verified. Refresh every **30min** (not kdmirah's 10), `--skip-foreground` | +| — | ➕ `capture.py` — the third lane, which was not in the plan at all | + +**The `wait_for` rewrite of `get_feed` was never done, on purpose.** `capture` +obsoleted the DOM read path before that work would have paid for itself. + +## What changed, by file + + cdp.py:179 get_or_open_tab(..., activate=True) — added, not replaced + platforms/threads.py DOM composer → ds_user_id cookie (it was LYING) + platforms/instagram.py → ds_user_id + platforms/twitter.py → twid + platforms/facebook.py → c_user, fixed sleep dropped + platforms/youtube.py → ytcfg.LOGGED_IN, fixed sleep dropped + platforms/tiktok.py untouched; a comment says why it stays foreground + refresher.py parallel check_all(skip_foreground=), capture wired in, + store wired in + capture.py NEW — network capture + in-document read + store.py NEW — SQLite + dashboard.html columns swapped, feed is cross-platform, "seen X ago" + deploy/home/* three units + +Measured: **97s → 34s** full pass, **9-10s** login-only, 3/3 stable. + +## Landmines earned tonight + +- 🔴 **Dropping a `sleep` is not free.** Removing X's `sleep(4)` made `check_login` + report OUT on a signed-in account once five tabs navigated at once — the DOM + signal had not painted. Fixed with the `twid` cookie, not by restoring the sleep. +- 🔴 **`%z` goes into `struct_time` and cannot come back out.** `_x_time_to_iso` + emitted `2026-09-11T19:41:10` with no offset; JS `Date.parse` reads a naked + datetime as *local*, shifting every X row by the machine's offset, silently. + Use `datetime.strptime(...).isoformat()`. +- 🔴 **A pure time sort is not a cross-platform feed.** X returns ~185 fresh rows a + pass against Instagram's 3 older ones, so IG existed in the database and never + on the page. `FEED_PER_PLATFORM = 10` — representation before the merge. +- 🔴 **`ts` must be written once and never rewritten** (`COALESCE(ts, ?)`). + Re-deriving a Threads notification's time each pass reintroduces exactly the + decay the table exists to stop. +- 🔴 **"just now" for a post of unknown age is a lie that sorts to the top.** A + Threads payload without `taken_at` falls back to `first_seen`; those rows read + **"seen X ago"** now. +- ⚠️ `bravemag` `capture.screenshot` fails with an **empty** error string — the old + stdout/stderr trap. DOM probes (`page.evaluate` for bounding boxes and counts) + proved the layout instead, and are better evidence anyway. + +--- + +## Open, abang's call + +- **`posts` has no writer.** Only a real `create_post` can produce it, so the + "posted" rows in the timeline stay hand-written until something records post + results. Does `create_post` write to the DB in step 2? +- **Five platforms have no `get_notifications` at all** — `server.py:397` is + `{"threads": threads}`. fb / ig / yt / tiktok / twitter raise. Worth building, + or is Threads + X feed enough for the story? +- **TikTok's background behaviour** — worth one proper probe (own window? spoofed + `visibilityState`?), or just leave `activate()` on it forever? +- **Cadence.** 10min is kdmirah's number, chosen for a box nobody watches. On this + desktop, with step 1 done, what is it worth? +- **Units for `:8321` and `:8322`** — both hand-started today, both die on reboot. + `deploy/kdmirah/` has both service files ready to adapt. + +--- + +## Landmines carried forward + +- 🔴 **A DOM signal on a social platform is a rendering question, not a login + question.** Three platforms proved it tonight. Reach for JS state first. +- 🔴 **A short `sleep()` returns a short list with no error** — indistinguishable + from "nothing happened". This is why `fetch_content` keeps old rows and records + `_refreshed.contentErrors` instead of writing `[]`. +- ⚠️ `SOCIALMAG_CDP_PORT` is read by **nothing**. `/root/.claude.json` sets it. + Only `SOCIALMAG_CDP_ENDPOINT` is consulted (`server.py:65`). +- ⚠️ Run as `python3 -m socialmag`, never `socialmag/server.py` — `selectors.py` + and `queue.py` shadow stdlib modules asyncio imports. +- ⚠️ openbridge being down is **silent** by design (progressive enhancement). The + feature looks deleted rather than broken. That is how tonight started. +- ⚠️ **Escaping a JS regex through a Python heredoc bit me tonight.** + `"/ds_user_id=\\d+/"` reaches JS as a literal backslash and never matches, so + three platforms polled the full 20s and reported signed-out. The first + parallel benchmark measured three fake timeouts. A test that fails the way + the real thing would fail is the dangerous kind. + +--- + +# The destination — Blotato, without the API + +`https://www.blotato.com` — unified social API + MCP, $29/mo, 9 platforms. +Abang's target shape. **Their moat is the OAuth app approvals**, which is their +heaviest operational cost and the thing "without an API" skips entirely. + +| # | Blotato block | socialmag today | Cost without an API | +|---|---|---|---| +| 01 | PUBLISH — 9 platforms, one call | **6 platforms, shipped** | LinkedIn, Bluesky, Pinterest — one module each, pattern exists | +| 02 | SCHEDULE — queue, next open slot, weekly slots | none | **Cheap.** Local queue + timer. Never touches the browser problem | +| 03 | COMMENTS & DMs — read inbox, reply, DM | Threads notifications, `twitter.reply` | **Expensive via DOM** — every inbox is a different page. Cheap via private API (see below) | +| 04 | DM AUTOMATIONS — keyword triggers | none | Needs 03. Highest account risk of anything here | +| 05 | ANALYTICS — top posts, snapshot history | `SCRAPE_JS` already pulls likes/RT/replies | **Cheap — this is step 2.** "Snapshot history" *is* `first_seen`/`last_seen` | +| 06 | VISUALS — infographics, carousels, video | none | Separate lane. House already has geminigen, piskelmag, blendmag | +| 07 | SOURCES — YouTube/PDF/article → post | none | Separate lane. An LLM + fetch problem, not a browser one | +| 08 | API & MCP | **socialmag is already an MCP server** | Done | + +Steps 1–3 aren't changed by this — they got a destination. Block 02 *is* the +timer with a queue on it. Block 05 *is* the SQLite. + +🔴 **Blotato has platform permission; we do not.** They call official APIs. We +drive a browser logged in as abang, from one home IP. So throughput is not their +1,800 req/hr — it is a sequential queue with 3–6s gaps, one tab at a time, and +that gap *is* the disguise. The risk of a login challenge lands on abang's six +real accounts, not on a service account. This is why the design is **single-seat, +not SaaS.** Do not chase their numbers. + +--- + +# Parallel — CDP supports it, and our client already does + +Measured tonight, 6 platforms, all six returning True: + +| | Wall | +|---|---| +| Sequential (with polling) | **19.0s** | +| Parallel (`asyncio.gather`) | **6.1s** | +| | **3.1×** | + +Against the 97s the code takes today, parallel + poll + background is **~16×**. + +CDP is built for this: one WebSocket, `flatten: True`, a `sessionId` per tab, an +`id` on every command. `cdp.py:127-143` already keeps futures in a dict keyed by +id with a single `_reader` dispatching replies. **Nothing needs rewriting.** + +🔴 **The bigger finding is in the poll column: every signal read `0.0s`.** It was +present the moment `navigate()` returned. `asyncio.sleep(4)` has been waiting for +something that had already happened. + +**Where parallel is allowed — abang's call, not a technical one:** + +- `check_login` — **parallel, agreed.** Read-only, no interaction, no writes. Six + tabs loading at once looks like a session restore. 19s → 6s. +- Post flows — **stay sequential with the gap.** That is where the pacing actually + works, and where one mistake is outward-facing. +- `get_feed` / `get_notifications` — parallel is fine, but it is two jobs; the win + is small. + +--- + +# The other lane — private APIs (abang's two links) + +⚠️ **`aegents/twitter-no-api` is a fork of `d60/twikit`**, created Dec 2024 and +untouched since; its README is twikit's, badges and all. Take the upstream: +**`d60/twikit` — 4,667★, pushed 2026-03-10, MIT, active.** + +**`subzeroid/aiograpi-rest` — 640★, pushed 2026-05-21, MIT.** REST boundary over +`aiograpi` (the maintained async fork of `instagrapi`) = Instagram's **private +mobile API**. Full write surface: post, DM, story. First-class +`challenge_required` and 2FA handling. + +These are a **different architecture**, not a socialmag upgrade: + +| | socialmag | twikit / aiograpi | +|---|---|---| +| How | drives abang's already-logged-in Brave over CDP | logs in with username+password, speaks the app's own internal API | +| Stores a password? | **No** — the session belongs to Brave | **Yes**, required | +| Needs a browser / display? | Yes | **No** — headless, server, VPS | +| Data | scraped DOM | structured JSON, pydantic-typed | +| Breaks when | the platform redesigns its DOM | the platform moves an internal endpoint | +| Ban vector | traffic pattern | **unofficial client + login from a new IP** | + +**Why this matters to the plan:** these libraries solve exactly what README's +"Running on a VPS" section says blocks us — no browser, no display, no tab, no +focus. Block 03 (Comments & DMs), which is expensive through the DOM, gets cheap +through `aiograpi`, because IG DMs have a real endpoint. + +🔴 **The price, stated once: both require storing the real account passwords and +logging in as an unofficial client.** That is the classic path to a challenge or +a lock. socialmag never holds a password — that is *why* it is slow. + +**The hybrid worth discussing on the couch:** private API for read-heavy and +scheduled work (analytics polling, inbox reads) where volume is the point, browser +for outward-facing writes where being genuinely abang's session is the point. Or +the reverse. Either way it is two lanes with one queue in front, not a migration. + +## 🔴 DECIDED 2026-09-12 04:06 — both repos are OUT + +Abang's call: **skip `twikit` and `aiograpi` entirely.** They log in via app +requests, and the ban risk lands on his six real accounts. The house lane does +not hold a password and does not need to. Kept in this napkin as the road not +taken, not as a backlog item. + +The section below is what replaced them — and it is better than either. + +--- + +# The third lane — structured JSON, no DOM, no password + +**This is the finding of the night.** Measured 2026-09-12 04:05-04:09 against +abang's live Brave. The question was "do we still need DOM" — the answer is no, +on all three platforms tested. + +The principle: the page's own app already fetches structured JSON to render +itself. **Read that, instead of reading the pixels it painted.** No credentials, +no unofficial client, no new IP — the traffic is genuinely abang's, we only +listen. + +## One principle, two mechanisms + +| Platform | Mechanism | Result | +|---|---|---| +| X | network capture — `/i/api/graphql/…/HomeLatestTimeline` | **190 tweets in 0.9s** | +| Threads | network capture — `/graphql/query` → `feedData` | 4 posts, 22.9KB body | +| IG | **in-document** — `RelayPrefetchedStreamCache` → `xdt_api__v1__feed__timeline` | 3 posts from a 150KB inline `