# NAPKIN — socialmag, going programmatic Whiteboard for the couch talk, 2026-09-12 ~03:50. Numbers here were measured tonight on this box against abang's live Brave (`:9336`), not estimated. Nothing in step 1–3 below is built yet. --- ## Where we are tonight Running at home right now, all by hand: | Piece | State | |---|---| | dashboard `:8321` | up — `http://127.0.0.1:8321/socialmag/dashboard.html` | | openbridge `:8322` | up (started 03:24; it was **down**, which is why notification clicks opened new tabs instead of reusing the platform tab) | | brave-abang CDP `:9336` | up — not a service, abang launches it | | `refresher.py` | works, one-shot, **pressed by hand** | Shipped tonight, unplanned: - `facebook.check_login` and `youtube.check_login` were **lying OUT on a signed-in account**. Both were DOM-signal checks. Swapped to JS state (`c_user` cookie, `ytcfg.get('LOGGED_IN')`). 4-in/2-out → **6-in/0-out**. - `refresher.fetch_content()` added — `notifications` and `feed` now come from a real scrape instead of being hand-pasted. Boris Cherny's 18h-old tweet gone; feed reads 22m/54m/1h. - Backups in `Archived/` (`facebook.py`, `youtube.py`, `refresher.py`, `dashboard.json`, all `20260912-03*`). --- ## The delay budget — where 97 seconds goes One full refresher pass, 8 jobs, measured 03:35:01 → 03:36:38. | Source | Cost | Kind | |---|---|---| | `PostQueue._space_out()` | 3.0–6.0s **between every job** → 7 gaps ≈ 21–42s | deliberate anti-bot pacing — **keep** | | `check_login` × 6 | navigate + `asyncio.sleep(4)` (fb/threads/twitter) or `sleep(5)` (ig/tiktok/yt) | blind sleep | | `get_feed` | `sleep(4)` + click Following + `sleep(3)` + scroll + `sleep(2)` + `sleep(1)` = **10s fixed** | blind sleep | | `get_notifications` | navigate + `sleep(4)` | blind sleep | | CDP attach + activate + real navigate | ~3–8s per platform | real work | **~55 of 97 seconds is unconditional `sleep()`.** Not a slow network — the code asks for it. X measured alone, warm tab, 3 runs: `get_feed` **15.0 / 11.2 / 11.1s**, `check_login` **4.8 / 5.2 / 5.1s**. Of the 11.1s, **10.0s is fixed sleep** — the actual DOM scrape of 10 articles takes ~1.1s. The post flows already poll properly (`threads._wait_for_composer`, `tiktok._wait_for_js`, `youtube._wait_for_editor_empty`). Only the **read** flows sleep blind. History, not a technical reason. --- ## Background rendering — measured, not assumed Question: can it work without stealing Brave's focus? Parked focus elsewhere, attached without `activate()`: | Platform | `visibilityState` | DOM signal | JS-state signal | |---|---|---|---| | twitter | `hidden` | ✅ True | ✅ True — **full scrape works**, 10 tweets, 11 articles at t=2s, flat to t=12s | | ig | `hidden` | ✅ True | ✅ cookie `ds_user_id` | | threads | `hidden` | 🔴 **False** | ✅ cookie `ds_user_id` | | fb | — | 🔴 False | ✅ cookie `c_user` | | yt | — | 🔴 False | ✅ `ytcfg.LOGGED_IN` | | tiktok | **unproven** | True | ❌ `sessionid` is httpOnly, invisible to `document.cookie` | 🔴 **Threads is the third liar**, same shape as fb and yt: the "What's new?" composer never mounts in a hidden tab, so a signed-in account reads as signed out. ⚠️ **TikTok could not be backgrounded.** Activated a sibling tab in the *same* window (`Browser.getWindowForTarget` → both `windowId` 915141824) and TikTok still reported `visible`. Either it lives in its own window, or it overrides `visibilityState` to keep video playing. Its DOM signal has never lied — but it has never been tested in a genuine background either. **Three states, not two: keep `activate()` on tiktok until there is evidence.** The headline: X renders in **2 seconds** in a hidden tab while the code sleeps **10**. Dropping focus is not the only prize — the sleeps go with it. --- ## Plan — three steps, each stands alone ### 1 · Background pass *(smallest, highest value; nothing else is tolerable without it)* - `threads.check_login` → cookie. **Required** — it lies today. - `get_or_open_tab(url, contains, activate=True)` — add the parameter, don't remove the behaviour. **Post flows still need foreground** for file dialogs. - Read paths pass `activate=False`: five `check_login`s, `get_notifications`, `get_feed`. TikTok stays `True`. - Fixed sleeps → `wait_for` in the three read flows. Estimate: **97s → ~25-30s, zero tab jumps.** ### 2 · SQLite behind the refresher Not for size — 4.2KB today, JSON holds that forever. JSON breaks because **every refresh replaces `feed` and `notifications` wholesale**: - No history. *"How often does a session drop?"* — the exact question README says we need answered before judging the VPS version — is unanswerable. - No concept of new. Tonight's three Threads rows are yesterday's three rows with `11h` aged to `1d`. Nothing knows they are the same. - 🔴 The derived `ts` decays. A row sinks an hour every hour. **`first_seen` is a timestamp we own** — exact, fixed, never guessed. Threads' missing clock stops being a problem instead of being patched. Shape: stdlib `sqlite3`, one file, `id = sha1(platform|text|url)`, `first_seen` / `last_seen` / `times_seen`. Refresher writes the DB, then emits `dashboard.json` as a derived snapshot — **`dashboard.html` does not change**. Not proposed: artifact `db` mode. The page supports it (`onSnapshot` over four collections) but it moves the data out of the house and fixes none of the three. ### 3 · The timer `deploy/kdmirah/socialmag-refresh.timer` already exists — `OnUnitActiveSec=10min`, `Persistent=true`, `OnBootSec=2min`. Written for `User=ijat` on kdmirah, **never installed here**. Adapt paths and user. Blocked on step 1 only. Installing it today means tabs jumping every 10 minutes all day, on the desktop abang is working at. --- # ✅ ALL THREE STEPS BUILT — 2026-09-12 04:10-06:00 The plan above is history now. What actually shipped, and where it differs: | Step | State | |---|---| | 1 · background pass | ✅ `activate=False` param on `get_or_open_tab`; threads/ig/twitter moved to JS-state signals; tiktok deliberately left foreground | | 2 · SQLite | ✅ `store.py` — `platform_checks` / `items` / `metrics`. `dashboard.json` is a derived view; `dashboard.html` unchanged for it | | 3 · timer | ✅ three units in `deploy/home/`, installed and verified. Refresh every **30min** (not kdmirah's 10), `--skip-foreground` | | — | ➕ `capture.py` — the third lane, which was not in the plan at all | **The `wait_for` rewrite of `get_feed` was never done, on purpose.** `capture` obsoleted the DOM read path before that work would have paid for itself. ## What changed, by file cdp.py:179 get_or_open_tab(..., activate=True) — added, not replaced platforms/threads.py DOM composer → ds_user_id cookie (it was LYING) platforms/instagram.py → ds_user_id platforms/twitter.py → twid platforms/facebook.py → c_user, fixed sleep dropped platforms/youtube.py → ytcfg.LOGGED_IN, fixed sleep dropped platforms/tiktok.py untouched; a comment says why it stays foreground refresher.py parallel check_all(skip_foreground=), capture wired in, store wired in capture.py NEW — network capture + in-document read store.py NEW — SQLite dashboard.html columns swapped, feed is cross-platform, "seen X ago" deploy/home/* three units Measured: **97s → 34s** full pass, **9-10s** login-only, 3/3 stable. ## Landmines earned tonight - 🔴 **Dropping a `sleep` is not free.** Removing X's `sleep(4)` made `check_login` report OUT on a signed-in account once five tabs navigated at once — the DOM signal had not painted. Fixed with the `twid` cookie, not by restoring the sleep. - 🔴 **`%z` goes into `struct_time` and cannot come back out.** `_x_time_to_iso` emitted `2026-09-11T19:41:10` with no offset; JS `Date.parse` reads a naked datetime as *local*, shifting every X row by the machine's offset, silently. Use `datetime.strptime(...).isoformat()`. - 🔴 **A pure time sort is not a cross-platform feed.** X returns ~185 fresh rows a pass against Instagram's 3 older ones, so IG existed in the database and never on the page. `FEED_PER_PLATFORM = 10` — representation before the merge. - 🔴 **`ts` must be written once and never rewritten** (`COALESCE(ts, ?)`). Re-deriving a Threads notification's time each pass reintroduces exactly the decay the table exists to stop. - 🔴 **"just now" for a post of unknown age is a lie that sorts to the top.** A Threads payload without `taken_at` falls back to `first_seen`; those rows read **"seen X ago"** now. - ⚠️ `bravemag` `capture.screenshot` fails with an **empty** error string — the old stdout/stderr trap. DOM probes (`page.evaluate` for bounding boxes and counts) proved the layout instead, and are better evidence anyway. --- ## Open, abang's call - **`posts` has no writer.** Only a real `create_post` can produce it, so the "posted" rows in the timeline stay hand-written until something records post results. Does `create_post` write to the DB in step 2? - **Five platforms have no `get_notifications` at all** — `server.py:397` is `{"threads": threads}`. fb / ig / yt / tiktok / twitter raise. Worth building, or is Threads + X feed enough for the story? - **TikTok's background behaviour** — worth one proper probe (own window? spoofed `visibilityState`?), or just leave `activate()` on it forever? - **Cadence.** 10min is kdmirah's number, chosen for a box nobody watches. On this desktop, with step 1 done, what is it worth? - **Units for `:8321` and `:8322`** — both hand-started today, both die on reboot. `deploy/kdmirah/` has both service files ready to adapt. --- ## Landmines carried forward - 🔴 **A DOM signal on a social platform is a rendering question, not a login question.** Three platforms proved it tonight. Reach for JS state first. - 🔴 **A short `sleep()` returns a short list with no error** — indistinguishable from "nothing happened". This is why `fetch_content` keeps old rows and records `_refreshed.contentErrors` instead of writing `[]`. - ⚠️ `SOCIALMAG_CDP_PORT` is read by **nothing**. `/root/.claude.json` sets it. Only `SOCIALMAG_CDP_ENDPOINT` is consulted (`server.py:65`). - ⚠️ Run as `python3 -m socialmag`, never `socialmag/server.py` — `selectors.py` and `queue.py` shadow stdlib modules asyncio imports. - ⚠️ openbridge being down is **silent** by design (progressive enhancement). The feature looks deleted rather than broken. That is how tonight started. - ⚠️ **Escaping a JS regex through a Python heredoc bit me tonight.** `"/ds_user_id=\\d+/"` reaches JS as a literal backslash and never matches, so three platforms polled the full 20s and reported signed-out. The first parallel benchmark measured three fake timeouts. A test that fails the way the real thing would fail is the dangerous kind. --- # The destination — Blotato, without the API `https://www.blotato.com` — unified social API + MCP, $29/mo, 9 platforms. Abang's target shape. **Their moat is the OAuth app approvals**, which is their heaviest operational cost and the thing "without an API" skips entirely. | # | Blotato block | socialmag today | Cost without an API | |---|---|---|---| | 01 | PUBLISH — 9 platforms, one call | **6 platforms, shipped** | LinkedIn, Bluesky, Pinterest — one module each, pattern exists | | 02 | SCHEDULE — queue, next open slot, weekly slots | none | **Cheap.** Local queue + timer. Never touches the browser problem | | 03 | COMMENTS & DMs — read inbox, reply, DM | Threads notifications, `twitter.reply` | **Expensive via DOM** — every inbox is a different page. Cheap via private API (see below) | | 04 | DM AUTOMATIONS — keyword triggers | none | Needs 03. Highest account risk of anything here | | 05 | ANALYTICS — top posts, snapshot history | `SCRAPE_JS` already pulls likes/RT/replies | **Cheap — this is step 2.** "Snapshot history" *is* `first_seen`/`last_seen` | | 06 | VISUALS — infographics, carousels, video | none | Separate lane. House already has geminigen, piskelmag, blendmag | | 07 | SOURCES — YouTube/PDF/article → post | none | Separate lane. An LLM + fetch problem, not a browser one | | 08 | API & MCP | **socialmag is already an MCP server** | Done | Steps 1–3 aren't changed by this — they got a destination. Block 02 *is* the timer with a queue on it. Block 05 *is* the SQLite. 🔴 **Blotato has platform permission; we do not.** They call official APIs. We drive a browser logged in as abang, from one home IP. So throughput is not their 1,800 req/hr — it is a sequential queue with 3–6s gaps, one tab at a time, and that gap *is* the disguise. The risk of a login challenge lands on abang's six real accounts, not on a service account. This is why the design is **single-seat, not SaaS.** Do not chase their numbers. --- # Parallel — CDP supports it, and our client already does Measured tonight, 6 platforms, all six returning True: | | Wall | |---|---| | Sequential (with polling) | **19.0s** | | Parallel (`asyncio.gather`) | **6.1s** | | | **3.1×** | Against the 97s the code takes today, parallel + poll + background is **~16×**. CDP is built for this: one WebSocket, `flatten: True`, a `sessionId` per tab, an `id` on every command. `cdp.py:127-143` already keeps futures in a dict keyed by id with a single `_reader` dispatching replies. **Nothing needs rewriting.** 🔴 **The bigger finding is in the poll column: every signal read `0.0s`.** It was present the moment `navigate()` returned. `asyncio.sleep(4)` has been waiting for something that had already happened. **Where parallel is allowed — abang's call, not a technical one:** - `check_login` — **parallel, agreed.** Read-only, no interaction, no writes. Six tabs loading at once looks like a session restore. 19s → 6s. - Post flows — **stay sequential with the gap.** That is where the pacing actually works, and where one mistake is outward-facing. - `get_feed` / `get_notifications` — parallel is fine, but it is two jobs; the win is small. --- # The other lane — private APIs (abang's two links) ⚠️ **`aegents/twitter-no-api` is a fork of `d60/twikit`**, created Dec 2024 and untouched since; its README is twikit's, badges and all. Take the upstream: **`d60/twikit` — 4,667★, pushed 2026-03-10, MIT, active.** **`subzeroid/aiograpi-rest` — 640★, pushed 2026-05-21, MIT.** REST boundary over `aiograpi` (the maintained async fork of `instagrapi`) = Instagram's **private mobile API**. Full write surface: post, DM, story. First-class `challenge_required` and 2FA handling. These are a **different architecture**, not a socialmag upgrade: | | socialmag | twikit / aiograpi | |---|---|---| | How | drives abang's already-logged-in Brave over CDP | logs in with username+password, speaks the app's own internal API | | Stores a password? | **No** — the session belongs to Brave | **Yes**, required | | Needs a browser / display? | Yes | **No** — headless, server, VPS | | Data | scraped DOM | structured JSON, pydantic-typed | | Breaks when | the platform redesigns its DOM | the platform moves an internal endpoint | | Ban vector | traffic pattern | **unofficial client + login from a new IP** | **Why this matters to the plan:** these libraries solve exactly what README's "Running on a VPS" section says blocks us — no browser, no display, no tab, no focus. Block 03 (Comments & DMs), which is expensive through the DOM, gets cheap through `aiograpi`, because IG DMs have a real endpoint. 🔴 **The price, stated once: both require storing the real account passwords and logging in as an unofficial client.** That is the classic path to a challenge or a lock. socialmag never holds a password — that is *why* it is slow. **The hybrid worth discussing on the couch:** private API for read-heavy and scheduled work (analytics polling, inbox reads) where volume is the point, browser for outward-facing writes where being genuinely abang's session is the point. Or the reverse. Either way it is two lanes with one queue in front, not a migration. ## 🔴 DECIDED 2026-09-12 04:06 — both repos are OUT Abang's call: **skip `twikit` and `aiograpi` entirely.** They log in via app requests, and the ban risk lands on his six real accounts. The house lane does not hold a password and does not need to. Kept in this napkin as the road not taken, not as a backlog item. The section below is what replaced them — and it is better than either. --- # The third lane — structured JSON, no DOM, no password **This is the finding of the night.** Measured 2026-09-12 04:05-04:09 against abang's live Brave. The question was "do we still need DOM" — the answer is no, on all three platforms tested. The principle: the page's own app already fetches structured JSON to render itself. **Read that, instead of reading the pixels it painted.** No credentials, no unofficial client, no new IP — the traffic is genuinely abang's, we only listen. ## One principle, two mechanisms | Platform | Mechanism | Result | |---|---|---| | X | network capture — `/i/api/graphql/…/HomeLatestTimeline` | **190 tweets in 0.9s** | | Threads | network capture — `/graphql/query` → `feedData` | 4 posts, 22.9KB body | | IG | **in-document** — `RelayPrefetchedStreamCache` → `xdt_api__v1__feed__timeline` | 3 posts from a 150KB inline `