socialmag/NAPKIN.md

# NAPKIN — socialmag, going programmatic

Whiteboard for the couch talk, 2026-09-12 ~03:50. Numbers here were measured
tonight on this box against abang's live Brave (`:9336`), not estimated.
Nothing in step 1–3 below is built yet.

---

## Where we are tonight

Running at home right now, all by hand:

| Piece | State |
|---|---|
| dashboard `:8321` | up — `http://127.0.0.1:8321/socialmag/dashboard.html` |
| openbridge `:8322` | up (started 03:24; it was **down**, which is why notification clicks opened new tabs instead of reusing the platform tab) |
| brave-abang CDP `:9336` | up — not a service, abang launches it |
| `refresher.py` | works, one-shot, **pressed by hand** |

Shipped tonight, unplanned:

- `facebook.check_login` and `youtube.check_login` were **lying OUT on a signed-in
  account**. Both were DOM-signal checks. Swapped to JS state (`c_user` cookie,
  `ytcfg.get('LOGGED_IN')`). 4-in/2-out → **6-in/0-out**.
- `refresher.fetch_content()` added — `notifications` and `feed` now come from a
  real scrape instead of being hand-pasted. Boris Cherny's 18h-old tweet gone;
  feed reads 22m/54m/1h.
- Backups in `Archived/` (`facebook.py`, `youtube.py`, `refresher.py`,
  `dashboard.json`, all `20260912-03*`).

---

## The delay budget — where 97 seconds goes

One full refresher pass, 8 jobs, measured 03:35:01 → 03:36:38.

| Source | Cost | Kind |
|---|---|---|
| `PostQueue._space_out()` | 3.0–6.0s **between every job** → 7 gaps ≈ 21–42s | deliberate anti-bot pacing — **keep** |
| `check_login` × 6 | navigate + `asyncio.sleep(4)` (fb/threads/twitter) or `sleep(5)` (ig/tiktok/yt) | blind sleep |
| `get_feed` | `sleep(4)` + click Following + `sleep(3)` + scroll + `sleep(2)` + `sleep(1)` = **10s fixed** | blind sleep |
| `get_notifications` | navigate + `sleep(4)` | blind sleep |
| CDP attach + activate + real navigate | ~3–8s per platform | real work |

**~55 of 97 seconds is unconditional `sleep()`.** Not a slow network — the code
asks for it.

X measured alone, warm tab, 3 runs: `get_feed` **15.0 / 11.2 / 11.1s**,
`check_login` **4.8 / 5.2 / 5.1s**. Of the 11.1s, **10.0s is fixed sleep** — the
actual DOM scrape of 10 articles takes ~1.1s.

The post flows already poll properly (`threads._wait_for_composer`,
`tiktok._wait_for_js`, `youtube._wait_for_editor_empty`). Only the **read** flows
sleep blind. History, not a technical reason.

---

## Background rendering — measured, not assumed

Question: can it work without stealing Brave's focus? Parked focus elsewhere,
attached without `activate()`:

| Platform | `visibilityState` | DOM signal | JS-state signal |
|---|---|---|---|
| twitter | `hidden` | ✅ True | ✅ True — **full scrape works**, 10 tweets, 11 articles at t=2s, flat to t=12s |
| ig | `hidden` | ✅ True | ✅ cookie `ds_user_id` |
| threads | `hidden` | 🔴 **False** | ✅ cookie `ds_user_id` |
| fb | — | 🔴 False | ✅ cookie `c_user` |
| yt | — | 🔴 False | ✅ `ytcfg.LOGGED_IN` |
| tiktok | **unproven** | True | ❌ `sessionid` is httpOnly, invisible to `document.cookie` |

🔴 **Threads is the third liar**, same shape as fb and yt: the "What's new?"
composer never mounts in a hidden tab, so a signed-in account reads as signed out.

⚠️ **TikTok could not be backgrounded.** Activated a sibling tab in the *same*
window (`Browser.getWindowForTarget` → both `windowId` 915141824) and TikTok still
reported `visible`. Either it lives in its own window, or it overrides
`visibilityState` to keep video playing. Its DOM signal has never lied — but it has
never been tested in a genuine background either. **Three states, not two: keep
`activate()` on tiktok until there is evidence.**

The headline: X renders in **2 seconds** in a hidden tab while the code sleeps
**10**. Dropping focus is not the only prize — the sleeps go with it.

---

## Plan — three steps, each stands alone

### 1 · Background pass  *(smallest, highest value; nothing else is tolerable without it)*

- `threads.check_login` → cookie. **Required** — it lies today.
- `get_or_open_tab(url, contains, activate=True)` — add the parameter, don't
  remove the behaviour. **Post flows still need foreground** for file dialogs.
- Read paths pass `activate=False`: five `check_login`s, `get_notifications`,
  `get_feed`. TikTok stays `True`.
- Fixed sleeps → `wait_for` in the three read flows.

Estimate: **97s → ~25-30s, zero tab jumps.**

### 2 · SQLite behind the refresher

Not for size — 4.2KB today, JSON holds that forever. JSON breaks because **every
refresh replaces `feed` and `notifications` wholesale**:

- No history. *"How often does a session drop?"* — the exact question README says
  we need answered before judging the VPS version — is unanswerable.
- No concept of new. Tonight's three Threads rows are yesterday's three rows with
  `11h` aged to `1d`. Nothing knows they are the same.
- 🔴 The derived `ts` decays. A row sinks an hour every hour. **`first_seen` is a
  timestamp we own** — exact, fixed, never guessed. Threads' missing clock stops
  being a problem instead of being patched.

Shape: stdlib `sqlite3`, one file, `id = sha1(platform|text|url)`,
`first_seen` / `last_seen` / `times_seen`. Refresher writes the DB, then emits
`dashboard.json` as a derived snapshot — **`dashboard.html` does not change**.

Not proposed: artifact `db` mode. The page supports it (`onSnapshot` over four
collections) but it moves the data out of the house and fixes none of the three.

### 3 · The timer

`deploy/kdmirah/socialmag-refresh.timer` already exists —
`OnUnitActiveSec=10min`, `Persistent=true`, `OnBootSec=2min`. Written for
`User=ijat` on kdmirah, **never installed here**. Adapt paths and user.

Blocked on step 1 only. Installing it today means tabs jumping every 10 minutes
all day, on the desktop abang is working at.

---

# ✅ ALL THREE STEPS BUILT — 2026-09-12 04:10-06:00

The plan above is history now. What actually shipped, and where it differs:

| Step | State |
|---|---|
| 1 · background pass | ✅ `activate=False` param on `get_or_open_tab`; threads/ig/twitter moved to JS-state signals; tiktok deliberately left foreground |
| 2 · SQLite | ✅ `store.py` — `platform_checks` / `items` / `metrics`. `dashboard.json` is a derived view; `dashboard.html` unchanged for it |
| 3 · timer | ✅ three units in `deploy/home/`, installed and verified. Refresh every **30min** (not kdmirah's 10), `--skip-foreground` |
| — | ➕ `capture.py` — the third lane, which was not in the plan at all |

**The `wait_for` rewrite of `get_feed` was never done, on purpose.** `capture`
obsoleted the DOM read path before that work would have paid for itself.

## What changed, by file

    cdp.py:179            get_or_open_tab(..., activate=True) — added, not replaced
    platforms/threads.py  DOM composer → ds_user_id cookie  (it was LYING)
    platforms/instagram.py  → ds_user_id
    platforms/twitter.py    → twid
    platforms/facebook.py   → c_user, fixed sleep dropped
    platforms/youtube.py    → ytcfg.LOGGED_IN, fixed sleep dropped
    platforms/tiktok.py     untouched; a comment says why it stays foreground
    refresher.py          parallel check_all(skip_foreground=), capture wired in,
                          store wired in
    capture.py            NEW — network capture + in-document read
    store.py              NEW — SQLite
    dashboard.html        columns swapped, feed is cross-platform, "seen X ago"
    deploy/home/*         three units

Measured: **97s → 34s** full pass, **9-10s** login-only, 3/3 stable.

## Landmines earned tonight

- 🔴 **Dropping a `sleep` is not free.** Removing X's `sleep(4)` made `check_login`
  report OUT on a signed-in account once five tabs navigated at once — the DOM
  signal had not painted. Fixed with the `twid` cookie, not by restoring the sleep.
- 🔴 **`%z` goes into `struct_time` and cannot come back out.** `_x_time_to_iso`
  emitted `2026-09-11T19:41:10` with no offset; JS `Date.parse` reads a naked
  datetime as *local*, shifting every X row by the machine's offset, silently.
  Use `datetime.strptime(...).isoformat()`.
- 🔴 **A pure time sort is not a cross-platform feed.** X returns ~185 fresh rows a
  pass against Instagram's 3 older ones, so IG existed in the database and never
  on the page. `FEED_PER_PLATFORM = 10` — representation before the merge.
- 🔴 **`ts` must be written once and never rewritten** (`COALESCE(ts, ?)`).
  Re-deriving a Threads notification's time each pass reintroduces exactly the
  decay the table exists to stop.
- 🔴 **"just now" for a post of unknown age is a lie that sorts to the top.** A
  Threads payload without `taken_at` falls back to `first_seen`; those rows read
  **"seen X ago"** now.
- ⚠️ `bravemag` `capture.screenshot` fails with an **empty** error string — the old
  stdout/stderr trap. DOM probes (`page.evaluate` for bounding boxes and counts)
  proved the layout instead, and are better evidence anyway.

---

## Open, abang's call

- **`posts` has no writer.** Only a real `create_post` can produce it, so the
  "posted" rows in the timeline stay hand-written until something records post
  results. Does `create_post` write to the DB in step 2?
- **Five platforms have no `get_notifications` at all** — `server.py:397` is
  `{"threads": threads}`. fb / ig / yt / tiktok / twitter raise. Worth building,
  or is Threads + X feed enough for the story?
- **TikTok's background behaviour** — worth one proper probe (own window? spoofed
  `visibilityState`?), or just leave `activate()` on it forever?
- **Cadence.** 10min is kdmirah's number, chosen for a box nobody watches. On this
  desktop, with step 1 done, what is it worth?
- **Units for `:8321` and `:8322`** — both hand-started today, both die on reboot.
  `deploy/kdmirah/` has both service files ready to adapt.

---

## Landmines carried forward

- 🔴 **A DOM signal on a social platform is a rendering question, not a login
  question.** Three platforms proved it tonight. Reach for JS state first.
- 🔴 **A short `sleep()` returns a short list with no error** — indistinguishable
  from "nothing happened". This is why `fetch_content` keeps old rows and records
  `_refreshed.contentErrors` instead of writing `[]`.
- ⚠️ `SOCIALMAG_CDP_PORT` is read by **nothing**. `/root/.claude.json` sets it.
  Only `SOCIALMAG_CDP_ENDPOINT` is consulted (`server.py:65`).
- ⚠️ Run as `python3 -m socialmag`, never `socialmag/server.py` — `selectors.py`
  and `queue.py` shadow stdlib modules asyncio imports.
- ⚠️ openbridge being down is **silent** by design (progressive enhancement). The
  feature looks deleted rather than broken. That is how tonight started.
- ⚠️ **Escaping a JS regex through a Python heredoc bit me tonight.**
  `"/ds_user_id=\\d+/"` reaches JS as a literal backslash and never matches, so
  three platforms polled the full 20s and reported signed-out. The first
  parallel benchmark measured three fake timeouts. A test that fails the way
  the real thing would fail is the dangerous kind.

---

# The destination — Blotato, without the API

`https://www.blotato.com` — unified social API + MCP, $29/mo, 9 platforms.
Abang's target shape. **Their moat is the OAuth app approvals**, which is their
heaviest operational cost and the thing "without an API" skips entirely.

| # | Blotato block | socialmag today | Cost without an API |
|---|---|---|---|
| 01 | PUBLISH — 9 platforms, one call | **6 platforms, shipped** | LinkedIn, Bluesky, Pinterest — one module each, pattern exists |
| 02 | SCHEDULE — queue, next open slot, weekly slots | none | **Cheap.** Local queue + timer. Never touches the browser problem |
| 03 | COMMENTS & DMs — read inbox, reply, DM | Threads notifications, `twitter.reply` | **Expensive via DOM** — every inbox is a different page. Cheap via private API (see below) |
| 04 | DM AUTOMATIONS — keyword triggers | none | Needs 03. Highest account risk of anything here |
| 05 | ANALYTICS — top posts, snapshot history | `SCRAPE_JS` already pulls likes/RT/replies | **Cheap — this is step 2.** "Snapshot history" *is* `first_seen`/`last_seen` |
| 06 | VISUALS — infographics, carousels, video | none | Separate lane. House already has geminigen, piskelmag, blendmag |
| 07 | SOURCES — YouTube/PDF/article → post | none | Separate lane. An LLM + fetch problem, not a browser one |
| 08 | API & MCP | **socialmag is already an MCP server** | Done |

Steps 1–3 aren't changed by this — they got a destination. Block 02 *is* the
timer with a queue on it. Block 05 *is* the SQLite.

🔴 **Blotato has platform permission; we do not.** They call official APIs. We
drive a browser logged in as abang, from one home IP. So throughput is not their
1,800 req/hr — it is a sequential queue with 3–6s gaps, one tab at a time, and
that gap *is* the disguise. The risk of a login challenge lands on abang's six
real accounts, not on a service account. This is why the design is **single-seat,
not SaaS.** Do not chase their numbers.

---

# Parallel — CDP supports it, and our client already does

Measured tonight, 6 platforms, all six returning True:

| | Wall |
|---|---|
| Sequential (with polling) | **19.0s** |
| Parallel (`asyncio.gather`) | **6.1s** |
| | **3.1×** |

Against the 97s the code takes today, parallel + poll + background is **~16×**.

CDP is built for this: one WebSocket, `flatten: True`, a `sessionId` per tab, an
`id` on every command. `cdp.py:127-143` already keeps futures in a dict keyed by
id with a single `_reader` dispatching replies. **Nothing needs rewriting.**

🔴 **The bigger finding is in the poll column: every signal read `0.0s`.** It was
present the moment `navigate()` returned. `asyncio.sleep(4)` has been waiting for
something that had already happened.

**Where parallel is allowed — abang's call, not a technical one:**

- `check_login` — **parallel, agreed.** Read-only, no interaction, no writes. Six
  tabs loading at once looks like a session restore. 19s → 6s.
- Post flows — **stay sequential with the gap.** That is where the pacing actually
  works, and where one mistake is outward-facing.
- `get_feed` / `get_notifications` — parallel is fine, but it is two jobs; the win
  is small.

---

# The other lane — private APIs (abang's two links)

⚠️ **`aegents/twitter-no-api` is a fork of `d60/twikit`**, created Dec 2024 and
untouched since; its README is twikit's, badges and all. Take the upstream:
**`d60/twikit` — 4,667★, pushed 2026-03-10, MIT, active.**

**`subzeroid/aiograpi-rest` — 640★, pushed 2026-05-21, MIT.** REST boundary over
`aiograpi` (the maintained async fork of `instagrapi`) = Instagram's **private
mobile API**. Full write surface: post, DM, story. First-class
`challenge_required` and 2FA handling.

These are a **different architecture**, not a socialmag upgrade:

| | socialmag | twikit / aiograpi |
|---|---|---|
| How | drives abang's already-logged-in Brave over CDP | logs in with username+password, speaks the app's own internal API |
| Stores a password? | **No** — the session belongs to Brave | **Yes**, required |
| Needs a browser / display? | Yes | **No** — headless, server, VPS |
| Data | scraped DOM | structured JSON, pydantic-typed |
| Breaks when | the platform redesigns its DOM | the platform moves an internal endpoint |
| Ban vector | traffic pattern | **unofficial client + login from a new IP** |

**Why this matters to the plan:** these libraries solve exactly what README's
"Running on a VPS" section says blocks us — no browser, no display, no tab, no
focus. Block 03 (Comments & DMs), which is expensive through the DOM, gets cheap
through `aiograpi`, because IG DMs have a real endpoint.

🔴 **The price, stated once: both require storing the real account passwords and
logging in as an unofficial client.** That is the classic path to a challenge or
a lock. socialmag never holds a password — that is *why* it is slow.

**The hybrid worth discussing on the couch:** private API for read-heavy and
scheduled work (analytics polling, inbox reads) where volume is the point, browser
for outward-facing writes where being genuinely abang's session is the point. Or
the reverse. Either way it is two lanes with one queue in front, not a migration.

## 🔴 DECIDED 2026-09-12 04:06 — both repos are OUT

Abang's call: **skip `twikit` and `aiograpi` entirely.** They log in via app
requests, and the ban risk lands on his six real accounts. The house lane does
not hold a password and does not need to. Kept in this napkin as the road not
taken, not as a backlog item.

The section below is what replaced them — and it is better than either.

---

# The third lane — structured JSON, no DOM, no password

**This is the finding of the night.** Measured 2026-09-12 04:05-04:09 against
abang's live Brave. The question was "do we still need DOM" — the answer is no,
on all three platforms tested.

The principle: the page's own app already fetches structured JSON to render
itself. **Read that, instead of reading the pixels it painted.** No credentials,
no unofficial client, no new IP — the traffic is genuinely abang's, we only
listen.

## One principle, two mechanisms

| Platform | Mechanism | Result |
|---|---|---|
| X | network capture — `/i/api/graphql/…/HomeLatestTimeline` | **190 tweets in 0.9s** |
| Threads | network capture — `/graphql/query` → `feedData` | 4 posts, 22.9KB body |
| IG | **in-document** — `RelayPrefetchedStreamCache` → `xdt_api__v1__feed__timeline` | 3 posts from a 150KB inline `<script>` |

X and Threads must be *heard*. IG must be *read* — Meta prefetches the feed into
the document, so nothing crosses the wire to capture.

## X — the same source, both ways

| | DOM scrape | Network capture |
|---|---|---|
| Tweets | 10 | **190** |
| Wall | 11.1s | **0.9s** |
| Author | `"Thariq @trq212 · 31m"` (display string) | `trq212` (real `screen_name`) |
| Counts | strings parsed out of the DOM | ints — `L=427 RT=19 RE=45` |
| Views | usually absent | **`V=17912`** |
| Posted at | `<time datetime>` attribute | full `created_at` |

IG returns the same quality: `@raybanmeta ts=1788538432 L=76919 C=945`. Exact
integers, not `"76.9K"` recovered from a rendered string.

## How it works

```
Network.enable on the tab session
  → listen for Network.responseReceived, filter the URL
  → wait for Network.loadingFinished for that requestId
  → Network.getResponseBody(requestId)
```

🔴 **`responseReceived` fires before the body has arrived.** The first probe read
`encodedDataLength` at that moment and reported 149-byte GraphQL responses for
Instagram. Waiting for `loadingFinished` turned the same Threads request into
**22,697 characters**. Sizing a body at `responseReceived` is reading a letter
while it is still in the post.

## The three real limits

1. **Capture must be live.** `getResponseBody` reads the renderer's buffer. Once
   the tab navigates away the body is gone — there is no asking later. The
   listener has to be armed *before* the request fires.
2. **We only get what the page asks for.** Want DMs? The page has to go to DMs.
   Navigation is still our job; this replaces scraping, not driving.
3. **Writes are still unanswered.** Reading is solved. Posting either stays on
   the DOM, or we capture the auth headers off one intercepted request and replay
   `fetch()` from inside the page — no password, no DOM. **Untested. It does not
   work until it has been fired.**

## Not proven — do not repeat these as facts

- ⚠️ **IG's 3 posts were all Meta-owned accounts** (`raybanmeta`, `whatsapp`,
  `metaglasses`). That is the prefetched/suggested set, **not abang's following
  feed**. His real feed most likely arrives in a later fetch — not captured yet.
- ⚠️ **Threads returned 4 posts against X's 190.** That is page one only; the rest
  comes from scroll-triggered fetches. The 190 is X being generous in one
  response, not a fair per-platform comparison.
- TikTok and YouTube have not been tried on this lane at all.

## What this does to the plan

- **Step 1 (background pass)** — unchanged and still first. Nothing here works on
  a timer while it steals focus.
- **Step 2 (SQLite)** — gets *better*. Exact integers and real epochs arrive for
  free, so analytics stops being a parsing problem. `first_seen` still solves
  Threads' missing clock for notifications.
- **Block 05 (analytics)** — close to free now. Views and counts ride along.
- **Block 03 (comments & DMs)** — possibly cheap without `aiograpi` after all,
  since the inbox pages fetch their own JSON. Worth one probe before assuming.

## Open, added tonight

- Which lane owns **writes**? DOM (proven, slow) or replayed `fetch()` with
  captured headers (fast, unproven, and the one that could post something wrong
  outward-facing)?
- Can IG's **real** following feed be captured, or is the prefetch all a logged-in
  web session gets without scrolling?
- Does the capture lane reach **DMs and comments**? One probe answers whether
  block 03 is cheap or expensive.
- TikTok and YouTube on this lane — untested.
- Where does capture live? A listener armed per call, or one long-lived recorder
  that watches abang's normal browsing and fills the DB as a side effect?