socialmag/NAPKIN.md

NAPKIN — socialmag, going programmatic

Whiteboard for the couch talk, 2026-09-12 ~03:50. Numbers here were measured tonight on this box against abang's live Brave (:9336), not estimated. Nothing in step 1–3 below is built yet.

---

Where we are tonight

Running at home right now, all by hand:

Piece	State
dashboard :8321	up — http://127.0.0.1:8321/socialmag/dashboard.html
openbridge :8322	up (started 03:24; it was down, which is why notification clicks opened new tabs instead of reusing the platform tab)
brave-abang CDP :9336	up — not a service, abang launches it
refresher.py	works, one-shot, pressed by hand

Shipped tonight, unplanned:

• facebook.check_login and youtube.check_login were lying OUT on a signed-in account. Both were DOM-signal checks. Swapped to JS state (c_user cookie, ytcfg.get('LOGGED_IN')). 4-in/2-out → 6-in/0-out.
• refresher.fetch_content() added — notifications and feed now come from a real scrape instead of being hand-pasted. Boris Cherny's 18h-old tweet gone; feed reads 22m/54m/1h.
• Backups in Archived/ (facebook.py, youtube.py, refresher.py, dashboard.json, all 20260912-03*).

---

The delay budget — where 97 seconds goes

One full refresher pass, 8 jobs, measured 03:35:01 → 03:36:38.

Source	Cost	Kind
PostQueue._space_out()	3.0–6.0s between every job → 7 gaps ≈ 21–42s	deliberate anti-bot pacing — keep
check_login × 6	navigate + asyncio.sleep(4) (fb/threads/twitter) or sleep(5) (ig/tiktok/yt)	blind sleep
get_feed	sleep(4) + click Following + sleep(3) + scroll + sleep(2) + sleep(1) = 10s fixed	blind sleep
get_notifications	navigate + sleep(4)	blind sleep
CDP attach + activate + real navigate	~3–8s per platform	real work

~55 of 97 seconds is unconditional sleep(). Not a slow network — the code asks for it.

X measured alone, warm tab, 3 runs: get_feed 15.0 / 11.2 / 11.1s, check_login 4.8 / 5.2 / 5.1s. Of the 11.1s, 10.0s is fixed sleep — the actual DOM scrape of 10 articles takes ~1.1s.

The post flows already poll properly (threads._wait_for_composer, tiktok._wait_for_js, youtube._wait_for_editor_empty). Only the read flows sleep blind. History, not a technical reason.

---

Background rendering — measured, not assumed

Question: can it work without stealing Brave's focus? Parked focus elsewhere, attached without activate():

Platform	visibilityState	DOM signal	JS-state signal
twitter	hidden	✅ True	✅ True — full scrape works, 10 tweets, 11 articles at t=2s, flat to t=12s
ig	hidden	✅ True	✅ cookie ds_user_id
threads	hidden	🔴 False	✅ cookie ds_user_id
fb	—	🔴 False	✅ cookie c_user
yt	—	🔴 False	✅ ytcfg.LOGGED_IN
tiktok	unproven	True	❌ sessionid is httpOnly, invisible to document.cookie

🔴 Threads is the third liar, same shape as fb and yt: the "What's new?" composer never mounts in a hidden tab, so a signed-in account reads as signed out.

⚠️ TikTok could not be backgrounded. Activated a sibling tab in the same window (Browser.getWindowForTarget → both windowId 915141824) and TikTok still reported visible. Either it lives in its own window, or it overrides visibilityState to keep video playing. Its DOM signal has never lied — but it has never been tested in a genuine background either. Three states, not two: keep activate() on tiktok until there is evidence.

The headline: X renders in 2 seconds in a hidden tab while the code sleeps 10. Dropping focus is not the only prize — the sleeps go with it.

---

Plan — three steps, each stands alone

1 · Background pass (smallest, highest value; nothing else is tolerable without it)

• threads.check_login → cookie. Required — it lies today.
• get_or_open_tab(url, contains, activate=True) — add the parameter, don't remove the behaviour. Post flows still need foreground for file dialogs.
• Read paths pass activate=False: five check_logins, get_notifications, get_feed. TikTok stays True.
• Fixed sleeps → wait_for in the three read flows.

Estimate: 97s → ~25-30s, zero tab jumps.

2 · SQLite behind the refresher

Not for size — 4.2KB today, JSON holds that forever. JSON breaks because every refresh replaces feed and notifications wholesale:

• No history. "How often does a session drop?" — the exact question README says we need answered before judging the VPS version — is unanswerable.
• No concept of new. Tonight's three Threads rows are yesterday's three rows with 11h aged to 1d. Nothing knows they are the same.
• 🔴 The derived ts decays. A row sinks an hour every hour. first_seen is a timestamp we own — exact, fixed, never guessed. Threads' missing clock stops being a problem instead of being patched.

Shape: stdlib sqlite3, one file, id = sha1(platform|text|url), first_seen / last_seen / times_seen. Refresher writes the DB, then emits dashboard.json as a derived snapshot — dashboard.html does not change.

Not proposed: artifact db mode. The page supports it (onSnapshot over four collections) but it moves the data out of the house and fixes none of the three.

3 · The timer

deploy/kdmirah/socialmag-refresh.timer already exists — OnUnitActiveSec=10min, Persistent=true, OnBootSec=2min. Written for User=ijat on kdmirah, never installed here. Adapt paths and user.

Blocked on step 1 only. Installing it today means tabs jumping every 10 minutes all day, on the desktop abang is working at.

---

✅ ALL THREE STEPS BUILT — 2026-09-12 04:10-06:00

The plan above is history now. What actually shipped, and where it differs:

Step	State
1 · background pass	✅ activate=False param on get_or_open_tab; threads/ig/twitter moved to JS-state signals; tiktok deliberately left foreground
2 · SQLite	✅ store.py — platform_checks / items / metrics. dashboard.json is a derived view; dashboard.html unchanged for it
3 · timer	✅ three units in deploy/home/, installed and verified. Refresh every 30min (not kdmirah's 10), --skip-foreground
—	➕ capture.py — the third lane, which was not in the plan at all

The wait_for rewrite of get_feed was never done, on purpose. capture obsoleted the DOM read path before that work would have paid for itself.

What changed, by file

cdp.py:179            get_or_open_tab(..., activate=True) — added, not replaced
platforms/threads.py  DOM composer → ds_user_id cookie  (it was LYING)
platforms/instagram.py  → ds_user_id
platforms/twitter.py    → twid
platforms/facebook.py   → c_user, fixed sleep dropped
platforms/youtube.py    → ytcfg.LOGGED_IN, fixed sleep dropped
platforms/tiktok.py     untouched; a comment says why it stays foreground
refresher.py          parallel check_all(skip_foreground=), capture wired in,
                      store wired in
capture.py            NEW — network capture + in-document read
store.py              NEW — SQLite
dashboard.html        columns swapped, feed is cross-platform, "seen X ago"
deploy/home/*         three units

Measured: 97s → 34s full pass, 9-10s login-only, 3/3 stable.

Landmines earned tonight

• 🔴 Dropping a sleep is not free. Removing X's sleep(4) made check_login report OUT on a signed-in account once five tabs navigated at once — the DOM signal had not painted. Fixed with the twid cookie, not by restoring the sleep.
• 🔴 %z goes into struct_time and cannot come back out. _x_time_to_iso emitted 2026-09-11T19:41:10 with no offset; JS Date.parse reads a naked datetime as local, shifting every X row by the machine's offset, silently. Use datetime.strptime(...).isoformat().
• 🔴 A pure time sort is not a cross-platform feed. X returns ~185 fresh rows a pass against Instagram's 3 older ones, so IG existed in the database and never on the page. FEED_PER_PLATFORM = 10 — representation before the merge.
• 🔴 ts must be written once and never rewritten (COALESCE(ts, ?)). Re-deriving a Threads notification's time each pass reintroduces exactly the decay the table exists to stop.
• 🔴 "just now" for a post of unknown age is a lie that sorts to the top. A Threads payload without taken_at falls back to first_seen; those rows read "seen X ago" now.
• ⚠️ bravemag capture.screenshot fails with an empty error string — the old stdout/stderr trap. DOM probes (page.evaluate for bounding boxes and counts) proved the layout instead, and are better evidence anyway.

---

Open, abang's call

• posts has no writer. Only a real create_post can produce it, so the "posted" rows in the timeline stay hand-written until something records post results. Does create_post write to the DB in step 2?
• Five platforms have no get_notifications at all — server.py:397 is {"threads": threads}. fb / ig / yt / tiktok / twitter raise. Worth building, or is Threads + X feed enough for the story?
• TikTok's background behaviour — worth one proper probe (own window? spoofed visibilityState?), or just leave activate() on it forever?
• Cadence. 10min is kdmirah's number, chosen for a box nobody watches. On this desktop, with step 1 done, what is it worth?
• Units for :8321 and :8322 — both hand-started today, both die on reboot. deploy/kdmirah/ has both service files ready to adapt.

---

Landmines carried forward

• 🔴 A DOM signal on a social platform is a rendering question, not a login question. Three platforms proved it tonight. Reach for JS state first.
• 🔴 A short sleep() returns a short list with no error — indistinguishable from "nothing happened". This is why fetch_content keeps old rows and records _refreshed.contentErrors instead of writing [].
• ⚠️ SOCIALMAG_CDP_PORT is read by nothing. /root/.claude.json sets it. Only SOCIALMAG_CDP_ENDPOINT is consulted (server.py:65).
• ⚠️ Run as python3 -m socialmag, never socialmag/server.py — selectors.py and queue.py shadow stdlib modules asyncio imports.
• ⚠️ openbridge being down is silent by design (progressive enhancement). The feature looks deleted rather than broken. That is how tonight started.
• ⚠️ Escaping a JS regex through a Python heredoc bit me tonight. "/ds_user_id=\\d+/" reaches JS as a literal backslash and never matches, so three platforms polled the full 20s and reported signed-out. The first parallel benchmark measured three fake timeouts. A test that fails the way the real thing would fail is the dangerous kind.

---

The destination — Blotato, without the API

https://www.blotato.com — unified social API + MCP, $29/mo, 9 platforms. Abang's target shape. Their moat is the OAuth app approvals, which is their heaviest operational cost and the thing "without an API" skips entirely.

#	Blotato block	socialmag today	Cost without an API
01	PUBLISH — 9 platforms, one call	6 platforms, shipped	LinkedIn, Bluesky, Pinterest — one module each, pattern exists
02	SCHEDULE — queue, next open slot, weekly slots	none	Cheap. Local queue + timer. Never touches the browser problem
03	COMMENTS & DMs — read inbox, reply, DM	Threads notifications, twitter.reply	Expensive via DOM — every inbox is a different page. Cheap via private API (see below)
04	DM AUTOMATIONS — keyword triggers	none	Needs 03. Highest account risk of anything here
05	ANALYTICS — top posts, snapshot history	SCRAPE_JS already pulls likes/RT/replies	Cheap — this is step 2. "Snapshot history" is first_seen/last_seen
06	VISUALS — infographics, carousels, video	none	Separate lane. House already has geminigen, piskelmag, blendmag
07	SOURCES — YouTube/PDF/article → post	none	Separate lane. An LLM + fetch problem, not a browser one
08	API & MCP	socialmag is already an MCP server	Done

Steps 1–3 aren't changed by this — they got a destination. Block 02 is the timer with a queue on it. Block 05 is the SQLite.

🔴 Blotato has platform permission; we do not. They call official APIs. We drive a browser logged in as abang, from one home IP. So throughput is not their 1,800 req/hr — it is a sequential queue with 3–6s gaps, one tab at a time, and that gap is the disguise. The risk of a login challenge lands on abang's six real accounts, not on a service account. This is why the design is single-seat, not SaaS. Do not chase their numbers.

---

Parallel — CDP supports it, and our client already does

Measured tonight, 6 platforms, all six returning True:

	Wall
Sequential (with polling)	19.0s
Parallel (asyncio.gather)	6.1s
	3.1×

Against the 97s the code takes today, parallel + poll + background is ~16×.

CDP is built for this: one WebSocket, flatten: True, a sessionId per tab, an id on every command. cdp.py:127-143 already keeps futures in a dict keyed by id with a single _reader dispatching replies. Nothing needs rewriting.

🔴 The bigger finding is in the poll column: every signal read 0.0s. It was present the moment navigate() returned. asyncio.sleep(4) has been waiting for something that had already happened.

Where parallel is allowed — abang's call, not a technical one:

• check_login — parallel, agreed. Read-only, no interaction, no writes. Six tabs loading at once looks like a session restore. 19s → 6s.
• Post flows — stay sequential with the gap. That is where the pacing actually works, and where one mistake is outward-facing.
• get_feed / get_notifications — parallel is fine, but it is two jobs; the win is small.

---

The other lane — private APIs (abang's two links)

⚠️ aegents/twitter-no-api is a fork of d60/twikit, created Dec 2024 and untouched since; its README is twikit's, badges and all. Take the upstream: d60/twikit — 4,667★, pushed 2026-03-10, MIT, active.

subzeroid/aiograpi-rest — 640★, pushed 2026-05-21, MIT. REST boundary over aiograpi (the maintained async fork of instagrapi) = Instagram's private mobile API. Full write surface: post, DM, story. First-class challenge_required and 2FA handling.

These are a different architecture, not a socialmag upgrade:

	socialmag	twikit / aiograpi
How	drives abang's already-logged-in Brave over CDP	logs in with username+password, speaks the app's own internal API
Stores a password?	No — the session belongs to Brave	Yes, required
Needs a browser / display?	Yes	No — headless, server, VPS
Data	scraped DOM	structured JSON, pydantic-typed
Breaks when	the platform redesigns its DOM	the platform moves an internal endpoint
Ban vector	traffic pattern	unofficial client + login from a new IP

Why this matters to the plan: these libraries solve exactly what README's "Running on a VPS" section says blocks us — no browser, no display, no tab, no focus. Block 03 (Comments & DMs), which is expensive through the DOM, gets cheap through aiograpi, because IG DMs have a real endpoint.

🔴 The price, stated once: both require storing the real account passwords and logging in as an unofficial client. That is the classic path to a challenge or a lock. socialmag never holds a password — that is why it is slow.

The hybrid worth discussing on the couch: private API for read-heavy and scheduled work (analytics polling, inbox reads) where volume is the point, browser for outward-facing writes where being genuinely abang's session is the point. Or the reverse. Either way it is two lanes with one queue in front, not a migration.

🔴 DECIDED 2026-09-12 04:06 — both repos are OUT

Abang's call: skip twikit and aiograpi entirely. They log in via app requests, and the ban risk lands on his six real accounts. The house lane does not hold a password and does not need to. Kept in this napkin as the road not taken, not as a backlog item.

The section below is what replaced them — and it is better than either.

---

The third lane — structured JSON, no DOM, no password

This is the finding of the night. Measured 2026-09-12 04:05-04:09 against abang's live Brave. The question was "do we still need DOM" — the answer is no, on all three platforms tested.

The principle: the page's own app already fetches structured JSON to render itself. Read that, instead of reading the pixels it painted. No credentials, no unofficial client, no new IP — the traffic is genuinely abang's, we only listen.

One principle, two mechanisms

Platform	Mechanism	Result
X	network capture — /i/api/graphql/…/HomeLatestTimeline	190 tweets in 0.9s
Threads	network capture — /graphql/query → feedData	4 posts, 22.9KB body
IG	in-document — RelayPrefetchedStreamCache → xdt_api__v1__feed__timeline	3 posts from a 150KB inline <script>

X and Threads must be heard. IG must be read — Meta prefetches the feed into the document, so nothing crosses the wire to capture.

X — the same source, both ways

	DOM scrape	Network capture
Tweets	10	190
Wall	11.1s	0.9s
Author	"Thariq @trq212 · 31m" (display string)	trq212 (real screen_name)
Counts	strings parsed out of the DOM	ints — L=427 RT=19 RE=45
Views	usually absent	V=17912
Posted at	<time datetime> attribute	full created_at

IG returns the same quality: @raybanmeta ts=1788538432 L=76919 C=945. Exact integers, not "76.9K" recovered from a rendered string.

How it works

Network.enable on the tab session
  → listen for Network.responseReceived, filter the URL
  → wait for Network.loadingFinished for that requestId
  → Network.getResponseBody(requestId)

🔴 responseReceived fires before the body has arrived. The first probe read encodedDataLength at that moment and reported 149-byte GraphQL responses for Instagram. Waiting for loadingFinished turned the same Threads request into 22,697 characters. Sizing a body at responseReceived is reading a letter while it is still in the post.

The three real limits

1. Capture must be live. getResponseBody reads the renderer's buffer. Once the tab navigates away the body is gone — there is no asking later. The listener has to be armed before the request fires.
2. We only get what the page asks for. Want DMs? The page has to go to DMs. Navigation is still our job; this replaces scraping, not driving.
3. Writes are still unanswered. Reading is solved. Posting either stays on the DOM, or we capture the auth headers off one intercepted request and replay fetch() from inside the page — no password, no DOM. Untested. It does not work until it has been fired.

Not proven — do not repeat these as facts

• ⚠️ IG's 3 posts were all Meta-owned accounts (raybanmeta, whatsapp, metaglasses). That is the prefetched/suggested set, not abang's following feed. His real feed most likely arrives in a later fetch — not captured yet.
• ⚠️ Threads returned 4 posts against X's 190. That is page one only; the rest comes from scroll-triggered fetches. The 190 is X being generous in one response, not a fair per-platform comparison.
• TikTok and YouTube have not been tried on this lane at all.

What this does to the plan

• Step 1 (background pass) — unchanged and still first. Nothing here works on a timer while it steals focus.
• Step 2 (SQLite) — gets better. Exact integers and real epochs arrive for free, so analytics stops being a parsing problem. first_seen still solves Threads' missing clock for notifications.
• Block 05 (analytics) — close to free now. Views and counts ride along.
• Block 03 (comments & DMs) — possibly cheap without aiograpi after all, since the inbox pages fetch their own JSON. Worth one probe before assuming.

Open, added tonight

• Which lane owns writes? DOM (proven, slow) or replayed fetch() with captured headers (fast, unproven, and the one that could post something wrong outward-facing)?
• Can IG's real following feed be captured, or is the prefetch all a logged-in web session gets without scrolling?
• Does the capture lane reach DMs and comments? One probe answers whether block 03 is cheap or expensive.
• TikTok and YouTube on this lane — untested.
• Where does capture live? A listener armed per call, or one long-lived recorder that watches abang's normal browsing and fills the DB as a side effect?