juxtapo/socialmag
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.6: cross-platform feed on the dashboard, home units, napkin

Browse Source 
dashboard.html — Feed and Timeline swap places; Feed is every platform's
capture now, not Twitter intel, so rows carry their own platform glyph and an
unknown platform renders neutral rather than being dropped. A row whose ts is
really first_seen reads "seen X ago": saying "just now" for a post of unknown
age is a lie that sorts it to the top.

dashboard.json — regenerated by the refresher; _yt_caveat dropped, the
activate fix made it stale.

deploy/home — dash, openbridge, refresh service and timer for this desktop.
Refresh every 30min with --skip-foreground, not kdmirah's 10min full pass:
there is a human at this box and the metrics table only needs enough
resolution to watch engagement move. No Restart= on the oneshot — a down
browser is a waiting state the refresher records and exits 0 for.

NAPKIN.md — the whiteboard: measured delay budget, the background-render
table, the three landmines earned (a DOM signal is a rendering question, %z
cannot come back out of struct_time, a pure time sort is not a feed), and the
open calls.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
juxtapo
2026-09-13 01:58:45 +08:00
co-authored by Claude Opus 5
parent 22e63ebf35
commit 15f3388e5e
7 changed files with 883 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files
NAPKIN.md
+429
View File
@@ -0,0 +1,429 @@
# NAPKIN — socialmag, going programmatic
Whiteboard for the couch talk, 2026-09-12 ~03:50. Numbers here were measured
tonight on this box against abang's live Brave (`:9336`), not estimated.
Nothing in step 1–3 below is built yet.
---
## Where we are tonight
Running at home right now, all by hand:
| Piece | State |
|---|---|
| dashboard `:8321` | up — `http://127.0.0.1:8321/socialmag/dashboard.html` |
| openbridge `:8322` | up (started 03:24; it was **down**, which is why notification clicks opened new tabs instead of reusing the platform tab) |
| brave-abang CDP `:9336` | up — not a service, abang launches it |
| `refresher.py` | works, one-shot, **pressed by hand** |
Shipped tonight, unplanned:
- `facebook.check_login` and `youtube.check_login` were **lying OUT on a signed-in
account**. Both were DOM-signal checks. Swapped to JS state (`c_user` cookie,
`ytcfg.get('LOGGED_IN')`). 4-in/2-out → **6-in/0-out**.
- `refresher.fetch_content()` added — `notifications` and `feed` now come from a
real scrape instead of being hand-pasted. Boris Cherny's 18h-old tweet gone;
feed reads 22m/54m/1h.
- Backups in `Archived/` (`facebook.py`, `youtube.py`, `refresher.py`,
`dashboard.json`, all `20260912-03*`).
---
## The delay budget — where 97 seconds goes
One full refresher pass, 8 jobs, measured 03:35:01 → 03:36:38.
| Source | Cost | Kind |
|---|---|---|
| `PostQueue._space_out()` | 3.0–6.0s **between every job** → 7 gaps ≈ 21–42s | deliberate anti-bot pacing — **keep** |
| `check_login` × 6 | navigate + `asyncio.sleep(4)` (fb/threads/twitter) or `sleep(5)` (ig/tiktok/yt) | blind sleep |
| `get_feed` | `sleep(4)` + click Following + `sleep(3)` + scroll + `sleep(2)` + `sleep(1)` = **10s fixed** | blind sleep |
| `get_notifications` | navigate + `sleep(4)` | blind sleep |
| CDP attach + activate + real navigate | ~3–8s per platform | real work |
**~55 of 97 seconds is unconditional `sleep()`.** Not a slow network — the code
asks for it.
X measured alone, warm tab, 3 runs: `get_feed` **15.0 / 11.2 / 11.1s**,
`check_login` **4.8 / 5.2 / 5.1s**. Of the 11.1s, **10.0s is fixed sleep** — the
actual DOM scrape of 10 articles takes ~1.1s.
The post flows already poll properly (`threads._wait_for_composer`,
`tiktok._wait_for_js`, `youtube._wait_for_editor_empty`). Only the **read** flows
sleep blind. History, not a technical reason.
---
## Background rendering — measured, not assumed
Question: can it work without stealing Brave's focus? Parked focus elsewhere,
attached without `activate()`:
| Platform | `visibilityState` | DOM signal | JS-state signal |
|---|---|---|---|
| twitter | `hidden` | ✅ True | ✅ True — **full scrape works**, 10 tweets, 11 articles at t=2s, flat to t=12s |
| ig | `hidden` | ✅ True | ✅ cookie `ds_user_id` |
| threads | `hidden` | 🔴 **False** | ✅ cookie `ds_user_id` |
| fb | — | 🔴 False | ✅ cookie `c_user` |
| yt | — | 🔴 False | ✅ `ytcfg.LOGGED_IN` |
| tiktok | **unproven** | True | ❌ `sessionid` is httpOnly, invisible to `document.cookie` |
🔴 **Threads is the third liar**, same shape as fb and yt: the "What's new?"
composer never mounts in a hidden tab, so a signed-in account reads as signed out.
⚠️ **TikTok could not be backgrounded.** Activated a sibling tab in the *same*
window (`Browser.getWindowForTarget` → both `windowId` 915141824) and TikTok still
reported `visible`. Either it lives in its own window, or it overrides
`visibilityState` to keep video playing. Its DOM signal has never lied — but it has
never been tested in a genuine background either. **Three states, not two: keep
`activate()` on tiktok until there is evidence.**
The headline: X renders in **2 seconds** in a hidden tab while the code sleeps
**10**. Dropping focus is not the only prize — the sleeps go with it.
---
## Plan — three steps, each stands alone
### 1 · Background pass *(smallest, highest value; nothing else is tolerable without it)*
- `threads.check_login` → cookie. **Required** — it lies today.
- `get_or_open_tab(url, contains, activate=True)` — add the parameter, don't
remove the behaviour. **Post flows still need foreground** for file dialogs.
- Read paths pass `activate=False`: five `check_login`s, `get_notifications`,
`get_feed`. TikTok stays `True`.
- Fixed sleeps → `wait_for` in the three read flows.
Estimate: **97s → ~25-30s, zero tab jumps.**
### 2 · SQLite behind the refresher
Not for size — 4.2KB today, JSON holds that forever. JSON breaks because **every
refresh replaces `feed` and `notifications` wholesale**:
- No history. *"How often does a session drop?"* — the exact question README says
we need answered before judging the VPS version — is unanswerable.
- No concept of new. Tonight's three Threads rows are yesterday's three rows with
`11h` aged to `1d`. Nothing knows they are the same.
- 🔴 The derived `ts` decays. A row sinks an hour every hour. **`first_seen` is a
timestamp we own** — exact, fixed, never guessed. Threads' missing clock stops
being a problem instead of being patched.
Shape: stdlib `sqlite3`, one file, `id = sha1(platform|text|url)`,
`first_seen` / `last_seen` / `times_seen`. Refresher writes the DB, then emits
`dashboard.json` as a derived snapshot — **`dashboard.html` does not change**.
Not proposed: artifact `db` mode. The page supports it (`onSnapshot` over four
collections) but it moves the data out of the house and fixes none of the three.
### 3 · The timer
`deploy/kdmirah/socialmag-refresh.timer` already exists —
`OnUnitActiveSec=10min`, `Persistent=true`, `OnBootSec=2min`. Written for
`User=ijat` on kdmirah, **never installed here**. Adapt paths and user.
Blocked on step 1 only. Installing it today means tabs jumping every 10 minutes
all day, on the desktop abang is working at.
---
# ✅ ALL THREE STEPS BUILT — 2026-09-12 04:10-06:00
The plan above is history now. What actually shipped, and where it differs:
| Step | State |
|---|---|
| 1 · background pass | ✅ `activate=False` param on `get_or_open_tab`; threads/ig/twitter moved to JS-state signals; tiktok deliberately left foreground |
| 2 · SQLite | ✅ `store.py` — `platform_checks` / `items` / `metrics`. `dashboard.json` is a derived view; `dashboard.html` unchanged for it |
| 3 · timer | ✅ three units in `deploy/home/`, installed and verified. Refresh every **30min** (not kdmirah's 10), `--skip-foreground` |
| — | ➕ `capture.py` — the third lane, which was not in the plan at all |
**The `wait_for` rewrite of `get_feed` was never done, on purpose.** `capture`
obsoleted the DOM read path before that work would have paid for itself.
## What changed, by file
cdp.py:179 get_or_open_tab(..., activate=True) — added, not replaced
platforms/threads.py DOM composer → ds_user_id cookie (it was LYING)
platforms/instagram.py → ds_user_id
platforms/twitter.py → twid
platforms/facebook.py → c_user, fixed sleep dropped
platforms/youtube.py → ytcfg.LOGGED_IN, fixed sleep dropped
platforms/tiktok.py untouched; a comment says why it stays foreground
refresher.py parallel check_all(skip_foreground=), capture wired in,
store wired in
capture.py NEW — network capture + in-document read
store.py NEW — SQLite
dashboard.html columns swapped, feed is cross-platform, "seen X ago"
deploy/home/* three units
Measured: **97s → 34s** full pass, **9-10s** login-only, 3/3 stable.
## Landmines earned tonight
- 🔴 **Dropping a `sleep` is not free.** Removing X's `sleep(4)` made `check_login`
report OUT on a signed-in account once five tabs navigated at once — the DOM
signal had not painted. Fixed with the `twid` cookie, not by restoring the sleep.
- 🔴 **`%z` goes into `struct_time` and cannot come back out.** `_x_time_to_iso`
emitted `2026-09-11T19:41:10` with no offset; JS `Date.parse` reads a naked
datetime as *local*, shifting every X row by the machine's offset, silently.
Use `datetime.strptime(...).isoformat()`.
- 🔴 **A pure time sort is not a cross-platform feed.** X returns ~185 fresh rows a
pass against Instagram's 3 older ones, so IG existed in the database and never
on the page. `FEED_PER_PLATFORM = 10` — representation before the merge.
- 🔴 **`ts` must be written once and never rewritten** (`COALESCE(ts, ?)`).
Re-deriving a Threads notification's time each pass reintroduces exactly the
decay the table exists to stop.
- 🔴 **"just now" for a post of unknown age is a lie that sorts to the top.** A
Threads payload without `taken_at` falls back to `first_seen`; those rows read
**"seen X ago"** now.
- ⚠️ `bravemag` `capture.screenshot` fails with an **empty** error string — the old
stdout/stderr trap. DOM probes (`page.evaluate` for bounding boxes and counts)
proved the layout instead, and are better evidence anyway.
---
## Open, abang's call
- **`posts` has no writer.** Only a real `create_post` can produce it, so the
"posted" rows in the timeline stay hand-written until something records post
results. Does `create_post` write to the DB in step 2?
- **Five platforms have no `get_notifications` at all** — `server.py:397` is
`{"threads": threads}`. fb / ig / yt / tiktok / twitter raise. Worth building,
or is Threads + X feed enough for the story?
- **TikTok's background behaviour** — worth one proper probe (own window? spoofed
`visibilityState`?), or just leave `activate()` on it forever?
- **Cadence.** 10min is kdmirah's number, chosen for a box nobody watches. On this
desktop, with step 1 done, what is it worth?
- **Units for `:8321` and `:8322`** — both hand-started today, both die on reboot.
`deploy/kdmirah/` has both service files ready to adapt.
---
## Landmines carried forward
- 🔴 **A DOM signal on a social platform is a rendering question, not a login
question.** Three platforms proved it tonight. Reach for JS state first.
- 🔴 **A short `sleep()` returns a short list with no error** — indistinguishable
from "nothing happened". This is why `fetch_content` keeps old rows and records
`_refreshed.contentErrors` instead of writing `[]`.
- ⚠️ `SOCIALMAG_CDP_PORT` is read by **nothing**. `/root/.claude.json` sets it.
Only `SOCIALMAG_CDP_ENDPOINT` is consulted (`server.py:65`).
- ⚠️ Run as `python3 -m socialmag`, never `socialmag/server.py` — `selectors.py`
and `queue.py` shadow stdlib modules asyncio imports.
- ⚠️ openbridge being down is **silent** by design (progressive enhancement). The
feature looks deleted rather than broken. That is how tonight started.
- ⚠️ **Escaping a JS regex through a Python heredoc bit me tonight.**
`"/ds_user_id=\\d+/"` reaches JS as a literal backslash and never matches, so
three platforms polled the full 20s and reported signed-out. The first
parallel benchmark measured three fake timeouts. A test that fails the way
the real thing would fail is the dangerous kind.
---
# The destination — Blotato, without the API
`https://www.blotato.com` — unified social API + MCP, $29/mo, 9 platforms.
Abang's target shape. **Their moat is the OAuth app approvals**, which is their
heaviest operational cost and the thing "without an API" skips entirely.
| # | Blotato block | socialmag today | Cost without an API |
|---|---|---|---|
| 01 | PUBLISH — 9 platforms, one call | **6 platforms, shipped** | LinkedIn, Bluesky, Pinterest — one module each, pattern exists |
| 02 | SCHEDULE — queue, next open slot, weekly slots | none | **Cheap.** Local queue + timer. Never touches the browser problem |
| 03 | COMMENTS & DMs — read inbox, reply, DM | Threads notifications, `twitter.reply` | **Expensive via DOM** — every inbox is a different page. Cheap via private API (see below) |
| 04 | DM AUTOMATIONS — keyword triggers | none | Needs 03. Highest account risk of anything here |
| 05 | ANALYTICS — top posts, snapshot history | `SCRAPE_JS` already pulls likes/RT/replies | **Cheap — this is step 2.** "Snapshot history" *is* `first_seen`/`last_seen` |
| 06 | VISUALS — infographics, carousels, video | none | Separate lane. House already has geminigen, piskelmag, blendmag |
| 07 | SOURCES — YouTube/PDF/article → post | none | Separate lane. An LLM + fetch problem, not a browser one |
| 08 | API & MCP | **socialmag is already an MCP server** | Done |
Steps 1–3 aren't changed by this — they got a destination. Block 02 *is* the
timer with a queue on it. Block 05 *is* the SQLite.
🔴 **Blotato has platform permission; we do not.** They call official APIs. We
drive a browser logged in as abang, from one home IP. So throughput is not their
1,800 req/hr — it is a sequential queue with 3–6s gaps, one tab at a time, and
that gap *is* the disguise. The risk of a login challenge lands on abang's six
real accounts, not on a service account. This is why the design is **single-seat,
not SaaS.** Do not chase their numbers.
---
# Parallel — CDP supports it, and our client already does
Measured tonight, 6 platforms, all six returning True:
| | Wall |
|---|---|
| Sequential (with polling) | **19.0s** |
| Parallel (`asyncio.gather`) | **6.1s** |
| | **3.1×** |
Against the 97s the code takes today, parallel + poll + background is **~16×**.
CDP is built for this: one WebSocket, `flatten: True`, a `sessionId` per tab, an
`id` on every command. `cdp.py:127-143` already keeps futures in a dict keyed by
id with a single `_reader` dispatching replies. **Nothing needs rewriting.**
🔴 **The bigger finding is in the poll column: every signal read `0.0s`.** It was
present the moment `navigate()` returned. `asyncio.sleep(4)` has been waiting for
something that had already happened.
**Where parallel is allowed — abang's call, not a technical one:**
- `check_login` — **parallel, agreed.** Read-only, no interaction, no writes. Six
tabs loading at once looks like a session restore. 19s → 6s.
- Post flows — **stay sequential with the gap.** That is where the pacing actually
works, and where one mistake is outward-facing.
- `get_feed` / `get_notifications` — parallel is fine, but it is two jobs; the win
is small.
---
# The other lane — private APIs (abang's two links)
⚠️ **`aegents/twitter-no-api` is a fork of `d60/twikit`**, created Dec 2024 and
untouched since; its README is twikit's, badges and all. Take the upstream:
**`d60/twikit` — 4,667★, pushed 2026-03-10, MIT, active.**
**`subzeroid/aiograpi-rest` — 640★, pushed 2026-05-21, MIT.** REST boundary over
`aiograpi` (the maintained async fork of `instagrapi`) = Instagram's **private
mobile API**. Full write surface: post, DM, story. First-class
`challenge_required` and 2FA handling.
These are a **different architecture**, not a socialmag upgrade:
| | socialmag | twikit / aiograpi |
|---|---|---|
| How | drives abang's already-logged-in Brave over CDP | logs in with username+password, speaks the app's own internal API |
| Stores a password? | **No** — the session belongs to Brave | **Yes**, required |
| Needs a browser / display? | Yes | **No** — headless, server, VPS |
| Data | scraped DOM | structured JSON, pydantic-typed |
| Breaks when | the platform redesigns its DOM | the platform moves an internal endpoint |
| Ban vector | traffic pattern | **unofficial client + login from a new IP** |
**Why this matters to the plan:** these libraries solve exactly what README's
"Running on a VPS" section says blocks us — no browser, no display, no tab, no
focus. Block 03 (Comments & DMs), which is expensive through the DOM, gets cheap
through `aiograpi`, because IG DMs have a real endpoint.
🔴 **The price, stated once: both require storing the real account passwords and
logging in as an unofficial client.** That is the classic path to a challenge or
a lock. socialmag never holds a password — that is *why* it is slow.
**The hybrid worth discussing on the couch:** private API for read-heavy and
scheduled work (analytics polling, inbox reads) where volume is the point, browser
for outward-facing writes where being genuinely abang's session is the point. Or
the reverse. Either way it is two lanes with one queue in front, not a migration.
## 🔴 DECIDED 2026-09-12 04:06 — both repos are OUT
Abang's call: **skip `twikit` and `aiograpi` entirely.** They log in via app
requests, and the ban risk lands on his six real accounts. The house lane does
not hold a password and does not need to. Kept in this napkin as the road not
taken, not as a backlog item.
The section below is what replaced them — and it is better than either.
---
# The third lane — structured JSON, no DOM, no password
**This is the finding of the night.** Measured 2026-09-12 04:05-04:09 against
abang's live Brave. The question was "do we still need DOM" — the answer is no,
on all three platforms tested.
The principle: the page's own app already fetches structured JSON to render
itself. **Read that, instead of reading the pixels it painted.** No credentials,
no unofficial client, no new IP — the traffic is genuinely abang's, we only
listen.
## One principle, two mechanisms
| Platform | Mechanism | Result |
|---|---|---|
| X | network capture — `/i/api/graphql/…/HomeLatestTimeline` | **190 tweets in 0.9s** |
| Threads | network capture — `/graphql/query` → `feedData` | 4 posts, 22.9KB body |
| IG | **in-document** — `RelayPrefetchedStreamCache` → `xdt_api__v1__feed__timeline` | 3 posts from a 150KB inline `<script>` |
X and Threads must be *heard*. IG must be *read* — Meta prefetches the feed into
the document, so nothing crosses the wire to capture.
## X — the same source, both ways
| | DOM scrape | Network capture |
|---|---|---|
| Tweets | 10 | **190** |
| Wall | 11.1s | **0.9s** |
| Author | `"Thariq @trq212 · 31m"` (display string) | `trq212` (real `screen_name`) |
| Counts | strings parsed out of the DOM | ints — `L=427 RT=19 RE=45` |
| Views | usually absent | **`V=17912`** |
| Posted at | `<time datetime>` attribute | full `created_at` |
IG returns the same quality: `@raybanmeta ts=1788538432 L=76919 C=945`. Exact
integers, not `"76.9K"` recovered from a rendered string.
## How it works
```
Network.enable on the tab session
→ listen for Network.responseReceived, filter the URL
→ wait for Network.loadingFinished for that requestId
→ Network.getResponseBody(requestId)
```
🔴 **`responseReceived` fires before the body has arrived.** The first probe read
`encodedDataLength` at that moment and reported 149-byte GraphQL responses for
Instagram. Waiting for `loadingFinished` turned the same Threads request into
**22,697 characters**. Sizing a body at `responseReceived` is reading a letter
while it is still in the post.
## The three real limits
1. **Capture must be live.** `getResponseBody` reads the renderer's buffer. Once
the tab navigates away the body is gone — there is no asking later. The
listener has to be armed *before* the request fires.
2. **We only get what the page asks for.** Want DMs? The page has to go to DMs.
Navigation is still our job; this replaces scraping, not driving.
3. **Writes are still unanswered.** Reading is solved. Posting either stays on
the DOM, or we capture the auth headers off one intercepted request and replay
`fetch()` from inside the page — no password, no DOM. **Untested. It does not
work until it has been fired.**
## Not proven — do not repeat these as facts
- ⚠️ **IG's 3 posts were all Meta-owned accounts** (`raybanmeta`, `whatsapp`,
`metaglasses`). That is the prefetched/suggested set, **not abang's following
feed**. His real feed most likely arrives in a later fetch — not captured yet.
- ⚠️ **Threads returned 4 posts against X's 190.** That is page one only; the rest
comes from scroll-triggered fetches. The 190 is X being generous in one
response, not a fair per-platform comparison.
- TikTok and YouTube have not been tried on this lane at all.
## What this does to the plan
- **Step 1 (background pass)** — unchanged and still first. Nothing here works on
a timer while it steals focus.
- **Step 2 (SQLite)** — gets *better*. Exact integers and real epochs arrive for
free, so analytics stops being a parsing problem. `first_seen` still solves
Threads' missing clock for notifications.
- **Block 05 (analytics)** — close to free now. Views and counts ride along.
- **Block 03 (comments & DMs)** — possibly cheap without `aiograpi` after all,
since the inbox pages fetch their own JSON. Worth one probe before assuming.
## Open, added tonight
- Which lane owns **writes**? DOM (proven, slow) or replayed `fetch()` with
captured headers (fast, unproven, and the one that could post something wrong
outward-facing)?
- Can IG's **real** following feed be captured, or is the prefetch all a logged-in
web session gets without scrolling?
- Does the capture lane reach **DMs and comments**? One probe answers whether
block 03 is cheap or expensive.
- TikTok and YouTube on this lane — untested.
- Where does capture live? A listener armed per call, or one long-lived recorder
that watches abang's normal browsing and fills the DB as a side effect?