Ingest log

Deterministic, extractive build: no LLM (no API key on the box).

Stats

• build: public
• files: 4
• sources: 19
• words: 39201
• sources_by_collection: {'nate': 16, 'pin': 3}
• words_by_collection: {'nate': 37064, 'pin': 2137}
• sections: 120
• entities: 40
• topics: 10
• pages: 73
• links_directed: 1358
• links_unique_pairs: 680
• llm_used: False

Pipeline

1. Discover files from sources.json globs (first rule wins; superseded issue drafts skipped); redact emails, IDs and secret-looking strings.
2. Split markdown by headings (code blocks kept as their own passages); transcripts into chapters and 90-second timestamp blocks.
3. Transcript name fixes: \b[Aa]ndre [Cc]arpathi\b → Andrej Karpathy; \b[Cc]arpathi\b → Karpathy; \bcloud code\b → Claude Code; \bCloud Code\b → Claude Code; \bLMM Wiki\b → LLM Wiki; \b[Ss]chool community\b → Skool community; \bCrohns?\b → cron; \bAIS OS\b → AIOS; \bAIs OS\b → AIOS.
4. TF-IDF (unigrams + bigrams) over 120 sections; k-means → 10 topic pages.
5. Entities: capitalized phrases and @handles seen ≥3 times, minus common words.
6. Extractive summaries, entity mention lists with citations, backlinks, static site with BM25 search, d3 graph and ask bot.

Run log

[ 0.07s] Loaded 4 files -> 19 sources, 39,201 words, 120 sections [ 0.11s] k-means: 10 topics (k=10) [ 0.40s] Entities: 40 (from 334 candidates) [ 0.50s] Pages: 73 (19 source, 10 topic, 40 entity, 4 index) · 680 linked pairs