Shroomery → Markdown Archiver

Paste a Shroomery.org URL and this app systematically crawls the forum and writes one Markdown document per thread into an organised folder tree — with no third-party dependencies (Python 3.9+ stdlib only).

What a URL gives you

You paste…	You get…
Forum index (/forums/)	EVERY board and nested subforum, all pages, all threads
A board (/forums/postlist.php/Board/2)	that board (+ its subforums), all pages, all threads
A thread (/forums/showflat.php/Number/17305253)	that one thread, every post

Quick start

# CLI
python3 run.py "https://www.shroomery.org/forums/postlist.php/Board/2" --output output

# web UI (paste URLs into a browser)
python3 -m shroomery.webapp --port 8000
open http://127.0.0.1:8000

Fetch sources (--via)

Source	Meaning	Notes
live (default)	the live site	best on your own machine; needs Cloudflare-free access
wayback	archived captures via web.archive.org	works even when the live site is blocked; enumerates every thread the archive has and groups each into its real board
tor	live site through a SOCKS5 proxy (--proxy host:port, default 127.0.0.1:9050)	bypasses ISP/content filters; tools/tor_up.sh brings up a fleet of fresh, isolated Tor nodes (new identities) and probes whether the site actually answers through them — use this to restart crawling whenever the old exit pool gets Cloudflare-flagged
relay	r.jina.ai open-web relay	the guaranteed lane when the site is SNI-filtered from this network AND Tor exits are challenged; ~15 req/min capacity is shared by the whole farm, so it's a survival lane, not a speed lane

# Bring up 7 fresh IsolateSOCKSAuth Tor nodes (new identities, + both old
# pools were Cloudflare-flagged), probe the site through them, and if it
# answers with real HTML, launch the 12-slot farm on them:
bash tools/tor_up.sh
python3 tools/supervise.py --via tor --delay 1.5 &

# Crawl every thread the Wayback Machine has archived (2010 -> now),
# one .md per thread, grouped by the thread's actual board.
python3 -m shroomery.cli "https://www.shroomery.org/forums/postlist.php/Board/2" \
    --via wayback --from 2010 --since 2010-01-01 --delay 1.5 -o output-wayback -v

# Same, but crawl a pre-enumerated id list (id per line) and resume safely:
python3 -m shroomery.cli <url> --via wayback --seed-ids threads.txt -o output-wayback

Lane selection on a network with a TLS-MITM box (self-signed CA in the chain): export SHROOMERY_SSL_NO_VERIFY=1 to make the fetcher skip TLS verification (needed for --via live behind such proxies). tools/host_upsert.sh picks the lane by content, not just status code: the Vodafone "restricted content" block page answers HTTP 200 and must never be taken for the real site.

Parallel live crawl (speed)

The board has ~14,000 list pages / ~350k threads. To speed up a live crawl, run several strided workers — each handles a 1/N slice of the board pages and writes to its own directory; thread-id filenames make merging collision-free:

# 6 workers, each taking every 6th page starting at offset w+1
for w in 0 1 2 3 4 5; do
  python3 -m shroomery.cli "https://www.shroomery.org/forums/postlist.php/Board/2" \
    --via tor --pages-start $((w+1)) --pages-step 6 --delay 1.0 \
    --since 2010-01-01 --output output-live-w$w &
done

# later, merge them:
python3 tools/merge_workers.py output-live-w0 ... output-live-w5 -o output-live

Long-running crawl: never stop until every thread is a .md file

Crawling ~350k threads takes a long time and the box may kill a worker mid-run (the crawler streams thread documents to disk precisely so a 500-page monster thread never balloons memory). tools/supervise.py keeps the farm going:

python3 tools/supervise.py              # launch/manage the 12 default slots
python3 tools/supervise.py --dry-run    # show the slot map

The supervisor starts one worker per slot (--pages-start 1..12 --pages-step 12, six on the :9050 Tor daemon, six on :9060), and relaunches any worker that dies before its log ends with a clean boards: ... [done] summary — the crawler resumes from output-live-wN/.crawler_state.json, so nothing is lost. Each slot is only marked complete when a full run of its stride finished with a clean summary; the supervisor exits 0 only when all 12 slots have completed. Ctrl-C / SIGTERM stops the farm cleanly. Later, tools/merge_workers.py combines the twelve output-live-w0..w11 trees into one.

Every thread is crawled to its natural end — threads are never truncated by page count (a walk stops only when the thread's pagination chain runs out). The one exception is the explicit --max-pages-per-thread N escape hatch, which should never be used for archive runs (it drops a thread's newest pages).

Repairing threads truncated by an old page cap

Early versions had a hard 500 pages per thread stop that silently dropped the newest pages of long (giant) threads. If you have output written by such a version, repair it in two steps:

# 1) find threads saved at exactly 500 pages, probe their real length, and
#    flag them in .crawler_state.json (redo: true):
python3 tools/backfill.py output-live-w0 output-live-w1 ... \
    --proxy 127.0.0.1:9050 --delay 0.5

# 2) re-crawl just the flagged threads, in place under their board folders
#    (no board re-walk):
python3 -m shroomery.cli "https://www.shroomery.org/forums/postlist.php/Board/2" \
    --via tor --redo-only --output output-live-w0

backfill.py probes each candidate's first page so a genuinely ~500-page thread is not re-fetched wastefully. Run --redo-only for every repaired directory once its normal crawl has finished.

For a large fleet, tools/redo_supervise.py keeps all the redo workers alive (relaunch on death/stall) and handles the sequencing automatically: it starts the already-finished dirs' redo workers immediately, then — once the main farm's slots 0-5 COMPLETED — arms two sharded redo workers per directory (--redo-shards 2 --redo-shard 0|1, splitting threads by id % 2) so the multi-hundred-thousand-page backfill uses twice the concurrency:

python3 tools/redo_supervise.py                # waits for the farm, then 12 workers
python3 tools/redo_supervise.py --proxies 127.0.0.1:9050,127.0.0.1:9060,127.0.0.1:9062

Trusted Cultivator digest

Once the archive exists (or at any point during the crawl) you can pull every post written by a Trusted Cultivator out of all saved thread documents into one browsable file — a knowledge base of what the site's most trusted growers have said:

# merged tree (or the raw worker dirs), any time:
python3 tools/trusted_cultivators.py output-live -o trustedcultivators.md

# wait for the whole farm + redo pass to finish, then build it:
python3 tools/trusted_cultivators.py output-live --wait -o trustedcultivators.md

# counts only, nothing written:
python3 tools/trusted_cultivators.py output-live --dry-run

The tool is a pure archive pass — it never touches the network. It scans every saved thread .md for the exact **Trusted Cultivator** author badge, dedupes threads by id (most posts wins), and streams one trustedcultivators.md to disk (the full board yields hundreds of thousands of picked posts, so the digest is never built in memory).

Each picked post keeps its original annotation and Shroomery link and carries its relevant context (not a fixed window):

• the post(s) it is replying to — resolved from the quoted-reply blocks embedded in the post body (matched by author + text overlap against earlier posts), or the immediately preceding post when it quotes nothing;
• the following replies (--after N, default 2).

Picked posts are marked with a ★. Options: --badge (repeatable, e.g. --badge "OG Cultivator"), --after, --context-depth, --require-done (refuse to build while any input's .crawler_state.json still has redo/partial threads — i.e. the crawl/repair is not finished), --wait (poll until that's true), --dry-run. Dirs without a state file (a merged tree) always count as finished.

Curated knowledge base (tools/curate.py)

The big one. Where trusted_cultivators.py picks by a single badge, tools/curate.py learns the forum's subject structure and cherry-picks the best, most highly-rated posts on every subject into a single category-sorted Markdown knowledge base — including a Tek Landscape section that maps how cultivation methods evolved over the archive's 17 years and what is recommended now (so old-but-still-correct advice is kept, while superseded techniques are labelled, not deleted).

# all worker dirs, or a merged tree:
python3 tools/curate.py output-live-w0 ... output-live-w11 -o kb.md

# wait for the whole farm + redo pass to finish, then build:
python3 tools/curate.py output-live --wait -o kb.md

# tighter/looser selection, and a counts-only dry run:
python3 tools/curate.py output-live -o kb.md --top 3 --min-score 0.55
python3 tools/curate.py output-live --dry-run

It is a pure archive pass (no network) with a streaming design:

1. Index (--index DIR, default a temp dir) — parses every saved thread once into a JSONL post index (author reputation, dates, community post score, quote-graph engagement, teks, text features) + aggregate stats. Re-runs reuse it instead of re-parsing.
2. Learn the taxonomy — a forum-grounded seed (the real subforum tree: Agar & LC, Cakes, Boxes, Substrates, Casing, Outdoor, Panaeolus, Mexicana/Tampanensis, Grow Logs, Archive …) is combined with content-based matching, and an emergent-topic detector finds subjects the seed missed. A learned-taxonomy report is written to kb.md.taxonomy.md for review (and can be overridden with --taxonomy FILE.json).
3. Score every post — an explainable weighted score: reputation (Trusted Cultivator/OG/Moderator/Vendor badges, author rating × votes, seniority), engagement (how often later posts quote it, thread scale, community post score), content (length, steps, code/tables, images; filler rejected), recency (decay so current practice wins), and tek-era currency (does the post recommend the method the community currently uses?).
4. Dedupe per subject — within each detected subject only the top --top posts are kept; near-duplicate answers are collapsed; era notes flag when a subject's posts span technique changes.
5. Compile — one .md: header + method legend + anchored table of contents → Tek Landscape & Current Recommendations (per-tek timeline: first/peak/last year, supersession edges, per-goal recommendations) → every category with its best posts. Each picked post shows its "why picked" score breakdown, the thread's OP as question context, its original body (signatures stripped) and its Shroomery link.

Options: --top N (posts kept per subject, default 2), --min-score F (default 0.50), --max-posts N (global cap), --summary (excerpts), --no-context, --taxonomy FILE.json, --index DIR / --rebuild-index, --require-done / --wait / --poll, --dry-run, -v. Dirs without a state file count as finished.

Tek Encyclopedia (encyclopedia/)

The evidence-first encyclopedia built from the same archive: an emergent taxonomy (Louvain communities, not predetermined chapters), per-tek 1-10 reliability scores with wheat/chaff verdicts, normalized SOPs with byte-offset provenance, and lineage/supersession DAGs — with a zero- hallucination guarantee (every number traces to a TekEvidence record or a verified physics constant). See encyclopedia/PLAN.md for the full spec map.

# two-pass streaming over the compact JSONL post index (out-of-core at
# 3.6M posts; giant 400 MB-thread files are never re-parsed wholesale)
python3 -m encyclopedia output-live-w0 output-live-w1 ... \
    --index /path/to/index.jsonl-dir -o encyclopedia-out

python3 -m encyclopedia output-live --explain 'PF Tek'   # scored evidence dump
python3 -m encyclopedia output-live --species coprophilous -o out
python3 -m encyclopedia output-live --require-done --wait   # crawl gate

Emits encyclopedia.md (9-field schema per tek: Tek Name, Emergent Category, Historical Timeline, Scientific Reliability Score, The Core Concept, Reliability Analysis, Community Consensus, Standard Operating Procedure, Successor), encyclopedia.json, a self-contained searchable encyclopedia.html, and diagnostics.json. Tests: tests/test_encyclopedia.py.

CLI options

python3 run.py <url> [options]
  -o, --output DIR       output directory (default: output)
      --delay SECONDS    seconds between requests (default 2.0; polite)
      --limit N          max threads archived per board (0 = all)
      --since YYYY-MM-DD skip threads whose last post predates that date
      --dry-run          enumerate pages/threads without writing files
      --no-resume        re-crawl threads already in the state file
      --ignore-robots    crawl even if robots.txt disallows a path
      --cookies FILE     Netscape cookies.txt (members-only boards)
      --cookie NAME=VALUE  send an extra cookie (repeatable)
      --via SOURCE       live | wayback | tor | relay   (see above; default live)
      --proxy HOST:PORT  SOCKS5 proxy for --via tor (default 127.0.0.1:9050)
      --from YEAR        --via wayback: earliest capture year (default 2010)
      --to YEAR          --via wayback: latest capture year (default: now)
      --seed-ids FILE    file of thread ids to crawl (skip re-enumeration)
      --max-pages-per-thread N  fetch at most N pages of any thread, default 0 =
                          unlimited (WARNING: capping drops newest posts)
      --redo-only        re-crawl only threads flagged for redo in state
      --redo-shard N     which redo shard this worker handles (0-based)
      --redo-shards M    split the redo set by id % M across M workers
      -v                 verbose progress

Output layout

output/
├── _index.md                     ← top-level index
├── mushroom-cultivation/         ← one folder per board
│   ├── _index.md                 ← board index (thread table)
│   ├── 17305253-guide-to-common-contaminate-mushrooms.md
│   └── 23130868-recognizing-and-dealing-with-contamination.md
└── .crawler_state.json           ← resume/state file

Each thread document:

• YAML front matter — title, board, thread URL, first author + post date, page and post counts, crawl timestamp
• every post in order — post number, author, date, direct link, then a blockquote with the author's full profile annotation (status badges such as Trusted Cultivator / Vendor / moderator, custom title, star rating, member #, registered, posts, location)
• quoted replies as Markdown blockquotes, images preserved (decorative icons dropped), spoilers, code blocks, tables, signatures

Notes / etiquette

• The crawler is polite by default: configurable delay, retries with backoff, a descriptive user-agent, robots.txt honoured (use --ignore-robots to override). Several boards have hundreds of thousands of threads — use --limit, --since and --dry-run to scope, and the state file to pause and resume.
• Shroomery is a very old site (~1997-vintage markup). If the theme changes, tunables live in shroomery/site.py (URL patterns) and shroomery/parser.py (selectors).
• Content is copyrighted; this tool is for personal archival use. Please be considerate and follow the site's terms.

Test

python3 -m unittest discover -s tests

Tests run against real archived HTML snapshots in tests/fixtures/ (see tests/fixtures/README.md for sources).