research-ingest

---
allowed-tools: Read, Write, Edit, Bash, Glob, Grep, WebSearch, WebFetch, Task
description: "Unified investigative research pipeline — articles, documents, books, media, multi-agent extraction, and theme tracking"
argument-hint: "<url or file path> — auto-detects source type"
---

# research-ingest

Unified investigative research ingestion skill (v1.2).

## Auto-detection dispatch

Inspect the first token of `$ARGUMENTS`:

| Input | Routes to |
|-------|-----------|
| URL containing `youtube.com` or `youtu.be` | `media` |
| Any other URL (starts with `http`) | `article` |
| Path ending in `.epub`, `.pdf`, or `.md` | `book` |
| First token is `doc` | `doc` subsystem |
| First token is `extract` | `extract` subsystem |
| First token is `theme` | `theme` subsystem |
| First token is `report` | `report` subsystem |

If no match, assume `article`.

---

## article — Paywall bypass cascade

Flags: `--title`, `--pub`, `--author`, `--date`, `--queue N`, `--project DIR`, `--no-save`

### CRITICAL: Ask first — never reconstruct silently

If Strategy 1 fails and the user is in the conversation, STOP. Say:
> "I couldn't get the full text from [domain]. Do you have access? Can you paste it?"

Only proceed to Strategy 2+ if user says go ahead. Fail loudly in first response line, never buried.

### Strategy cascade (try in order, stop when you have full text)

1. **Direct fetch + Trafilatura** — WebFetch original URL
2. **Jina AI Reader** — `WebFetch: https://r.jina.ai/{URL}` — clean Markdown, works for soft paywalls
3. **Wayback CDX check** — `http://web.archive.org/cdx/search/cdx?url={URL}&output=json&fl=timestamp,statuscode&filter=statuscode:200&limit=1` — if empty, skip; else fetch `https://web.archive.org/web/{TIMESTAMP}/{URL}`
4. **archive.ph** — `https://archive.ph/newest/{URL}` — conditional, only if previously archived
5. **Syndication hunt** — WebSearch: `"{title}" site:yahoo.com OR site:msn.com OR site:britbrief.co.uk`
6. **AMP/mobile versions** — `https://amp.{domain}/{path}` or `https://{domain}/amp/{path}`
7. **Secondary reconstruction** — only with explicit user permission
8. **Parallel agent** — Task (Sonnet) deep research sweep

### Source file format

Save to `03_Resources/{project}/sources/{publication}-{year}-{slug}.md`:

```
# {Article Title}

**Source:** {Publication}
**Author:** {Author}
**Date:** {Month Day, Year}
**URL:** {url}
**Access:** {DONE | PARTIAL | DEAD} — {notes}
**Retrieved:** {today}

---

## Summary

{2–3 sentence summary}

## Key Content

{Full article text}

## Key Facts for Sourcing

- **{Fact}** — {context}
```

---

## book — Multi-agent extraction

Splits the source into 8,000-character chunks. Launches 6 specialist agents per chunk in parallel.

| Agent | Model | Extracts |
|-------|-------|---------|
| People | Haiku | Names, roles, relationships |
| Timeline | Haiku | Dated events, sequences |
| Locations | Haiku | Places, addresses, geography |
| Bibliography | Haiku | Citations, cross-references |
| Claims | Sonnet | Assertions + certainty scores |
| Scenes | Sonnet | Narrative moments: who, where, what was said |

Each agent returns structured JSON. Facts accumulate to `facts/facts.jsonl`. After all chunks complete, run `generate_views.py` and `validate.py`.

---

## extract — Run extraction on a saved source

Point at any saved source file. Runs the same 6-agent pipeline as `book`. Use after ingesting articles or docs when you want deeper structured extraction beyond the initial key facts.

```
python3 scripts/extract_facts_from_source.py sources/filename.md
python3 scripts/build_source_registry.py
python3 scripts/generate_views.py
python3 scripts/validate.py
```

---

## theme — Investigative thread tracking

Themes are cross-cutting patterns that accumulate evidence across sources. Distinct from `tags`.

### Actions

**`theme define <name>`**
Create a theme. Ask for: display name, description, keywords (comma-separated). Write to `facts/themes.jsonl`.

**`theme tag <fact-id> <theme-id>`**
Manually assign a fact to a theme. Update both the fact in `facts.jsonl` (add theme_id to `themes[]`) and the theme entry (add fact_id to `fact_ids[]`).

**`theme suggest`**
Scan all facts in `facts.jsonl` against all theme keyword lists. For each potential match, use Haiku to classify (yes/no/uncertain). Present proposed mappings for user approval before writing.

**`theme report <theme-name>`**
Generate structured evidence summary:
- Fact count + source count + date range
- Core claim synthesis (from highest-certainty facts)
- Evidence by time period
- Strongest sources by fact count
- Coverage gaps

### Schema (facts/themes.jsonl)

```json
{
  "id": "T-001",
  "name": "financial-concealment",
  "display_name": "Financial Concealment",
  "description": "Methods used to hide assets: shell companies, offshore trusts, nominee accounts",
  "keywords": ["offshore", "shell", "trust", "launder", "conceal", "nominee"],
  "fact_ids": [],
  "source_ids": [],
  "fact_count": 0,
  "source_count": 0,
  "earliest_date": null,
  "latest_date": null,
  "created": "{today}",
  "updated": "{today}",
  "status": "active"
}
```

Facts gain a `themes: [theme_id, ...]` field alongside existing `tags`.

---

## report — Project status

Show:
- Facts in `facts.jsonl` (total, by source type, last 7 days)
- Sources in `sources.jsonl` (total, by type)
- Themes in `themes.jsonl` (active, fact coverage)
- Any validation errors

---

## Project config

Default: uses paths configured in the project's `CLAUDE.md`.

With `--project <name>`: reads `~/.research-ingest/<name>.toml`:
```toml
[project]
name = "my-investigation"
vault_path = "~/research/my-investigation"
facts_path = "{vault_path}/facts/facts.jsonl"
themes_path = "{vault_path}/facts/themes.jsonl"
sources_path = "{vault_path}/facts/sources.jsonl"
sources_dir = "{vault_path}/sources"
```

---

## Rules

- **Never fabricate content.** If you can't find a quote, don't make one up.
- **Ask before reconstructing.** Strategy 1 failure = ask user, not proceed silently.
- **Fail loudly.** Access failure goes in first response line, never buried.
- **User-pasted content = full access.** If the article is in the conversation, use it.
- **Save before analyzing.** Content not saved will evaporate between sessions.
- **Check for existing source files** before creating new ones — don't duplicate.
- **Always run validate.py** after bulk extraction. Zero errors = clean corpus.

---

## Examples

```
ingest https://bloomberg.com/... --pub Bloomberg --queue 7
ingest https://substack.com/... --no-save
ingest ~/Downloads/perversion-of-justice.epub --author "Julie K. Brown"
ingest https://www.youtube.com/watch?v=abc123
ingest https://www.youtube.com/@bekahdayyy/videos --channel --confidence MEDIUM
doc court-filing.pdf
extract sources/bloomberg-2025-epstein-emails.md --agents claims,scenes
theme define "victim-network"
theme suggest --source sources/bloomberg-2025-epstein-emails.md
theme report financial-concealment
report
```