A retrieval pipeline over web content is normally four services stitched together: a fetcher, a cleaner, a chunker and a vector store, each with its own failure mode and bill. Here it is two endpoints. This guide covers ingesting a corpus, choosing chunk settings, querying with provenance, and keeping the corpus fresh without re-ingesting everything.
Ingest: one call, any source
POST /api/v1/etl accepts a single `url`, a `urls` array, raw `html`, raw `text`, or an explicit `sources` list where each entry carries its own metadata. It extracts, cleans, chunks, embeds and persists. Billing is PER SOURCE at a 5-credit floor, settled higher-of against model cost — and only structured extraction incurs model cost, so a plain text or markdown ingest bills the floor.
curl -X POST https://twin-browser.com/api/v1/etl \
-H "Authorization: Bearer $TWIN_API_KEY" \
-H "content-type: application/json" \
-d '{
"urls": [
"https://docs.example.com/guides/auth",
"https://docs.example.com/guides/billing"
],
"collection": "example-docs",
"formats": ["markdown"],
"chunkSize": 1200,
"chunkOverlap": 120,
"metadata": { "product": "example", "ingested_by": "nightly" }
}'
# per-document: { source, url, title, chunks, embedded, stored, document_id }
# plus totals and credits_chargedCollections are the namespace — use them
Pass `collection` at ingest to group documents and at query time to scope the search. Without it everything lands in one undifferentiated pool, which is fine until the second corpus arrives and then is not. Collections are also how you retire a corpus: build the new one under a new name and switch the query.
Choosing chunk size and overlap
Chunk size is the main quality dial. The defaults are 1200 characters with 120 of overlap. Too large and a match returns mostly irrelevant text around the sentence that mattered; too small and the passage loses the context that made it meaningful. Overlap exists so an answer that straddles a boundary is not lost. Keep the settings consistent within one collection — mixing them makes similarity scores harder to compare.
Query: flat-priced, with provenance
POST /api/v1/etl/query embeds your question and returns the nearest chunks — each with its similarity, its source url and title, and its document and chunk index. It is a vector lookup, not a generation, so it costs a flat 1 credit and makes no model call. `k` defaults to 8 and is capped at 50.
curl -X POST https://twin-browser.com/api/v1/etl/query \
-H "Authorization: Bearer $TWIN_API_KEY" \
-H "content-type: application/json" \
-d '{ "query": "how do I rotate an API key?",
"k": 8,
"collection": "example-docs" }'
# [ { "content": "…", "url": "https://docs.example.com/guides/auth",
# "title": "Authentication", "similarity": 0.83,
# "document_id": "…", "chunk_index": 4 }, … ]Where the sources come from
Three usual inlets, and they compose. Crawl a documentation subtree and ingest what it returns. Run a search and ingest the URLs it ranked. Or ingest raw text you already hold — a support export, a wiki dump — by passing `text` or `html` directly, in which case nothing is fetched at all.
- Crawl → ETL: bound the crawl with includePaths, then ingest the pages it read.
- Search → ETL: take the URLs from a shallow search and ingest them.
- Direct: pass html or text (or a sources array) and skip fetching entirely.
Keeping the corpus fresh
Ingest does not deduplicate: re-ingesting the same URL adds a new document. So refresh deliberately rather than on a blind schedule. A monitor on the pages that actually change is the cheap way to know when to re-ingest — an observe check is a single credit, against a floor per source for a re-ingest of a page that did not move.