CAPABILITY · ETL

Web pages into a content store you can query by meaning

Extract, clean, chunk, embed and load any source — URL, HTML or raw text — into a queryable store, then search it with a natural-language query for a flat credit.

Start free API reference

POST /api/v1/etl · POST /api/v1/etl/query

Capability

Inside etl & semantic query

A retrieval pipeline over web content is usually four services stitched together: a fetcher, a cleaner, a chunker and a vector store. This is one endpoint. Give it URLs, raw HTML or text — one or many per call — and it extracts, cleans, optionally runs structured extraction, chunks, embeds and persists. A second endpoint queries what you loaded by meaning and returns the matching chunks with their source.

Any source, one or many

A single `url`, a `urls` array, raw `html`, raw `text`, or an explicit `sources` list where each entry carries its own metadata. The same call handles all of them.

Collections as namespaces

A `collection` groups documents at ingest and scopes the query, so one tenant can keep several corpora apart without separate stores.

Structured extraction in the pipeline

Ask for the `json` format, or pass `schema` / `fields` / `prompt`, and structured extraction runs as the transform step — the only part of ETL that incurs model cost.

Query returns provenance

A query returns each matching chunk with its similarity, its source url and title, and the document and chunk it came from — so a citation is available, not reconstructed.

How it works

The mechanism, in execution order

4 stages, in the order the runtime performs them — not a summary of them.

1ExtractEach source is fetched (or taken as given) and read into text, markdown, HTML or JSON as you asked.
2TransformThe content is cleaned, and structured extraction runs when you passed a schema, fields or the json format.
3Chunk and embedThe result is chunked (default 1200 chars with 120 overlap, both tunable) and embedded — set `embedding: false` to store without.
4Load, then queryDocuments persist to the content store unless `store: false`. POST /api/v1/etl/query searches them by meaning and returns the top-k chunks.

In practice

A call you can paste and run

Ingest is per source and metered; the query over what you ingested is a flat credit with no model in it.

etl.shbash

# Ingest a set of pages into a named collection
curl -X POST https://twin-browser.com/api/v1/etl \
  -H "Authorization: Bearer $TWIN_API_KEY" \
  -H "content-type: application/json" \
  -d '{"urls":["https://docs.example.com/a","https://docs.example.com/b"],
      "collection":"example-docs",
      "formats":["markdown"],
      "chunkSize":1200,"chunkOverlap":120}'

# Query it by meaning — flat 1 credit, no model call
curl -X POST https://twin-browser.com/api/v1/etl/query \
  -H "Authorization: Bearer $TWIN_API_KEY" \
  -H "content-type: application/json" \
  -d '{"query":"how do I rotate an API key?","k":8,"collection":"example-docs"}'

# [ { "content": "…", "url": "https://…", "title": "…",
#     "similarity": 0.83, "document_id": "…", "chunk_index": 4 }, … ]

api.twin-browser.com

ETL & semantic query

Extractdone
Transformrunning
Chunk and embedqueued
Load, then queryqueued

What it costs, how to switch it on

Priced from the same rate card the API serves

Every credit figure on this page is read from the rate card behind GET /api/v1/pricing — it is not typed into the copy, so it cannot drift from what you are billed.

5-credit floor per source · 1 credit per query

Ingest bills PER SOURCE: higher-of(5-credit floor, metered model cost). Only structured extraction incurs model cost, so a plain text or markdown ingest bills the floor. A query is a flat 1 credit and makes no model call — it is a vector lookup, not a generation.

Full rate card

Turning it on

1Pick a collection namePass `collection` at ingest and at query time. Without it everything lands in one undifferentiated pool.
2IngestPOST /api/v1/etl with `urls` (or html/text). Embedding is on by default; set `embedding: false` for store-only.
3QueryPOST /api/v1/etl/query with a natural-language `query`, a `k`, and the same `collection`.
4Keep it fedPair it with crawl for whole-site ingest, or with a monitor so a changed page triggers a re-ingest.

At a glance

The contract, in the fewest rows that say it

Property	Twin Browser
Ingest	POST /api/v1/etl
Query	POST /api/v1/etl/query
Sources	url · urls · html · text · sources[]
Formats	text (default) · markdown · html · json
Chunking	1200 chars / 120 overlap, tunable
Query size	`k` 1–50, default 8

Limits

Where it stops, and what it deliberately does not do

A capabilities page with no limits section is a brochure. These are the ceilings, the defaults that will surprise you, and the things this capability is not.

Limits and defaults

Ingest is billed per source, so a 500-URL call is 500 floors — batch deliberately.
`k` is capped at 50 per query.
Store-only (`store: false`) returns the processed content without persisting it, which means it is not queryable afterwards.
Chunk size and overlap are per call, not per collection: mixing very different settings inside one collection makes similarity scores harder to compare.

What it does not do

It does not generate an answer. The query returns chunks with provenance; the synthesis is yours.
It does not re-crawl on its own. Ingest is a call you make; keeping a corpus fresh is a crawl or a monitor you schedule.
It does not deduplicate across ingests. Re-ingesting the same URL adds a new document.
It does not expose a per-collection retention policy — deleting a corpus is not part of this endpoint pair.

FAQ

Questions about etl & semantic query

Is this a RAG pipeline?

It is the retrieval half of one, as a single API: extract, clean, chunk, embed, load, and query by meaning with provenance on every chunk. Generation stays on your side, with whatever model you already use.

What actually costs model tokens here?

Only structured extraction. Asking for the json format or passing schema / fields runs a metered read; plain text or markdown ingest bills the 5-credit floor. The query itself is a flat 1 credit with no model call.

How do I keep several corpora apart?

Use `collection` as a namespace: pass it on ingest to group documents and on query to scope the search. It is the same field on both endpoints.

Can I load content I already have?

Yes. Pass `html` or `text` directly, or a `sources` array where each entry has exactly one of url / html / text plus its own metadata. Nothing is fetched for those.

Keep reading

The rest of the platform

The capabilities this one works with, the tools that expose it, the guides that use it end to end, and the vocabulary behind it.

etl etl_query submit_crawl

Guides

Build a retrieval pipeline over web content in two calls Crawl a website without crawling the whole website

Vocabulary

ETL (extract, transform, load)Chunking Embedding Semantic search

Put your agent to work. Keep the decision.

Start free. Hand your agent a goal on a site you authorize, set the guardrails, and let the first successful run compile the skill every run after it replays.

Start free Read the docs