What is a collection for?

A namespace. Ingest one customer’s docs, one product’s pages or one crawl into its own collection, and etl_query can scope a search to it. Without one, everything shares a single space — fine for a small store, painful once you have several unrelated corpora.

Can I use it purely as a cleaner, without storing anything?

Yes. store: false returns the cleaned content and persists nothing; embedding: false persists without vectors. The pipeline is four independent switches, not an all-or-nothing.

Observation & data

Ingest a source into a queryable content store

Extract, clean, optionally structure, chunk, embed and load — any mix of URLs, raw HTML and raw text, one or many per call, into a namespace you can then search semantically.

Get an API key MCP setup

etl

What it is

The job: Turn web content into something you can ask questions of, without building a pipeline.

The five stages an ingest pipeline always has — fetch, clean, chunk, embed, store — are one call here. Sources can be a single url, a list of urls, raw html, raw text, or an explicit `sources` array where each entry carries its own metadata. Everything lands in the same content store, optionally namespaced by `collection`.

Both of the last two stages are switchable. embedding: false stores without vectors when you only want the cleaned content; store: false returns the content without persisting anything, which turns the same call into a batch cleaner. Chunk size and overlap are yours to tune.

Add a schema or fields and the transform stage also runs structured extraction, so each document comes back with `data` alongside its chunks. That is the only part of the pipeline that costs model time; a plain text ingest runs no model at all.

What you ingest here is what etl_query searches.

The call

Call it exactly like this.

Copied from the tool's registration and the route handler — not paraphrased.

MCPetl.mcp.jsonjson

// MCP tool call — server "twin-browser"
{
  "tool": "etl",
  "arguments": {
    "urls": [
      "https://example.com/docs/intro",
      "https://example.com/docs/api"
    ],
    "collection": "example-docs",
    "chunkSize": 1200,
    "chunkOverlap": 120
  }
}

POST /api/v1/etlrequest.shbash

curl -X POST https://twin-browser.com/api/v1/etl \
  -H "Authorization: Bearer $TWIN_API_KEY" \
  -H "content-type: application/json" \
  -d '{"urls":["https://example.com/docs/intro","https://example.com/docs/api"],"collection":"example-docs","chunkSize":1200,"chunkOverlap":120}'

Parameters accepted by etl
Parameter	Type	What it does
url	string	A single URL to ingest.
urls	string[]	Multiple URLs to ingest in one call.
html	string	Raw HTML to ingest — no fetch.
text	string	Raw text to ingest — no fetch.
sources	object[]	Explicit source list; each entry has exactly one of { url \| html \| text } plus optional metadata.
formats	string[]	Output formats to return: text (default), markdown, html, json. 'json' — or a schema/fields — runs structured extraction.
schema	object	JSON Schema for structured extraction.
fields	array \| object	Fields spec for structured extraction — names, or { name, description }.
prompt	string	Instruction to guide structured extraction.
embedding	boolean	Embed chunks so etl_query can find them. Default true.
store	boolean	Persist to the content store. Default true; false returns content without saving it.
collection	string	Namespace to group and later filter these documents.
chunkSize	number	Chunk size in characters. Default 1200.
chunkOverlap	number	Chunk overlap in characters. Default 120.
metadata	object	Metadata stored with every document in this call.
proxy	string	Your own egress proxy URL, for url sources.
waitMs	number	Settle wait in milliseconds before reading url sources.

Returns

response.jsonjson

{
  "documents": [
    { "source": "url", "url": "https://example.com/docs/intro", "title": "Intro",
      "chunks": 14, "embedded": true, "stored": true, "document_id": "…",
      "data": { … } }          // present when schema/fields ran
  ],
  "credits_charged": 5
}

What it costs

5 credits per source

A 5-credit floor per source, settled higher-of against metered model cost. Only structured extraction (schema or fields) invokes a model; a plain fetch-clean-chunk-embed ingest has no LLM cost and settles at the floor.

See the full rate card

Which one

When a different tool is the right call.

The honest answer is often the neighbouring tool. These are the trades.

extract

extract returns data to you and forgets the page. etl keeps it. If you will ask a second question about the same content later, ingest it once instead of extracting it twice.

extract

submit_crawl

A crawl is how you find the URLs; etl is how you make them queryable. They compose — crawl to enumerate, etl to ingest.

submit_crawl

etl_query

The read side of the same store. Nothing you ingest is useful until you query it.

etl_query

Questions

etl, answered.

What is a collection for?: A namespace. Ingest one customer’s docs, one product’s pages or one crawl into its own collection, and etl_query can scope a search to it. Without one, everything shares a single space — fine for a small store, painful once you have several unrelated corpora.
Can I use it purely as a cleaner, without storing anything?: Yes. store: false returns the cleaned content and persists nothing; embedding: false persists without vectors. The pipeline is four independent switches, not an all-or-nothing.

Keep going

The rest of the tool set.

etl_query

Ask a question of the content you ingested, and get back the passages that answer it.

extract

Get data off a page in the shape your code already expects.

submit_crawl

Read a defined slice of a site, once, without either missing it or over-reading it.

map_site

Find out what is on a site before deciding how much of it to read.

The mechanism behind it:Token-efficient DOM state

Get an API key All tools