API Quickstart

Useful Endpoints

POST /scrape → stream chunks from uploaded files
POST /extract → stream structured extraction; ends with { "extraction_complete": true }
GET /get_tokens_available → remaining tokens

The hosted API accepts uploaded files only (PDF, DOCX, PPTX, XLSX/CSV, images, audio, video, Jupyter notebooks, ZIP). URL scraping is not supported; save a web page as a PDF or HTML file first.

Auth header

Every request uses an Authorization header:

Authorization: Bearer <thepipe-api-key>

Scrape API

Stream chunks (text + optional images) from files.

Endpoint: POST /scrape Content-Type: multipart/form-data Auth: Authorization: Bearer <token> Response: streaming NDJSON (one JSON object per line)

Form fields

files: one or more uploaded files (repeat the key)
text_only_input: true|false (default: false) → don't send page images to the vision model
text_only_output: true|false (default: false) → omit images from the returned chunks
ai_extraction: true|false (default: false) → use a vision-language model to transcribe each PDF page to clean markdown and crop out figures/tables with bounding boxes. Works on scanned documents.
chunking_method: one of chunk_by_document | chunk_by_page | chunk_by_section | chunk_by_length | chunk_by_keywords | chunk_semantic
keywords: JSON array of strings (required with chunk_by_keywords)

Curl examples

Single PDF file

curl -N \
  -H "Authorization: Bearer $YOUR_API_KEY" \
  -F "files=@/path/to/file.pdf" \
  -F "ai_extraction=true" \
  -F "chunking_method=chunk_by_page" \
  https://thepipe-api.up.railway.app/scrape

Sample streamed line

{
  "result": {
    "chunk_index": 0,
    "source": "file.pdf",
    "content": [
      { "type": "text", "text": "..." },
      { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,..." } }
    ],
    "metadata": {
      "page": 1,
      "model": "google/gemini-2.5-flash-lite",
      "figures": [{ "description": "...", "bbox": [0.1, 0.2, 0.9, 0.5] }]
    }
  },
  "tokens_used": 123
}

Consume as an event stream (each line is a complete JSON object). metadata carries provenance such as page, slide, row, section, or figure bounding boxes depending on the source type.

Token usage is tracked and deducted automatically per chunk.

Extract API

Structured data extraction against a schema from files. Extraction uses structured outputs, so every row conforms to your schema; missing values are returned as null.

Endpoint: POST /extract Content-Type: multipart/form-data Auth: Authorization: Bearer <token> Response: streaming NDJSON (one JSON object per line) ending with { "extraction_complete": true }

Required fields

files: one or more uploaded files
schema: JSON describing fields. Supported types: string, int, float, bool, date. Two accepted shapes:
1. Ordered form (recommended):
```
{
  "name": { "type": "string", "order": 0 },
  "price": { "type": "float", "order": 1 }
}
```
2. Simple form (backend will assign order automatically):
```
{ "name": "string", "price": "float" }
```

Optional fields

ai_extraction: true|false (default: false) → VLM scraping of the document before extraction (recommended for scanned or complex PDFs)
text_only_input, text_only_output
chunking_method: chunk_by_document | chunk_by_page | chunk_by_section | chunk_by_length | chunk_by_keywords | chunk_semantic
multiple_extractions: true|false → return many rows per chunk
keywords: JSON array (for chunk_by_keywords)
custom_prompt: extra guidance appended to the default extraction prompt

Curl examples

Multiple PDFs, many rows per chunk

curl -N \
  -H "Authorization: Bearer $YOUR_API_KEY" \
  -F 'schema={"item":{"type":"string","order":0},"qty":{"type":"int","order":1}}' \
  -F 'multiple_extractions=true' \
  -F files=@/a.pdf -F files=@/b.pdf \
  -F chunking_method=chunk_by_page \
  https://thepipe-api.up.railway.app/extract

Sample streamed line

{
  "result": {
    "chunk_index": 0,
    "source": "a.pdf",
    "extraction": [{ "item": "Widget", "qty": 3 }]
  },
  "tokens_used": 456,
  "chunk": { "path": "a.pdf", "text": "...", "images": [], "metadata": { "page": 1 } }
}

Without multiple_extractions, the fields appear directly on result alongside chunk_index and source. If a chunk fails, result.error contains the message.

The stream ends with:

{ "extraction_complete": true }

Errors

Standard HTTP status codes; body contains { "error": "message" }. Sending urls returns 400.

Python Quickstart Data Sources