Python Quickstart

Thepipe can be installed via the command line:

pip install thepipe-api

The default install only pulls in CPU-friendly dependencies. Heavier dependencies are available as optional extras:

Audio and video scraping additionally require the ffmpeg binary on your PATH:

apt-get update && apt-get install -y ffmpeg
pip install thepipe-api[audio]

Default setup (OpenAI)

By default, thepipe uses the OpenAI API (opens in a new tab), so VLM features will work out-of-the-box provided you pass in an OpenAI client.

Custom VLM server setup (OpenRouter, OpenLLM, etc.)

If you wish to use a local vision-language model or a different cloud provider, you can provide a custom OpenAI client, for example, by setting the base url to https://openrouter.ai/api/v1 for OpenRouter (opens in a new tab), or http://localhost:3000/v1 for a local server such as OpenLLM (opens in a new tab). Note that you must also pass the api key to your non-OpenAI cloud provider into the OpenAI client. The model name can be changed with the model parameter. By default, the model will be gpt-4o (configurable via the DEFAULT_AI_MODEL environment variable).

Scraping

from thepipe.scraper import scrape_file
 
# scrape text and page images from a PDF
chunks = scrape_file(filepath="paper.pdf")
 
for chunk in chunks:
  print(chunk.text)

For enhanced scraping with a vision-language model, you can pass in an OpenAI-compatible client and a model name. Each page is transcribed to clean markdown, and figures, charts, and tables are cropped out of the page using bounding boxes returned by the model (instead of a full page render). This also works on scanned, image-only PDFs.

from openai import OpenAI
from thepipe.scraper import scrape_file
 
# create an OpenAI-compatible client
client = OpenAI()
 
# scrape clean markdown and page images from a PDF
chunks = scrape_file(
  filepath="paper.pdf",
  openai_client=client,
  model="gpt-4o"
)

All available options:

chunks = scrape_file(
  filepath="paper.pdf",          # path to the file to scrape
  verbose=True,                  # print progress logs
  chunking_method=chunk_by_page, # chunking function
  openai_client=client,          # openai-like client; enables AI scraping
  model="gpt-4o",                # vision-capable model (default: DEFAULT_AI_MODEL)
  include_input_images=True,     # send page images to the VLM for enhanced scraping
  include_output_images=True,    # include scraped images in the returned chunks
  max_input_image_size=1024,     # max pixel size of images sent to the VLM (does not affect output images)
)

To scrape every file in a folder, use scrape_directory. An optional inclusion_pattern regex filters files by their full path:

from thepipe.scraper import scrape_directory
 
chunks = scrape_directory("path/to/folder", inclusion_pattern=r"\.pdf$")

Each returned Chunk has path, text, images (a list of PIL images), and metadata (for PDFs: page number, model used, and figure bounding boxes).

Chunking

To satisfy token-limit constraints, the following chunking methods are available to split the content into smaller chunks.

• chunk_by_document: Returns one chunk with the entire content of the file.
• chunk_by_page: Returns one chunk for each page (for example: each PDF page or PowerPoint slide).
• chunk_by_length: Splits chunks by length (max_tokens, default 10000).
• chunk_by_section: Splits chunks by markdown section (section_separator, default "## ").
• chunk_by_keywords: Splits chunks at a list of keywords.
• chunk_semantic (experimental, requires thepipe-api[semantic]): Returns chunks split by spikes in semantic changes, with a configurable threshold.
• chunk_agentic (experimental, requires an OpenAI-compatible client): Returns chunks split by an LLM agent that attempts to find semantically meaningful sections.

For example,

from thepipe.scraper import scrape_file
from thepipe.chunker import chunk_by_document, chunk_by_page
 
# optionally, pass in chunking_method
# chunk_by_document returns one chunk for the entire document
chunks = scrape_file(
  filepath="paper.pdf",
  chunking_method=chunk_by_document
)
 
# you can also re-chunk later.
# chunk_by_page returns one chunk for each page (for example: each PDF page or PowerPoint slide).
chunks = chunk_by_page(chunks)

OpenAI Chat Integration 🤖

from openai import OpenAI
from thepipe.core import chunks_to_messages
 
# Initialize OpenAI client
client = OpenAI()
 
# Use OpenAI-formatted chat messages
messages = [{
  "role": "user",
  "content": [{
      "type": "text",
      "text": "What is the paper about?"
    }]
}]
 
# Simply add the scraped chunks to the messages
messages += chunks_to_messages(chunks)
 
# Call LLM
response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
)

chunks_to_messages takes in an optional text_only parameter to only output text from the source document. This is useful for downstream use with LLMs that lack multimodal capabilities. You can also pass max_resolution to downscale images before sending them, and include_paths=True to prefix each chunk with its source path.

⚠️ It is important to be mindful of your model's token limit. Be sure your prompt is within the token limit of your model. You can use chunking to split your messages into smaller chunks.

LLamaIndex Integration 🦙

Install the optional extra and then call .to_llamaindex:

pip install thepipe-api[llama-index]

A chunk can then be converted to LlamaIndex Document/ImageDocument with .to_llamaindex().

Structured extraction 🗂️

Extraction uses OpenAI structured outputs (opens in a new tab), so results always conform to your Pydantic schema. An OpenAI-compatible client is required.

from thepipe.extract import extract
from openai import OpenAI
from pydantic import BaseModel
 
client = OpenAI()
 
class LineItem(BaseModel):
    description: str
    amount_usd: float
 
results, tokens_used = extract(
    chunks=chunks,
    schema=LineItem,
    multiple_extractions=True,  # extract multiple rows of data per chunk
    openai_client=client
)

extract returns a list of dicts (one per chunk, each including a chunk_index) and the total tokens used. You can also pass a custom extraction_prompt, or use extract_from_file(...) to scrape and extract in one call.

How it works 🛠️

thepipe uses a combination of computer-vision models and heuristics to scrape clean content from the source and process it for downstream use with large language models (opens in a new tab), or vision-language models (opens in a new tab). You can feed these messages directly into the model, or alternatively you can chunk these messages for downstream storage in a vector database such as ChromaDB, LLamaIndex, or an equivalent RAG framework.

Supported File Types 📚

Web pages, YouTube videos, tweets, and GitHub repositories are no longer scraped by the Python library. To scrape a web page, save it locally as an .html file or PDF first.

Configuration & Environment

Set these environment variables to control image hosting and model defaults:

# Image hosting: instead of embedding base64 data URLs in LLM messages, save
# images to ./images and reference them as {HOST_URL}/images/{id}.jpg.
# HOST_URL must serve that folder and is required when HOST_IMAGES is true.
export HOST_IMAGES=true
export HOST_URL=https://your-domain.com
 
# Control scraping defaults
export DEFAULT_AI_MODEL=gpt-4o
export DEFAULT_EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2
 
# Window length (in seconds) for splitting audio/video transcripts into chunks
export MAX_WHISPER_DURATION=600
 
# Used by the CLI when --openai-api-key is not passed
export OPENAI_API_KEY=sk-...

CLI Usage

thepipe <source> [options]

Where <source> is a file path or directory. Output is written to thepipe_output/.

AI scraping options

--openai-api-key=KEY To enable VLM scraping, pass in your OpenAI API key (defaults to the OPENAI_API_KEY environment variable)

--openai-model=MODEL Model to use for scraping (default is DEFAULT_AI_MODEL, currently gpt-4o)

--openai-base-url=URL Custom LLM endpoint, for local LLMs or hosted APIs like OpenRouter (default: https://api.openai.com/v1 (opens in a new tab))

--max-input-image-size=PIXELS Max size of the largest axis of images sent to the VLM (does not affect output images)

General scraping options

--text-only Output text only (suppress images)

-i REGEX, --inclusion-pattern=REGEX Include only files whose full path matches REGEX (for dirs/zips)

-v, --verbose Print detailed progress messages