Source Types
The scraping and extraction functions can handle various source types in different manners:
| Source Type | Input pattern | Multimodal | Notes |
|---|---|---|---|
| ✔️ | Page markdown and page images. With a VLM client, each page is transcribed to clean markdown and figures/tables are cropped using bounding boxes. Works on scanned documents | ||
| Word Document | *.docx | ✔️ | Scrapes text, tables, and images |
| PowerPoint | *.pptx | ✔️ | Scrapes text and images from each slide |
| Image | *.jpg, *.jpeg, *.png | ✔️ | Returned as an image chunk for downstream use with a VLM |
| Spreadsheet | *.csv, *.xls, *.xlsx | ❌ | Scrapes each row to a chunk containing JSON (row_index added as key) |
| Jupyter Notebook | *.ipynb | ✔️ | Scrapes markdown, code, outputs, and images |
| HTML | *.html (local files) | ❌ | Converts HTML to markdown |
| Plain Text | *.txt, *.md, etc | ❌ | Scrapes text content |
| Video | *.mp4, *.mov, *.wmv | ✔️ | Transcribes audio with Whisper and extracts frames (requires thepipe-api[audio] and ffmpeg) |
| Audio | *.wav, *.mp3 | ✔️ | Transcribes audio with Whisper (requires thepipe-api[audio] and ffmpeg) |
| ZIP Archive | *.zip | ✔️ | Extracts contents and scrapes each file |
| Directory | path/to/folder | ✔️ | Recursively scrapes every file, with an optional inclusion_pattern regex |
Web pages, GitHub repositories, tweets, and YouTube videos are no longer supported as sources in the Python library.