crawlberg

Rust crate for crawlberg — the core high-performance web crawling engine used by every language binding. Provides async crawling, structured scrape output, SSRF-safe networking, browser rendering, and Markdown conversion.

What This Package Provides

• Same crawler as every binding — one Rust engine behind Python, Node.js, Ruby, Go, Java, .NET, PHP, Elixir, Dart, Kotlin Android, Swift, Zig, WASM, and C FFI.
• Structured scrape output — HTML, Markdown, metadata, links, assets, response headers, and extraction warnings with consistent field names.
• Crawl controls — depth, page limits, concurrency, URL filters, robots/sitemap handling, rate limits, and partial failure reporting.
• Rendering path — optional browser rendering for JavaScript-heavy pages; direct HTTP path for fast static pages.

Installation

cargo add crawlberg

Agent plugin

The crawlberg plugin is available via the xberg-io/plugins marketplace.

/plugin marketplace add xberg-io/plugins
/plugin install crawlberg@xberg

Works with Claude Code, Codex, Cursor, Gemini CLI, Factory Droid, GitHub Copilot CLI, and opencode. See the marketplace README for harness-specific install instructions.

Quick Start

use crawlberg::{crawl, create_engine, scrape, CrawlConfig, CrawlError};

#[tokio::main]
async fn main() -> Result<(), CrawlError> {
    // Simplest case: scrape a single page with default settings.
    let engine = create_engine(None)?;
    let result = scrape(&engine, "https://example.com/").await?;
    println!("Title: {}", result.metadata.title.as_deref().unwrap_or(""));
    println!("Status: {}", result.status_code);
    println!("Links found: {}", result.links.len());

    // Crawl from a seed URL, limited to one hop and a handful of pages.
    let config = CrawlConfig::builder().max_depth(1).max_pages(5).build();
    let crawl_engine = create_engine(Some(config))?;
    let crawl_result = crawl(&crawl_engine, "https://en.wikipedia.org/wiki/Web_scraping").await?;
    println!("Pages crawled: {}", crawl_result.pages.len());

    Ok(())
}

API Reference

Full API documentation is available at docs.crawlberg.xberg.io.

Key functions:

• create_engine(config?) — Create a crawl engine with optional configuration
• scrape(engine, url) — Scrape a single URL
• crawl(engine, url) — Crawl a website following links
• map_urls(engine, url) — Discover all pages on a site
• batch_scrape(engine, urls) — Scrape multiple URLs concurrently
• batch_crawl(engine, urls) — Crawl multiple seed URLs concurrently

Contributing

Contributions are welcome! Please see our Contributing Guide for details.

Part of Xberg.io

• Xberg — the open-source content-intelligence engine: text, tables, and metadata from 101 formats (115 file extensions), with OCR, transcription, and code intelligence. MIT.
• Xberg Pro — a complete self-hosted content-intelligence backend in a single container. Commercial.
• Xberg Enterprise — the distributed, governed content-intelligence platform, scaled on Kubernetes with team governance and support. Commercial.
• crawlberg — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
• html-to-markdown — fast, lossless HTML→Markdown engine.
• liter-llm — universal LLM API client with native bindings for 14 languages and 165 providers.
• tree-sitter-language-pack — tree-sitter grammars and code-intelligence primitives.
• alef — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.
• Discord — community, roadmap, announcements.

License

This project is licensed under MIT License.

Links

• Documentation
• GitHub Repository
• Issue Tracker