Installation

Prerequisites

• Rust toolchain (stable, 1.75+). Install via rustup if you do not have it.
• Vulkan-capable GPU. Forager uses Burn with the Wgpu backend, which requires Vulkan drivers. Most discrete GPUs and recent integrated GPUs work. Verify with vulkaninfo or vkcube if unsure.
• Internet connection for the first run (model download) and for crawling.

Install

Clone the repository and install the binary:

git clone https://github.com/your-org/forager.git
cd forager
cargo install --path crates/forager

This compiles the full workspace and places the forager binary in your Cargo bin directory (usually ~/.cargo/bin/).

Model download

On first run, forager automatically downloads the MiniLM-L6-v2 sentence transformer model from HuggingFace. This is a ~23 MB download. The model is cached locally and reused for all subsequent runs. No manual setup is needed.

Verify

forager --help

Expected output:

Deep RL web crawler

Usage: forager [OPTIONS] <COMMAND>

Commands:
  new     Create a new crawl package
  import  Import terms from a CSV file
  run     Run or resume a crawl
  status  Show crawl state, parameters, and statistics
  tune    Override a learned parameter
  log     Show crawl run history
  list    List all packages in data/
  query   Run a raw GQL query
  help    Print this message or the help of the given subcommand(s)

Options:
  -d, --db <DB>  Override the database path
  -h, --help     Print help

If you see this, the installation is complete. Proceed to Your First Crawl.