Marigold: A Self-Hosted Inference API for Open-Weight Models
The major AI providers bundle two things that should be separate: the model and the infrastructure. When you call the OpenAI or Anthropic API, your data travels to their servers, runs against their model, and returns a result. The provider sits between you and the model, whatever their policy says about logging, retention, or training.
Open-weight models remove that party. Llama, Mistral, Qwen, and others publish their weights publicly, and the model can run anywhere. Marigold is that anywhere: an OpenAI-compatible API, a model-serving worker, and a chat interface, all running on hardware you control. No request reaches OpenAI, Anthropic, or any other external provider. Once your models are cached, it runs fully airgapped.
Install and run
pip install bayis-marigold
git clone https://github.com/bayinfosys/marigold-examples
marigold deployment start marigold-examples/chat
The first command installs the marigold CLI. The second pulls a set of
pre-written application packages, each one a models.yaml (which models
to load) and a marigold.toml (how to run it). The third brings up the
full stack for that package, Postgres, a cache-init step that downloads
the declared models, the worker, the API, and (for the chat package)
Open WebUI. The model download is the only point in the process that
needs an internet connection; everything after that runs offline.
The full walkthrough, including how to confirm it’s working and where the model cache lives, is in the setup guide.
A drop-in replacement for the major AI APIs
Marigold exposes the same interface as OpenAI and Anthropic. Existing application code that calls those APIs can point at Marigold instead without changes. The models underneath are open-weight equivalents, running on your own hardware rather than a shared cloud server.
(For the broader architectural case, see Private Inference: Running AI Inside Your Own Infrastructure.)
Models as the stable layer
The model cache is shared across every package on the machine, independent of which one is currently running. Download a model once and every application package that needs it reuses the same copy, rather than each package pulling its own.
Application packages sit on top as the thin, disposable layer: a
models.yaml, a marigold.toml, and application code. Marigold runs
the package; it never runs the package’s own code directly, by design.
This is the same split described in
Models as a Fat Protocol –
the model as the durable, shared layer, applications as thin clients
built on top of it, the shape that made early internet protocols last.
Workflows and pipelines
Single model calls handle straightforward tasks. More complex automation requires composing multiple steps: embed a document, classify its content, generate a structured summary, evaluate the output. Marigold supports this through a typed workflow layer, where each step conditions on the outputs of the previous one and the whole pipeline is declared rather than hand-coded.
This is the same pattern described in the runfox workflow engine, which Marigold uses directly as its execution substrate.
What it’s used for
The model types cover more than chat: text and image embedding, instruction-following, image generation, image-to-text, image segmentation, depth estimation, text-to-speech, and evaluation models for scoring text, image, and text-image alignment. In combination, these support workflows like quality-gated image generation (generate, score, regenerate until a threshold is met), visual conformance checking (compare an observation against a reference, diff the discrepancy, describe it in natural language), and batch tabular classification against a small labelled set, with low-confidence predictions routed to an instruct model for explanation.
Getting started
Marigold is open source, on GitHub and PyPI. The tutorials cover setup, local document search, and adding a new model to an existing package; the examples are pre-built application packages to clone and run directly. Sizing your own hardware for this is its own decision: see our GPU options guide for what VRAM and which vendors actually support self-hosted inference in practice.
Marigold publishes benchmark data and implementation notes on its own Build Log.