TensorZero Gateway Deployment Guide

It’s easy to get started with the TensorZero Gateway.

To deploy the TensorZero Gateway, you need to:

• Setup a ClickHouse database
• Create a configuration file
• Run the gateway

ClickHouse

The TensorZero Gateway stores inference and feedback data in a ClickHouse database. This data is later used for model observability, experimentation, and optimization.

Development

For development purposes, you can run a single-node ClickHouse instance locally (e.g. using Homebrew or Docker) or a cheap Development-tier cluster on ClickHouse Cloud.

See the ClickHouse documentation for more details on configuring your ClickHouse deployment.

Production

Managed Services

For production deployments, the easiest setup is to use a managed service like ClickHouse Cloud.

ClickHouse Cloud is also available through the AWS Marketplace, GCP Marketplace, and Azure Marketplace.

Other options for managed ClickHouse deployments include Tinybird (serverless) and Altinity (hands-on support).

Self-Managed Deployment

You can alternatively run your own self-managed ClickHouse instance or cluster.

Caution

The TensorZero Gateway does not automatically enable data replication for ClickHouse tables.

If you are using a self-managed distributed ClickHouse deployment, you must set up replication yourself. See the ClickHouse replication documentation for more details.

ClickHouse Cloud automatically sets up data replication, so this step is not necessary if you’re using the managed service.

Configuration

After setting up your database, you need to configure the TENSORZERO_CLICKHOUSE_URL environment variable with the connection details. The variable takes a standard format.

.env

TENSORZERO_CLICKHOUSE_URL="http[s]://[username]:[password]@[hostname]:[port]/[database]"

# Example: ClickHouse running locally
TENSORZERO_CLICKHOUSE_URL="http://localhost:8123/tensorzero"

# Example: ClickHouse Cloud
TENSORZERO_CLICKHOUSE_URL="https://USERNAME:[email protected]:8443/tensorzero"

# Example: TensorZero Gateway running in a container, ClickHouse running on host machine
TENSORZERO_CLICKHOUSE_URL="http://host.docker.internal:8123/tensorzero"

The TensorZero Gateway automatically applies database migrations on startup.

Disabling Observability (Not Recommended)

You can disable observability features if you’re not interested in storing any data for experimentation and optimization. In this case, you won’t need to set up ClickHouse, and the TensorZero Gateway will act as a simple model gateway.

To disable observability, set the following configuration in the tensorzero.toml file:

tensorzero.toml

[gateway]
observability.enabled = false

If you only need to disable observability temporarily, you can pass a dryrun: true parameter to the inference and feedback API endpoints.

TensorZero Gateway

Development

Running with Docker (Recommended)

You can easily run the TensorZero Gateway locally using Docker.

You need to provide it with a path to a folder containing your tensorzero.toml file as well as its dependencies (e.g. schemas and templates), as well as the environment variables discussed above.

Running with Docker

docker run \
  --name tensorzero-gateway \
  -v "./config:/app/config" \
  --env-file .env \
  -p 3000:3000 \
  -d \
  tensorzero/gateway

Building from source

Alternatively, you can build the TensorZero Gateway from source and run it directly on your host machine using Cargo:

Building from source

cargo run --release --bin gateway -- path/to/your/tensorzero.toml

Production

You can deploy the TensorZero Gateway alongside your application (e.g. as a sidecar container) or as a standalone service.

A single gateway instance can handle over 1k QPS/core with sub-millisecond latency (see Benchmarks), so a simple deployment should suffice for the vast majority of applications. If you deploy it as an independent service, we recommend deploying at least two instances behind a load balancer for high availability. The gateway is stateless, so you can easily scale horizontally and don’t need to worry about persistence.

Running with Docker (Recommended)

The recommended way to run the TensorZero Gateway in production is to use Docker.

There are many ways to run Docker containers in production. A simple solution is to use Docker Compose. We provide an example docker-compose.yml for reference.

Building from source

Alternatively, you can build the TensorZero Gateway from source and run it directly on your host machine using Cargo. For production deployments, we recommend enabling performance optimizations:

Building from source

cargo run --profile performance --bin gateway -- path/to/your/tensorzero.toml

Configuration

To run the TensorZero Gateway, first you need to create a tensorzero.toml configuration file. Read more about the configuration file here.

Once you have a configuration file, you can run the TensorZero Gateway using one of the following methods.

Model Provider Credentials

In addition to the TENSORZERO_CLICKHOUSE_URL environment variable discussed above, the TensorZero Gateway accepts the following environment variables for provider credentials. Unless you specify an alternative credential location in your configuration file, these environment variables are required for the providers that are used in a variant with positive weight. If required credentials are missing, the gateway will fail on startup.

Provider	Environment Variable(s)
Anthropic	ANTHROPIC_API_KEY
AWS Bedrock	AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION (optional)
Azure OpenAI	AZURE_OPENAI_API_KEY
Fireworks	FIREWORKS_API_KEY
GCP Vertex AI Anthropic	GCP_VERTEX_CREDENTIALS_PATH (see below for details)
GCP Vertex AI Gemini	GCP_VERTEX_CREDENTIALS_PATH (see below for details)
Google AI Studio Gemini	GOOGLE_AI_STUDIO_GEMINI_API_KEY
Hyperbolic	HYPERBOLIC_API_KEY
Mistral	MISTRAL_API_KEY
OpenAI	OPENAI_API_KEY
Together	TOGETHER_API_KEY
xAI	XAI_API_KEY

Notes:

• AWS Bedrock supports many authentication methods, including environment variables, IAM roles, and more. See the AWS documentation for more details.
• If you’re using the GCP Vertex provider, you also need to mount the credentials for a service account in JWT form (described here) to /app/gcp-credentials.json using an additional -v flag.

Tip

See .env.example for a complete example with every supported environment variable.

Clients

Python

We provide a Python client for interacting with the TensorZero Gateway. It supports both synchronous and asynchronous usage.

The library is available on PyPI under the name tensorzero. You can install it using your favorite Python package manager (e.g. pip or uv).

TensorZero is also (mostly) compatible with the OpenAI Python client.

Other

TensorZero can be used via standard HTTP requests in any programming language.