TensorZero Gateway Clients

The TensorZero Gateway can be used with the TensorZero Python client, with OpenAI Python client, or via its HTTP API in any programming language.

Python

TensorZero Client

The TensorZero client offers the most flexibility. It can be used with a built-in embedded (in-memory) gateway or a standalone HTTP gateway. Additionally, it can be used synchronously or asynchronously.

You can install the TensorZero Python client with pip install tensorzero.

Embedded Gateway

The TensorZero Client includes a built-in embedded (in-memory) gateway, so you don’t need to run a separate service.

Synchronous

from tensorzero import TensorZeroGateway

with TensorZeroGateway.build_embedded(
    clickhouse_url="http://chuser:chpassword@localhost:8123",  # optional: for observability
    config_file="config/tensorzero.toml",  # optional: for custom functions, models, metrics, etc.
) as client:
    response = client.inference(
        model_name="openai::gpt-4o-mini",  # or: function_name="your_function_name"
        input={
            "messages": [
                {
                    "role": "user",
                    "content": "Write a haiku about artificial intelligence.",
                }
            ]
        },
    )

Asynchronous

from tensorzero import AsyncTensorZeroGateway


async with await AsyncTensorZeroGateway.build_embedded(
    clickhouse_url="http://chuser:chpassword@localhost:8123",  # optional: for observability
    config_file="config/tensorzero.toml",  # optional: for custom functions, models, metrics, etc.
) as gateway:
    inference_response = await gateway.inference(
        model_name="openai::gpt-4o-mini",  # or: function_name="your_function_name"
        input={
            "messages": [
                {
                    "role": "user",
                    "content": "Write a haiku about artificial intelligence.",
                }
            ]
        },
    )

    feedback_response = await gateway.feedback(
        inference_id=inference_response.inference_id,
        metric_name="task_success",  # assuming a `task_success` metric is configured
        value=True,
    )

Standalone HTTP Gateway

The TensorZero Client can optionally be used with a standalone HTTP Gateway instead.

Synchronous

from tensorzero import TensorZeroGateway

# Assuming the TensorZero Gateway is running on localhost:3000...

with TensorZeroGateway.build_http(gateway_url="http://localhost:3000") as client:
    # Same as above...

Asynchronous

from tensorzero import AsyncTensorZeroGateway

# Assuming the TensorZero Gateway is running on localhost:3000...

async with await AsyncTensorZeroGateway.build_http(gateway_url="http://localhost:3000") as client:
    # Same as above...

OpenAI Client

You can use the OpenAI client to run inference requests with TensorZero. You can deploy the TensorZero Gateway as a separate service and configure the OpenAI client to talk to the TensorZero Gateway.

See Deployment for instructions on how to deploy the TensorZero Gateway.

from openai import OpenAI

# Assuming the TensorZero Gateway is running on localhost:3000...

with OpenAI(base_url="http://localhost:3000/openai/v1") as client:
    response = client.chat.completions.create(
        model="tensorzero::model_name::openai::gpt-4o-mini",
        messages=[
            {
                "role": "user",
                "content": "Write a haiku about artificial intelligence.",
            }
        ],
    )

Tip

In the OpenAI interface, the model parameter should be one of the following:

tensorzero::function_name::<your_function_name>

For example, if you have a function named generate_haiku, you can use tensorzero::function_name::generate_haiku.

tensorzero::model_name::<your_model_name>

For example, if you have a model named my_model in the config file, you can use tensorzero::model_name::my_model. Alternatively, you can use default models like tensorzero::model_name::openai::gpt-4o-mini.

You need to use the TensorZero Gateway for feedback requests.

Other Languages and Platforms

The TensorZero Gateway exposes every feature via its HTTP API. You can deploy the TensorZero Gateway as a standalone service and interact with it from any programming language by making HTTP requests.

See Deployment for instructions on how to deploy the TensorZero Gateway.

TensorZero HTTP API

curl -X POST "http://localhost:3000/inference" \
  -H "Content-Type: application/json" \
  -d '{
    "model_name": "openai::gpt-4o-mini",
    "input": {
      "messages": [
        {
          "role": "user",
          "content": "Write a haiku about artificial intelligence."
        }
      ]
    }
  }'

curl -X POST "http://localhost:3000/feedback" \
  -H "Content-Type: application/json" \
  -d '{
    "inference_id": "00000000-0000-0000-0000-000000000000",
    "metric_name": "task_success",
    "value": true,
  }'

OpenAI HTTP API

You can make OpenAI-compatible requests to the TensorZero Gateway.

curl -X POST "http://localhost:3000/openai/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tensorzero::model_name::openai::gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Write a haiku about artificial intelligence."
      }
    ]
  }'

Tip

See above for instructions on how to use the model parameter.