Skip to content
TensorZero Logo TensorZero Logo TensorZero Docs

Getting Started with Hyperbolic

This guide shows how to set up a minimal deployment to use the TensorZero Gateway with the Hyperbolic API.

Simple Setup

You can use the short-hand hyperbolic::model_name to use a Hyperbolic model with TensorZero, unless you need advanced features like fallbacks or custom credentials.

You can use Hyperbolic models in your TensorZero variants by setting the model field to hyperbolic::model_name. For example:

[functions.my_function_name.variants.my_variant_name]
type = "chat_completion"
model = "hyperbolic::meta-llama/Meta-Llama-3-70B-Instruct"

Additionally, you can set model_name in the inference request to use a specific Hyperbolic model, without having to configure a function and variant in TensorZero.

Terminal window
curl -X POST http://localhost:3000/inference \
  -H "Content-Type: application/json" \
  -d '{
    "model_name": "hyperbolic::meta-llama/Meta-Llama-3-70B-Instruct",
    "input": {
      "messages": [
        {
          "role": "user",
          "content": "What is the capital of Japan?"
        }
      ]
    }
  }'

Advanced Setup

In more complex scenarios (e.g. fallbacks, custom credentials), you can configure your own model and Hyperbolic provider in TensorZero. For this minimal setup, you’ll need just two files in your project directory:

  • Directoryconfig/
    • tensorzero.toml
  • docker-compose.yml

For production deployments, see our Deployment Guide.

Configuration

Create a minimal configuration file that defines a model and a simple chat function:

config/tensorzero.toml
[models."meta-llama/Meta-Llama-3-70B-Instruct"]
routing = ["hyperbolic"]


[models."meta-llama/Meta-Llama-3-70B-Instruct".providers.hyperbolic]
type = "hyperbolic"
model_name = "meta-llama/Meta-Llama-3-70B-Instruct"


[functions.my_function_name]
type = "chat"


[functions.my_function_name.variants.my_variant_name]
type = "chat_completion"
model = "meta-llama/Meta-Llama-3-70B-Instruct"

See the list of models available on Hyperbolic.

Credentials

You must set the HYPERBOLIC_API_KEY environment variable before running the gateway.

You can customize the credential location by setting the api_key_location to env::YOUR_ENVIRONMENT_VARIABLE or dynamic::ARGUMENT_NAME. See the Credential Management guide and Configuration Reference for more information.

Deployment (Docker Compose)

Create a minimal Docker Compose configuration:

docker-compose.yml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment


services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    environment:
      - HYPERBOLIC_API_KEY=${HYPERBOLIC_API_KEY:?Environment variable HYPERBOLIC_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"

You can start the gateway with docker compose up.

Inference

Make an inference request to the gateway:

Terminal window
curl -X POST http://localhost:3000/inference \
  -H "Content-Type: application/json" \
  -d '{
    "function_name": "my_function_name",
    "input": {
      "messages": [
        {
          "role": "user",
          "content": "What is the capital of Japan?"
        }
      ]
    }
  }'