Getting Started with Text Generation Inference (TGI)

This guide shows how to set up a minimal deployment to use the TensorZero Gateway with self-hosted LLMs using Text Generation Inference (TGI).

We’re using Phi-4 in this example, but you can use virtually any model supported by TGI.

Setup

This guide assumes that you are running TGI locally with

docker run \
    --gpus all \
    # Set shared memory size - needed for loading large models and processing requests
    --shm-size 64g \
    # Map the host's port 8080 to the container's port 80
    -p 8080:80 \
    # Mount the host's './data' directory to the container's '/data' directory
    -v $PWD/data:/data \
    ghcr.io/huggingface/text-generation-inference:3.0.1 \
    --model-id microsoft/phi-4

Make sure to update the api_base in the configuration below to match your TGI server.

For this minimal setup, you’ll need just two files in your project directory:

Directoryconfig/
- tensorzero.toml
docker-compose.yml

For production deployments, see our Deployment Guide.

Configuration

Create a minimal configuration file that defines a model and a simple chat function:

[models.phi_4]
routing = ["tgi"]

[models.phi_4.providers.tgi]
type = "tgi"
api_base = "http://host.docker.internal:8080/v1/"  # for TGI running locally on the host
api_key_location = "none"  # by default, TGI requires no API key

[functions.my_function_name]
type = "chat"

[functions.my_function_name.variants.my_variant_name]
type = "chat_completion"
model = "phi_4"

Credentials

The api_key_location field in your model provider configuration specifies how to handle API key authentication:

If your endpoint does not require an API key (e.g. TGI by default):
```
api_key_location = "none"
```
If your endpoint requires an API key, you have two options:
1. Configure it in advance through an environment variable:
```
api_key_location = "env::ENVIRONMENT_VARIABLE_NAME"
```
  You’ll need to set the environment variable before starting the gateway.
2. Provide it at inference time:
```
api_key_location = "dynamic::ARGUMENT_NAME"
```
  The API key can then be passed in the inference request.

See the Configuration Reference and the API reference for more details.

In this example, TGI is running locally without authentication, so we use api_key_location = "none".

Deployment (Docker Compose)

Create a minimal Docker Compose configuration:

# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    # environment:
    #   - TGI_API_KEY=${TGI_API_KEY:?Environment variable TGI_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"

You can start the gateway with docker compose up.

Inference

Make an inference request to the gateway:

curl -X POST http://localhost:3000/inference \
  -H "Content-Type: application/json" \
  -d '{
    "function_name": "my_function_name",
    "input": {
      "messages": [
        {
          "role": "user",
          "content": "What is the capital of Japan?"
        }
      ]
    }
  }'