Integrations

The TensorZero Gateway integrates with the major LLM providers.

Model Providers

Provider	Chat Functions	JSON Functions	Streaming	Tool Use	Embeddings
Anthropic	✅	✅	✅	✅	❌
AWS Bedrock	✅	✅	✅	✅	❌
Azure OpenAI Service	✅	✅	✅	✅	❌
Fireworks AI	✅	✅	✅	✅	❌
GCP Vertex AI	✅	✅	✅	✅	❌
Google AI Studio	✅	✅	✅	✅	❌
Mistral	✅	✅	✅	✅	❌
OpenAI (& OpenAI-Compatible)	✅	✅	✅	✅	✅
Together AI	✅	✅	✅	❌	❌
vLLM	✅	✅	✅	❌	❌

Guides

See the following guides for more information on how to use the TensorZero Gateway with each provider:

• Getting Started with Anthropic
• Getting Started with AWS Bedrock
• Getting Started with Azure OpenAI Service
• Getting Started with Fireworks AI
• Getting Started with GCP Vertex AI Anthropic
• Getting Started with GCP Vertex AI Gemini
• Getting Started with Google AI Studio
• Getting Started with Mistral
• Getting Started with OpenAI
• Getting Started with OpenAI-Compatible Endpoints (e.g. Ollama)
• Getting Started with Together AI
• Getting Started with vLLM (+ Llama 3.1)

Limitations

The TensorZero Gateway makes a best effort to normalize configuration across providers. For example, certain providers don’t support tool_choice: required; in these cases, TensorZero Gateway will coerce the request to tool_choice: auto under the hood.

Currently, Fireworks AI and OpenAI are the only providers that support parallel_tool_calls. Additionally, TensorZero Gateway only supports strict for OpenAI (Structured Outputs) and vLLM (Guided Decoding).

Below are the known limitations for each supported model provider.

• Anthropic
  • The Anthropic API doesn’t support consecutive messages from the same role.
  • The Anthropic API doesn’t support tool_choice: none.
  • The Anthropic API doesn’t support seed.
• AWS Bedrock
  • The TensorZero Gateway currently doesn’t support AWS Bedrock guardrails and traces.
  • The TensorZero Gateway uses a non-standard structure for storing ModelInference.raw_response for AWS Bedrock inference requests.
  • The AWS Bedrock API doesn’t support tool_choice: none.
  • The AWS Bedrock API doesn’t support seed.
• Azure OpenAI Service
  • The Azure OpenAI Service API doesn’t provide usage information when streaming.
  • The Azure OpenAI Service API doesn’t support tool_choice: required.
• Fireworks AI
  • The Fireworks API doesn’t support seed.
• GCP Vertex AI
  • The TensorZero Gateway currently only supports the Gemini and Anthropic models.
  • The GCP Vertex AI API doesn’t support tool_choice: required for Gemini Flash models.
  • The Anthropic models have the same limitations as those listed under the Anthropic provider.
• Mistral
  • The Mistral API doesn’t support seed.