Comparison: TensorZero vs. Portkey

TensorZero and Portkey offer diverse features to streamline LLM engineering, including an LLM gateway, observability tools, and more. TensorZero is fully open-source and self-hosted, while Portkey offers an open-source gateway but otherwise requires a paid commercial (hosted) service. Additionally, TensorZero has more features around LLM optimization (e.g. advanced fine-tuning workflows and inference-time optimizations), whereas Portkey has a broader set of features around the UI (e.g. prompt playground).

Similarities

• Unified Inference API. Both TensorZero and Portkey offer a unified inference API that allows you to access LLMs from most major model providers with a single integration, with support for structured outputs, batch inference, tool use, streaming, and more.
  → TensorZero Gateway Quick Start

• Automatic Fallbacks, Retries, & Load Balancing for Higher Reliability. Both TensorZero and Portkey offer automatic fallbacks, retries, and load balancing features to increase reliability.
  → Retries & Fallbacks with TensorZero

• Experimentation (A/B Testing or Canary Testing). Both TensorZero and Portkey offer experimentation features to help you test your prompts and models.
  → Experimentation (A/B Testing) with TensorZero

• Schemas, Templates. Both TensorZero and Portkey offer schema and template features to help you manage your LLM applications.
  → Prompt Templates & Schemas with TensorZero

• Multimodal Inference. Both TensorZero and Portkey support multimodal inference (VLMs).
  → Multimodal Inference with TensorZero

Key Differences

TensorZero

• Open-Source Observability. TensorZero offers built-in open-source observability features, collecting inference and feedback data in your own database. Portkey also offers observability features, but they are limited to their commercial (hosted) offering.

• Built-in Evaluations. TensorZero offers built-in evaluation functionality, including heuristics and LLM judges. Portkey doesn’t offer any evaluation features.
  → TensorZero Evaluations Tutorial

• Open-Source Inference Caching. TensorZero offers open-source inference caching features, allowing you to cache requests to improve latency and reduce costs. Portkey also offers inference caching features, but they are limited to their commercial (hosted) offering.
  → Inference Caching with TensorZero

• Open-Source Fine-Tuning Workflows. TensorZero offers open-source built-in fine-tuning workflows, allowing you to create custom models using your own data. Portkey also offers fine-tuning features, but they are limited to their enterprise ($$$) offering.
  → Fine-Tuning Recipes with TensorZero

• Advanced Fine-Tuning Workflows. TensorZero offers advanced fine-tuning workflows, including the ability to curate datasets using feedback signals (e.g. production metrics) and the ability to use RLHF for reinforcement learning. Portkey doesn’t offer similar features.
  → Fine-Tuning Recipes with TensorZero

• Inference-Time Optimizations. TensorZero offers built-in inference-time optimizations (e.g. dynamic in-context learning), allowing you to optimize your inference performance. Portkey doesn’t offer any inference-time optimizations.
  → Inference-Time Optimizations with TensorZero

• Programmatic & GitOps-Friendly Orchestration. TensorZero can be fully orchestrated programmatically in a GitOps-friendly way. Portkey can manage some of its features programmatically, but certain features depend on its external commercial hosted service.

Portkey

• Access Control. Portkey offers access control features, including virtual keys and budgets; that said, these features are only available on their commercial (hosted) offering. TensorZero doesn’t offer built-in access control features, and instead requires you to manage it externally (e.g using Nginx).

• Prompt Playground. Portkey offers a prompt playground in its commercial (hosted) offering, allowing you to test your prompts and models in a graphical interface. TensorZero doesn’t offer a prompt playground today (coming soon!).

• Guardrails. Portkey offers guardrails features, including integrations with third-party guardrails providers and the ability to use custom guardrails using webhooks. For now, TensorZero doesn’t offer built-in guardrails, and instead requires you to manage integrations yourself.

• Managed Service. Portkey offers a paid managed (hosted) service in addition to the open-source version. TensorZero is fully open-source and self-hosted.