The TensorZero Gateway is designed from the ground up with performance in mind. It achieves <1ms P99 latency overhead under extreme load (see Benchmarks). Even in default settings, the gateway is fast and lightweight enough to be unnoticeable in most applications.

​
Optimizations

If you care about extreme concurrency and low latency, we recommend the following settings and workflows.

​
TensorZero Gateway

  • Enable gateway.observability.async_writes to offload the responsibility of writing inference responses to ClickHouse to a background task, instead of waiting for ClickHouse to return the inference response. Learn more →
  • Ensure your application, the TensorZero Gateway, and ClickHouse are deployed in the same region to minimize network latency.

​
Python Client

  • Initialize the client once and reuse it as much as possible, to avoid initialization overhead and to keep the connection alive.