ONNX Runtime vs. TensorRT in Triton: Where the Crossover Actually Is

On BERT-base (seq len 128, fp16), served through Triton on an A10G, ONNX Runtime’s CUDA execution provider stayed within 9% of TensorRT all the way through batch size 8 — while skipping a 6-minute engine build and the deployment brittleness that comes with a serialized TensorRT plan.

TensorRT did pull away. Just not everywhere. In our run the crossover sat at batch size 16.

“Which backend is faster?” is the wrong question. TensorRT is faster. The useful question is where it gets faster by enough to pay for itself.

The setup

The model was BERT-base, fine-tuned for sequence classification at a fixed sequence length of 128. I picked it because it looks like the inference path we actually care about in production: roughly fixed shapes, GPU-bound execution, and enough tensor work that the backend choice shows up in the numbers.

The environment was pinned. TensorRT performance moves between versions, so this is not optional:

GPU: A10G (24 GB)
Triton: 24.05
CUDA: 12.4
cuDNN: 9.1
TensorRT: 10.0
ONNX Runtime: 1.18
Precision: fp16

Both backends got the same Triton settings wherever it was possible: same input shapes, same dynamic batching policy, same instance count, same client workload. The point was to change the backend and nothing else.

ONNX Runtime, on the CUDA EP:

platform: "onnxruntime_onnx"

instance_group [
  {
    kind: KIND_GPU
    count: 1
  }
]

dynamic_batching {
  preferred_batch_size: [ 4, 8, 16, 32 ]
  max_queue_delay_microseconds: 1000
}

TensorRT, from a prebuilt plan:

platform: "tensorrt_plan"

instance_group [
  {
    kind: KIND_GPU
    count: 1
  }
]

dynamic_batching {
  preferred_batch_size: [ 4, 8, 16, 32 ]
  max_queue_delay_microseconds: 1000
}

Numbers came from perf_analyzer with a concurrency sweep:

perf_analyzer \
  -m bert_base_onnx \
  -i grpc \
  --concurrency-range 1:64:2 \
  --percentile=99 \
  --measurement-mode=time_windows

Warm latency and throughput were measured apart from cold start. TensorRT’s engine build sat outside the hot path. It did not sit outside the accounting.

The curve

The answer was not “TensorRT wins.” It was not “ONNX Runtime is good enough” either. It depended entirely on where the model was operating.

At small batch sizes, ONNX Runtime stayed close:

Batch 1:  TensorRT was 13% faster
Batch 4:  TensorRT was 11% faster
Batch 8:  TensorRT was 9% faster

That gap was not free. It was also not large enough to justify standing up a separate TensorRT build pipeline on its own.

At larger batches, the story changed:

Batch 16: TensorRT was 16% faster
Batch 32: TensorRT was 21% faster
Batch 64: TensorRT was 26% faster

This is where TensorRT does what TensorRT is for: take a stable, optimized shape and squeeze more out of the GPU.

The p99 curve is the one that matters for a service. For latency-bound traffic, the gap between backends was smaller than the gap in operational cost. For throughput-bound traffic, TensorRT’s lead got a lot easier to justify.

The tax TensorRT hides

The latency chart does not show the whole bill.

TensorRT needs an engine build:

Engine build time: 6 min 20 s
Engine size: 263 MB
Optimization profiles: min 1×128, opt 8×128, max 32×128

That matters because the engine is not “the model.” It is bound to the TensorRT version, the CUDA stack, the GPU architecture, and the optimization profiles it was built with. Which buys you a few standing costs:

• a build step before every deployment
• new optimization profiles or new engines when shapes change
• version upgrades that can invalidate old assumptions
• harder debugging once the model is a serialized engine
• backend work when an op is unsupported or awkward

None of that makes TensorRT bad. It means TensorRT has to win by enough to pay for the machinery around it.

The decision rule

The rule I would take from this benchmark:

For this model, TensorRT became clearly worth it at batch size 16.

Below that point, ONNX Runtime was close enough that simplicity won. Above it, the throughput lead was big enough to pay the engine-build tax without complaint.

Appendix: commands

ONNX Runtime run:

perf_analyzer \
  -m bert_base_onnx \
  -i grpc \
  --concurrency-range 1:64:2 \
  --percentile=99

TensorRT run:

perf_analyzer \
  -m bert_base_trt \
  -i grpc \
  --concurrency-range 1:64:2 \
  --percentile=99

Engine build:

trtexec \
  --onnx=model.onnx \
  --saveEngine=model.plan \
  --fp16 \
  --minShapes=input_ids:1x128 \
  --optShapes=input_ids:8x128 \
  --maxShapes=input_ids:32x128

TensorRT was faster. It was not automatically worth it. The question was never which backend tops the chart — it was where the speedup grows large enough to pay for everything the chart leaves out.