Amazon SageMaker added 13 inference capabilities so far in 2026
AWS recaps this year's SageMaker AI inference launches, including automated deployment recommendations, instance fallback pools, an OpenAI-compatible API and container caching.
AWS has published a recap of the 13 inference capabilities it shipped for Amazon SageMaker AI so far in 2026. They cover two deployment paths: managed endpoints, where AWS handles provisioning, scaling and monitoring, and SageMaker HyperPod Inference for teams that want Kubernetes-native control over dedicated GPU clusters. Seven of the launches land on the managed endpoint side.
Recommendations and capacity fallback
Picking an instance type, serving container and optimization settings for a generative model normally means benchmarking more than 1,000 combinations by hand over two to three weeks, according to AWS. Inference recommendations automate that: the customer names a model and a goal (cost, latency or throughput) and gets back a SageMaker Model Package with deployment-ready configurations plus time to first token, inter-token latency, P50/P90/P99 percentiles, throughput and a cost projection. In the example AWS shows, throughput optimization on GPT-OSS-20B doubled tokens per second at the same request latency. Generating recommendations carries no extra charge.
Instance pools address capacity shortages, which previously could stop an endpoint before it served a single request. Teams list up to five instance types in priority order; SageMaker falls back down the list at creation and during scale-out, and removes fallback instances first during scale-in. Each pool entry can point to its own optimized model configuration.
OpenAI-compatible path and faster scale-out
SageMaker endpoints now expose an /openai/v1 path supporting Chat Completions with streaming. Applications built on the OpenAI SDK, LangChain or Strands Agents only need to change the endpoint URL; authentication uses bearer tokens derived from existing AWS credentials and valid for up to 12 hours. The feature is available in 14 AWS Regions.
Container caching pre-pulls serving images so new instances start with the container already local. Running Qwen3-8B on ml.g6.2xlarge with the LMI container (17.7 GB compressed), end-to-end startup latency fell from 525 to 258 seconds, a 51% cut, while model download dropped from 168 to 77 seconds. Early access customers reported improvements between 38% and 65%.
The full list and a comparison of the two paths are in the AWS blog post.
