🇺🇸 English 🇰🇷 한국어 🇯🇵 日本語
Home › Developer Tools
Developer Tools

AWS Opens SGLang-Neuron for Trainium Chips

AWS made SGLang-Neuron publicly available on Nov. 12, 2026, letting developers run the open-source SGLang inference engine on Trainium chips with the same interface they use on GPUs. The Apache 2.0 public preview lists nine LLM checkpoints, including Kimi-K3 served across three Trainium3 nodes, but no speed or price benchmarks.

Amazon Web Services on Nov. 12 made SGLang-Neuron publicly available, letting developers run the open-source SGLang inference engine on AWS's own Trainium AI chips. The plugin, released by RadixArk with Amazon's Annapurna Labs, ships as an AWS Deep Learning Container and serves models "using the same SGLang interface you use today," according to the AWS Machine Learning Blog.

Key takeaways

  • SGLang-Neuron v0.1.0 is a public preview under the Apache 2.0 license; large language models run on Trainium3 (Trn3) instances and diffusion models on trn2.48xlarge.
  • Nine LLM checkpoints are listed, from Qwen3-0.6B to Moonshot AI's 2.8-trillion-parameter Kimi-K3, which is served across three Trainium3 nodes with prompts of up to 1 million tokens.
  • AWS's announcement includes no throughput, latency or price comparison with GPUs.
Server cabinets lined up inside a hosting data center
Server cabinets in a data center, file photo. SGLang-Neuron targets cloud servers built on AWS Trainium chips. · 사진: NeoSpire / Flickr(CC BY 2.0)

What SGLang-Neuron does on AWS Trainium

SGLang-Neuron is an "out-of-tree" plugin, meaning it lives outside SGLang's main code. Installing it registers a Neuron platform, and setting SGLANG_PLATFORM=neuron selects it when the server starts. Developers then launch models and call the endpoint with the same generation APIs they use on GPUs.

SGLang's core still runs the API server, the request lifecycle, RadixAttention and prefix caching. The plugin supplies the Trainium-specific pieces: models, attention, mixture-of-experts layers and the graph runner. It runs on TorchNeuron, AWS's native PyTorch backend, with custom kernels written in the Neuron Kernel Interface and compiled through torch.compile.

AWS says the split lets Trainium support and SGLang's core "develop independently and iterate faster." The plugin is pinned to one SGLang commit, 43881a7, and stops a launch on any other revision. It requires Neuron SDK 2.33.0.

Which models SGLang-Neuron supports

The project's GitHub page lists OpenAI's gpt-oss-120b, Qwen3-0.6B, Qwen3-8B, Qwen3-30B-A3B, GLM-4-9B, DeepSeek-V2-Lite, GLM-5.2, Kimi-K3 and Qwen3.5-397B-A17B. It also lists five diffusion pipelines, including FLUX.1-dev and the Wan2.2 A14B video models.

The largest deployment is Kimi-K3. It runs "disaggregated": one Trainium3 node handles the prefill stage, which reads the prompt, and two nodes generate the answer. The KV cache moves between them using NIXL over AWS's Elastic Fabric Adapter network. GPT-OSS-120B and GLM-5.2 also get "giga-kernel" decoding, which runs every layer of a decode step in one kernel call.

Limits developers should know

The README calls the release a public preview and warns that features, APIs and supported models can change. The plugin refuses LoRA adapters, embedding servers and SGLang's weight-update endpoints. By default it samples greedily, so requests must set temperature to 0 unless the server launches with a separate sampler setting.

Context: from limited beta to public release

LMSYS, the nonprofit behind SGLang, opened a limited beta on Oct. 22 for Trainium2 and Trainium3. In its first reinforcement-learning test, SGLang generated rollouts for Qwen3-30B-A3B on Trainium3 while LMSYS's Miles framework trained the policy on a single NVIDIA H200 node. LMSYS said the run was stable.

The release lands days after NVIDIA, AMD and Intel joined LMSYS's new SGLang Consortium on Nov. 12. AWS is pushing the same idea seen in its TorchNeuron launch: make Trainium look like any other PyTorch device so teams can move work off GPUs with few code changes.

What's next

The open question is cost and speed. AWS has not published head-to-head numbers, so developers will need to test their own workloads. Watch whether the SGLang pin moves to newer upstream releases and whether more of the plugin's patches land in SGLang itself, as the project says it intends.

FAQ

Is SGLang-Neuron free to use?
Yes. The code is open source under Apache 2.0. Running it requires paid Trainium instances on AWS.

Which Trainium chips does it support?
In this release, LLMs run on Trainium3 instances and diffusion models on trn2.48xlarge, according to the README.

Do I need to rewrite my SGLang code?
AWS says no. Models launch through SGLang's usual server and generation APIs; the plugin is selected with one environment variable.

References

#SGLang-Neuron #AWS Trainium #Trainium3 #SGLang #RadixArk #Annapurna Labs #TorchNeuron #LLM inference #Kimi-K3 #open source
S
Seung Jung
AI Newsway · Developer

Editor responsible for AI Newsway. Writes about AI, developer tools, cloud and productivity technology.

More by this reporter ›
Copyright © AI Newsway All rights reserved. Unauthorized reproduction, redistribution, and AI training prohibited.
Share
f X K

Comments 0

Comments are currently disabled.

Be the first to comment.