Lab note

Debugging an AI accelerator you do not own

For most of the history of specialised AI hardware, contributing to its software stack meant one thing first: getting your hands on the hardware. If you did not have the card, you read the code, you guessed, and you moved on. This year I contributed a bug report to Tenstorrent’s tt-metal without ever owning one of their accelerators. These are my notes on how that worked and why I think the barrier is now much lower than most people assume.

Why I was looking at quantization at all

Part of my research work is on transformer models: how attention-layer weights are distributed across different models, and what those distributions imply for quantization. Quantization is where theory meets the machine. A weight distribution can look friendly on paper, but what actually happens depends on how a given accelerator represents scales, which data types it accepts, and what it refuses.

So I wanted to understand how quantization is implemented for Tenstorrent accelerators specifically. Not in the abstract, but in the code that decides what a real chip will be asked to do.

The setup: no hardware, two tools

The whole setup was two things:

  • ttnn, installed with a plain pip install ttnn. This is the library you use to describe operations and tensors for the hardware.
  • ttsim, Tenstorrent’s simulator, which gives you a virtual Wormhole chip to run against.

That is the part I want people to notice. There was no procurement, no lab access, no borrowed machine. A laptop, a package install and a simulator were enough to start asking real questions of a real production codebase.

The bug: same input, different answer

What I found was a consistency problem in validation. The quantization operation accepted or rejected the very same scale tensor depending on parameters that had nothing to do with the scale itself.

Reading the code, the reason became clear. There were several code paths through the operation. Three of them widened the data type of the scale before checking it. One did not. So whether your scale tensor passed validation depended on which path your other, unrelated settings happened to send you down.

From a user’s point of view this is the worst kind of error. It is not a crash with a clear message. It is an input that is valid in one call and invalid in the next, for reasons you would never think to look at. Anyone hitting it would reasonably assume they had made a mistake.

Why I could see it without running a kernel

The key detail is where the validation happens. In this case it runs host-side, on the CPU, before anything is sent to the device. That means the behaviour I was investigating was fully observable without executing a single kernel on accelerator hardware.

This is worth generalising. A large share of the logic in any hardware software stack lives on the host: argument checks, shape and type rules, dispatch decisions. That code is ordinary software. It can be read, called and tested like any other library, and its bugs are just as real for the people using the hardware as any bug on the chip.

The simulator matters for the rest: it lets you confirm that what you expect to run on the device actually does, without the device.

Reporting it so it could be acted on

I reported the issue on the tt-metal repository. The report was accepted, and a fix has been opened. I do not want to overstate my part: I found an inconsistency and described it clearly. The maintainers did the rest.

But describing it clearly is a real piece of work, and it is the same skill I use as a product owner. A good report says what you expected, what happened instead, and the smallest set of conditions that separates the two. Here that meant showing one scale tensor, two sets of unrelated parameters, and two different outcomes. Once the contrast is that sharp, the cause is half explained.

Why the barrier is low now

I think a lot of capable engineers still assume that AI accelerator software is closed territory unless you work at the company or have the card. For this kind of work, that is no longer true:

  • The library installs like any other Python package.
  • A simulator stands in for the chip.
  • Much of the logic worth checking runs host-side anyway.
  • The code is public, so you can read the paths an operation takes instead of guessing.

None of this replaces testing on real hardware for performance or numerical behaviour on the device. But for understanding how an operation is meant to behave, and for noticing when it does not behave consistently, the tools are already on the table.

What I took from it

Two things. First, consistency is a feature. An operation that gives the same answer for the same input, regardless of unrelated settings, is easier to trust and easier to build on. When I write specifications, I now look for this explicitly: which inputs should never change the outcome?

Second, the distance between research questions and production code is shorter than it looks. I started from a question about weight distributions and quantization. Following that question into the actual implementation is what surfaced the bug. If you are curious about how your models will behave on specialised hardware, the most useful next step may simply be to install the library and read the validation code.

All lab notes