Question
General Q: Is computation run on a Tenstorrent Tensix core accelerator card guaranteed to be deterministic?
Specifically, imagine you set up several (n+1; n>1) servers with identical hardware + a N300S card and identical Ubuntu 20.04 + TT-BUDA software stack (including all dependencies), then run the same AI model (from tt-buda-demos/model_demos) with the same configuration/inputs on each server. Would you expect to get exactly the same output from the model run on every server?
Assumptions: there is no randomness intentionally added to the model runs, only interested in potential sources of non-determinism in the hardware architecture, multi-core parallel computation or TT-BUDA/Metalium software stack.
Answer
Yes the output should be identical run to run assuming no randomness in the model. Same input => same output.