points by xlayn 2 days ago | 0 comments
I want to propose a called AI Index, if you can do say 1000t/s on processing and 50t/s on generation on a min 25B model you can call your device AI9 and you have AI7 and AI5 and AI3 with 3 being 100t/s processing and 10t/s generation on a min 10B model.
Values above are on a 100k context size minimum.
OFFLINE.
So any effing claim of being an AI device doesn't end on the demo of best buy laptop where you remove a dog from a photo....
Or that you have "better noise removal" or "you look better on the cam" because that's called noise cancelling and instagram filter and it's being done for ages...
[–] argee 2 days ago
Can't you just look at memory bandwidth for generation?
[–] josephg 2 days ago
You also need to look at memory size (to store the model). And processing can still be the bottleneck.
[–] stingraycharles 2 days ago
It’s more nuanced than that, but yes that’s the biggest factor.
[–] bastawhiz 1 day ago
You can run a model at 1000t/s on a low and android phone. The output will be dogshit bad, but it'll pump tokens like nobody's business. The problem with what you're proposing is that there's no baseline definition of what those tokens are (for what purpose? of what quality?). And pinning that to a certain model or level of correctness means your benchmark becomes almost immediately obsolete.
My Strix Halo now can run some models through the NPU and its roughly as fast as its GPU+CPU (on some of the models - some its slower)