Qwen3.8-27B on the iPhone 18 Neural Engine at 15–70 tok/s (1.5 bpw)
Confirmed 24K context on iPhone with both K/V on int8. Delta net stays FP16. Decoding also peaks at 70 tok/s with easy to predict context.
Apple Neural Engine on A20 is fast 🚀
Apple Neural Engine on A20 is fast 🚀
Summary
The latest findings reveal that the iPhone 18’s Neural Engine can handle impressive decoding speeds of up to 70 tokens per second, leveraging a 24K context with efficient int8 processing. This performance highlights significant advancements in Apple’s A20 chip technology.
Comments (6 of ) ⌛