4. KV Cache Compression 900000x Beyond TurboQuant and Per-Vector Shannon Limit (arxiv.org)

Recent work on KV cache quantization, culminating in TurboQuant, has approached the Shannon entropy limit for per-vector compression of transformer key-value caches. We observe that this limit applies to a strictly weaker problem than the one that actually matters: compressing the KV cache as a sequence. The tokens stored in a KV cache are not arbitrary floating-point data -- they are samples from the exact formal language the model was trained on, and the model is by construction a near-optimal predictor of that language. We introduce sequential KV compression, a two-layer architecture that exploits this structure. The first layer, probabilistic prefix deduplication, identifies semantically equivalent shared prefixes across sessions using the trie metric d_T(s, s') = -log_2 P_M(s ^ s') from Probabilistic Language Tries (PLTs). The second layer, predictive delta coding, stores only the residual of each new KV vector from the model's own prediction of it, achieving a per-token entropy bound of H(KV_{i+1} | KV_{<=i}) <= H(token_{i+1} | token_{<=i}). We prove that at typical language model perplexity -- approximately 10-20 for fluent English text -- this bound is 3.3-4.3 bits on average per token position, compared to TurboQuant's 3 bits per vector component (with typical attention heads having 64-128 components). The theoretical compression ratio over TurboQuant is approximately 914,000x at the Shannon limit. Even at 1000x above the entropy floor -- a deliberately pessimistic worst-case overhead, two orders of magnitude above the 2-5x typical of practical source coders -- the ratio remains approximately 914x over TurboQuant, with compression improving rather than degrading as context length grows. The two layers are orthogonal and compose with existing per-vector quantization methods including TurboQuant.

9. Tim Davis – Probabilistic engineering and the 24-7 employee (www.timdavis.com)

Software is quietly becoming a probabilistic system. Generation has become cheap but validation has not, and correctness is becoming something you believe rather than know. The shift from deterministic to probabilistic engineering, the splitting of roles, and the 24-7 employee whose agent fleet work

14. Expansion Artifacts (mattstromawn.com)

Matt Ström-Awn · Designer, leader, and coach focused on building exceptional products and teams.

20. Real-Time Visualization of Human Finger Joint Cavitation (2015) (journals.plos.org)

Cracking sounds emitted from human synovial joints have been attributed historically to the sudden collapse of a cavitation bubble formed as articular surfaces are separated. Unfortunately, bubble collapse as the source of joint cracking is inconsistent with many physical phenomena that define the joint cracking phenomenon. Here we present direct evidence from real-time magnetic resonance imaging that the mechanism of joint cracking is related to cavity formation rather than bubble collapse. In this study, ten metacarpophalangeal joints were studied by inserting the finger of interest into a flexible tube tightened around a length of cable used to provide long-axis traction. Before and after traction, static 3D T1-weighted magnetic resonance images were acquired. During traction, rapid cine magnetic resonance images were obtained from the joint midline at a rate of 3.2 frames per second until the cracking event occurred. As traction forces increased, real-time cine magnetic resonance imaging demonstrated rapid cavity inception at the time of joint separation and sound production after which the resulting cavity remained visible. Our results offer direct experimental evidence that joint cracking is associated with cavity inception rather than collapse of a pre-existing bubble. These observations are consistent with tribonucleation, a known process where opposing surfaces resist separation until a critical point where they then separate rapidly creating sustained gas cavities. Observed previously in vitro, this is the first in-vivo macroscopic demonstration of tribonucleation and as such, provides a new theoretical framework to investigate health outcomes associated with joint cracking.

28. Corner-Case RCU Implementations (people.kernel.org)

RCU has a way of popping up unexpectedly. In the words of Fedor Pikus, "In fact, you may already be using the RCU approach in your progr...

Article Details

Browse by Date

Type to search HN and engineering blog articles
Press Escape to close  |  ↑↓ to navigate  |  Enter to open