Semiconductor Engineering

Microarchitecture Tailored to 3D-Stacked Near-Memory Processing LLM Decoding (U. of Edinburgh, Peking U., Cambridge et al.)

A new technical paper, “Rethinking Compute Substrates for 3D-Stacked Near-Memory LLM Decoding: Microarchitecture-Scheduling Co-Design,” was published by researchers at University of Edinburgh, Peking ...
EurekAlert!

SPECTRA: Towards a new framework that accelerates large language model inference

This figure shows an overview of SPECTRA and compares its functionality with other training-free state-of-the-art approaches across a range of applications. SPECTRA comprises two main modules, namely ...
Semiconductor Engineering

Arithmetic Intensity In Decoding: A Hardware-Efficient Perspective (Princeton University)

“LLM decoding is bottlenecked for large batches and long contexts by loading the key-value (KV) cache from high-bandwidth memory, which inflates per-token latency, while the sequential nature of ...
Geeky Gadgets

Speculative decoding what is it and why does it matter?

In the rapidly evolving world of technology and digital communication, a new method known as speculative decoding is enhancing the way we interact with machines. This technique is making a notable ...
InfoWorld

Multi-token prediction technique triples LLM inference speed without auxiliary draft models

With reported 3x speed gains and limited degradation in output quality, the method targets one of the biggest pain points in production AI systems: latency at scale. High inference latency and ...
Forbes

The New Frontier Of LLM Inference: Where The Next Tenfold Gains Will Come From

Shakti P. Singh, Principal Engineer at Intuit and former OCI model inference lead, specializing in scalable AI systems and LLM inference. Generative models are rapidly making inroads into enterprise ...
InfoQ

Researchers Open-Source LLM Jailbreak Defense Algorithm SafeDecoding

Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with content, and download exclusive resources. Dany Lepage discusses the architectural ...