Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP
Sasha Rush · 3 HN recommendations
Video Q&A
Ask AI to explain this scene or summarize the whole video. When an answer includes timestamps, select one to jump to that moment.
Try asking: What is the speaker explaining here?
Google sign-in is required. Review the maximum credit cost before confirming.
Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.
Why HN recommends it: HN commenters recommend it for building intuition about state space models and the tradeoffs between transformer and SSM layers.
“I highly recommend Sasha Rush's video on it” Read on HN ↗