MAMBA from Scratch: Neural Nets Better and Faster than Transformers

As someone actively working on this stuff, this channel has the best explanations on the internet, and the 'tuber actually understands what is going on.

wow, you've made some difficult i mean extremely difficult algorithms look easy. thank you.

One small note on RNN's, reservoir computing is a very high dimensional random RNN with linear regression readout, therefore there is no exploding nor vanishing gradient. Reservoir computing is currently the standard for non-linear dynamic time series prediction

Thanks for the clear explanation. This gives me enough understanding to not only implement it myself, but to also have some ideas for sensible architecture modifications.

please open your community tab your content is incredible

Nice video! I just wanted to point out that the parallel scan algorithm can be also implemented in O(n) time (instead of the O(n log(n)) version peresented in the video. and this is the version that the MAMBA uses.

About peer review: As one comment noted, there could be many more candidate papers presented than could be accommodated at the venue. However, this video argues, the rejection justification for this paper is inadequate at best. Some comments ask whether the rejection is important; for academics, the answer is yes, because presentations and publications count for tenure, promotions, and raises plus continued funding of the research. Since several comments plus the video indicate that the algorithm had already received a lot of publicity, for the sake of the project it may not matter if it can continue to be funded, especially if commercial implementations are successful. What is interesting in any case is that the paper exists; in effect it has been published; the authors may not get the desired credit for formal publication, but their work and the reviewer comments are out there now. A couple of decades ago that would not have been the case; most people in the field would be unaware of the algorithm. In terms of peer review, in general (outside of AI), in my field, one of the natural sciences, a paper I submitted for publication encountered an editor plus two reviewers who were well qualified in the field; after asking for two revisions to the manuscript, the third version was rejected. Interestingly, all three scientists had published research which my paper undermined; they may well have lost funding for their research or even their position had that manuscript of mine been published (I speculate here). Peer review cuts both ways. While iterating with the editor and reviewers I continued to expand my research project and made some additional discoveries. Following the rejection I wrote a completely different paper which incorporated my initial work supplemented by the new discoveries; happily it was published a few months ago (in a different journal). I'm formally retired now, but continue to do research. To young researchers -- never give up. Learn from rejection, refine your work, be humble, exercise integrity and honesty, and take pride in your accomplishments, even if only a few know about them. Peer review (by humans) is a necessity and will continue to be. There is no such thing as a perfect filter, but science and technology would be overwhelmed by irrelevancy, dishonesty, and duplication of effort without it. AI may become a useful filtering tool, but science is a human endeavor.

Absolutely amazing vid. Just subbed after getting recommended to this channel. Never stop making videos dude <3

absolutely love the quality and information of this video!!! please keep up the good work this is amazing

Currently testing it on molecular generation, so excited to see where these strengths hold and where they falter :)

Thank you! Your channel is an invaluable resource on here. Hope you keep making these videos!

The level of details and intuition you dig into are excellent 💯🔥

Brutal. I'm going to have to watch this about 30 times. Love it.

Awesome video. I love the speed and the depth of this, it's perfect

I love how you nail the level of detail in the explanations. Perfect for me at least.

Wow, excellent explaination. It covers all the essense of the paper with just enough math/algo. Thank you so much ! If you dont mind, plz make a video for RWKV (v6 has some new modifications), which is another strong linear RNN model. I am curious how does it compares to mamba.

This was very good, and I hope you make more videos like this!

Amazing explanation, thank you!

You have such a pleasant voice 😊 Thanks for helping me understand better. Please keep making videos. ❤

Subscribed! Thats some 3Blue1Brown level stuff! Amazing!