I forgot to mention that this model is trained like a normal transformer and since everything is causal, you should be able to train using the efficient parallel technique that the transformer uses, a single forward pass for an entire sequence of data.
I just opened your channel to ask your for Mamba video and here I see this video. You are awesome dude, I can express how much you contribute to my life. Thank you many times!!!
19:50 I think A is DxN because they use diagonal matrix. They mention S4D, and that paper has example of also linear initialization: "A = -0.5 + 1j * np.pi * np.arange(N//2) # S4D-Lin initialization". It's structured after all.
Wow Gabrial, Great job! I like your calm attitude and simple way of explaining this complex subject! As electrical engineer and as a data scientist I highly appreciate your content!
I was actually wrong about the HiPPO "A" matrix being learnable. I think this matrix is actually static, which makes sense as it adds some basic structure to the model.
I forgot to mention that this model is trained like a normal transformer and since everything is causal, you should be able to train using the efficient parallel technique that the transformer uses, a single forward pass for an entire sequence of data.
I just opened your channel to ask your for Mamba video and here I see this video. You are awesome dude, I can express how much you contribute to my life. Thank you many times!!!
19:50 I think A is DxN because they use diagonal matrix. They mention S4D, and that paper has example of also linear initialization: "A = -0.5 + 1j * np.pi * np.arange(N//2) # S4D-Lin initialization". It's structured after all.
Please don't stop with this videos. They are extremely useful to go through with you. Much love
Wow Gabrial, Great job!
I like your calm attitude and simple way of explaining this complex subject!
As electrical engineer and as a data scientist I highly appreciate your content!
Thanks for the vid. I Can't wait to see if it's overhyped or not hehe. TriDao knows his attention mechanisms.
thx for doing this paper, was a bit lost on state space models
I was a bit lost.. now I'm more lost. ;)
@@acasualviewer5861haha, i did watch some lectures by the first author tho
shouldn't 24:28 A,B, and C be LxN not LxD ?
I think it's independent because you can diagonalize the state transition matrix and then each value only interacts with itself.
점점 물리학의 개념들에 가까워지는 기분이.. 🙂
If all the matrices are learnable, I wonder why the authors use the HiPPO matrix to initialize A？ What's the point?
I was actually wrong about the HiPPO "A" matrix being learnable. I think this matrix is actually static, which makes sense as it adds some basic structure to the model.