Masked Autoencoders (MAE) Paper Explained

Paper link: arxiv.org/abs/2111.06377
In this video, I explain how masked autoencoders work by inspiring ideas from BERT paper and pertaining a vision transformer without requiring any additional label.
Table of Content:
00:00 Intro
00:19 BERT idea
02:09 Language and vision difference
05:29 Proposed Architecture
11:30 After pertaining
14:03 Masking ratio

[@pythonerdhanabanshi4554]

Hello man....my thanks are for you. Your realization with MAE is clean, which is absent in almost all other explanations on CZcams.

[@soroushmehraban]

Glad you enjoyed it!

[@dddz7738]

presentation skills = lit !

[@soroushmehraban]

Thanks 😃

[@mehdipashazadeh9071]

awesome

[@user-fm7ol4jr9o]

explained in detail

[@soroushmehraban]

Glad you liked it😃

[@joseluisdiaz233]

Thanks

[@rohollahhosseyni8564]

Great

[@lorryzou9367]

Kaiming He is just the best

[@soroushmehraban]

Indeed

[@alihadimoghadam8931]

🫡 well done

[@buh357]

does it work on small dataset ? let say 1000 images?

[@soroushmehraban]

I don't think so. Transformers are data hungry and need a lot of data to generalize. the smallest dataset for pretraining that I saw was on ViTPose that they pretrained using this technique on 150k images and when they doubled the data it got only 1.3% better.