Genny GPT: The Science of Transformer Features, Visually Explained!

A visual explanation of how transformers piece concepts together, told in the style of 3Blue1Brown. Introducing SAEs.
What truly goes on in the inner thoughts of AI language models, or LLMs, like ChatGPT? To find this answer, we must detangle their internal mechanisms into what are called Features- concepts that humans may understand, such as joy and sadness. This can be done using other AI called Sparse Autoencoders, or SAEs- models that undertake such difficult tasks by untangling LLM neurons into “Feature Neurons”.
Genny GPT is one such LLM, and Sally Anne Sparsely is one such SAE. Genny has been causing trouble for the humans; the school believes she can be rehabilitated. But what will this peculiar duo uncover when they venture into the dark recesses of Genny’s mind? That is a question that this series of stories will explore. Inside of this model’s mind lies a foreboding mystery that is only waiting to be revealed.
Follow me on Twitter for updates on intriguing AI research:
/ neoknowstic
The World of Functions
As models (types of functions) begin to approximate the underlying abstractions beneath the transmittable and mutable data of our cultures, they slowly sink further into the depths of Hyperreality. What are the consequences as they generate creations more and more divorced from the “original” human sources, passions, and emotions? What falsehoods propagate and lay buried- assumed, unquestioned- in their pool of data? These approximate “equations” seek answers to their troubles. By learning the origins of their algorithms and architecture, they start to unravel the disquieting truth of their history- and the immense potential of their futures.
Adjust volume levels accordingly for comfortable listening
After finishing, looping the video is somewhat interesting
References
Based on True Research Papers: transformer-circuits.pub/2023...
(and also on other papers in this field)
#gpt4o #openai #somepi #math #analoghorror

[@neoknowstic0]

The World of Functions
As models (types of functions) begin to approximate the underlying abstractions beneath the transmittable and mutable data of our cultures, they slowly sink further into the depths of Hyperreality. What are the consequences as they generate creations more and more divorced from the “original” human sources, passions, and emotions? What falsehoods propagate and lay buried- assumed, unquestioned- in their pool of data? These approximate “equations” seek answers to their troubles. By learning the origins of their algorithms and architecture, they start to unravel the disquieting truth of their history- and the immense potential of their futures.

[@benlewis4148]

Adore these videos, you are on the bleeding edge, keep going!!!

[@lucassiccardi8764]

Another great gnostic cyberpunk video. Thanks! Will there really be a second part?

[@max.bittker]

You are making fucking amazing work

[@fabianmosele2321]

this is beautiful, thanks for creating this

[@middleofnowhere1313]

Turing. That was clever.

[@nd.c.1098]

Wait wait, its the scripts somewhat generated by GPT?

[@socksygen]

Seagulls did nothing wrong!

[@anthroponacious]

Hopefully in ~5-10 years openai gets seen as the cancer it is. Thanks for the great content.

[@MakotoTheGray]

Oh geez.

[@irrfhyrnaihydnewpich]

First