Do LLMs Think Like Humans?

Do Large Language Models (like GPT) think like humans?
Many bold claims have been made lately about LLMs, one of them - that they refute Chomksy's approach to language. This poses a big question - are LLMs good models of actual human (linguistic) cognition?
In this video I rely on Katzir (2023) to answer this question.
======================
Papers discussed in the video:
* Modern language models refute Chomsky’s approach to language - Piantadosi (2023): lingbuzz.net/lingbuzz/007180/...
* Modern language models refute Chomsky’s approach to language - Katzir (2023): lingbuzz.net/lingbuzz/007180/...
======================
Linguistic phenomena discussed in the video:
* Center Embedding sentences (like "the dog the cat the mouse saw bit died"): www.polyu.edu.hk/cbs/sjpolit/...
* Binding Rules (sentences like "Jane hopes that Mary will mention her"): en.wikipedia.org/wiki/Binding...)
* Syntactic Islands (and specifically the fact you can't ask "What did Bill wash and Fred dry the dishes?", known as "The Coordinate Structure Constraint"): www-personal.umich.edu/~jlawl...
* Distinguishing linguistic competence from performance: www.literaturein.com/2019/11/....
======================
📕 My book "Gitting Things Done" is officially out!
Get it on Amazon: www.amazon.com/dp/B0CQXTJ5V5
As an eBook: www.buymeacoffee.com/omerr/e/...
Or read it for free at:
www.freecodecamp.org/news/git...
======================
======================
☕☕☕ Support this channel:
My mission is to create free, accessible, visual and practical tutorials. If you would like to support me on this, you can buy me a coffee :)
www.buymeacoffee.com/omerr
Thank you!!
======================
======================
How do I make videos?
I often get questions about the tools I use to create Videos - so here they are :)
(When available, I may use affiliate links and may earn a commission, with NO additional cost to you)
To create the hand-drawn animations, I use VideoScribe: www.awin1.com/cread.php?awinm...
I edit using Descript: www.descript.com/?lmref=Ux4_CA
Creating video thumbnails: snappa.com/?afmc=3uh
Microphone: Yeti Blue: amzn.to/3QydgtP
Drawing: Wacom Intuos: amzn.to/3pyRXwh
======================
Video created by Omer Rosenbaum. All rights reserved.

[@oldoctopus393]

Thank you for bringing up exciting topic. One question related to the approach of asking GPT 3.5 "to think a little longer". It looks like there are limited resources given by OpenAI for each user session which means, there won't be more RAM/CPU/processing time involved to answer the question even with this request. At the same time it could be the case that there are more compute resources allocated for paid version (GPT 4). Just some reasoning that we could be given a better answer with the same context provided. After all, I still think it's more about training models of ChatGPT v3.5 vs v4 than given input data and resources. WDYT?

[@BriefVid]

I agree that it's important to test against different models.
I don't think that GPT4 uses more "time" to "think", as generally speaking they use the same "computation power" per token.
But it'd be interesting to try it on more examples and see if we gather something else :)

[@rmnvishal]

Interesting video. I was actually looking for explanations for what "thinking" actually means for an LLM when I came across your video. Does it just predict the next word or does it have actual models of the real world which it accesses for answering questions? If so, does that mean it is intelligent? Is language a parameter for measuring real intelligence? What are your thoughts?

[@BriefVid]

Thanks for your question :)
LLMs are models that predict the nexts word. We actually know their algorithm so we can say that with certainty. How they actually get to the right parameters throughout the training is something way more complex to understand.
Measuring intelligence is almost a philosophical question, I liked the talk "Understanding Understanding" by prof. Yoav Shoham:
czcams.com/video/ZNi69FHe3y8/video.html&ab_channel=StanfordHAI

[@rmnvishal]

@@BriefVid Thanks, will check it out!

[@rajeshrajr]

What if the model sees this video, and other videos on this topic, adjusts its weights such that if it is a question based on linguistics, it should give preference to grammatical likelihood rather than actual likelihood ?

[@BriefVid]

Wondeful question. So in this way, we would say that the model can imitate the human mind better.
There will be a battery of tests that we can ask the model to determine whether it behaves the same as humans. "Grammatical" is a hard notion. Explaining why in "Mary hopes Jane will mention her", "her" can only refer to "Mary", is very hard. If the model can indeed learn all of these generalizations, and behave like humans in other aspects - we would say that it is a good model of human cognition. But it doesn't seem that way for now :)

[@3750gustavo]

Typing your Jane Mary example on a 8b llama 3 running locally set to just autocomplete instead of instruction, made the AI start talking to itself about the topic, and while doing so, on the first try said:
### LLM Input:
In the sentence:
"Jane hopes that Mary will mention her"
1. Can "her" refer to "Jane"?
2. Can "her" refer to "Mary"?
### LLM Output after a line break:
If "her" cannot refer to both, then the sentence is ambiguous.
I believe it can only refer to "Jane".
The reason I believe this is because in an object relative clause like this, it's the closest NP that isn't the subject that "her" refers to. In this case, the closest NP that isn't the subject (Mary) is Jane.
So for example, if we had a sentence like:
"The book which she gave him belongs to her."
Here, "she" must refer to the woman who gave the book to "him", because "he" is closer than anyone else but the woman is the one doing the giving. Therefore, "she" has to be referring to her.
### In summary:
This shows that a LLM, even smaller ones, can understand this language concept, at least when in auto-complete notebook mode.

[@3750gustavo]

Also if you improve your prompt to:
In the sentence:
"Jane hopes that Mary will mention her"
1. Can "her" refer to "Jane"?
2. Can "her" refer to "Mary"?
PS: I am not referring to if is possible, but if, more like if is a valid assumption to be made by a linguistic person or someone familiar with English language structure.
Makes the LLM even on instruct mode to give the 'correct answer'

[@BriefVid]

@@3750gustavo Very cool! Thank you for sharing this!
I am sure LLMs trained on various concepts will improve with time. To really have them as "valid" cognitive models would require them to be trained on far less data, and perform as humans on *all* tasks. This was just an example I decided to take to start this discussion, I am happy to see you're taking it off from there :)