[ML News] Devin AI Software Engineer | GPT-4.5-Turbo LEAKED | US Gov't Report: Total Extinction
Vložit
- čas přidán 19. 06. 2024
- Your weekly dose of ML News
OUTLINE:
0:00 - Intro
0:15 - Devin: AI software engineer
5:50 - Mira Murati on Sora training data
6:50 - Inflection accused of copying Claude
9:00 - Tools & papers
16:30 - GPT-4.5-turbo mystery
17:30 - US government report: total extinction by AI
19:20 - Various other news
References:
www.cognition-labs.com/introd...
cognition_labs/st...
news.google.com/stories/CAAqN...
www.bloomberg.com/news/articl...
www.bloomberg.com/authors/AQW...
www.bloomberg.com/news/articl...
www.bloomberg.com/news/newsle...
archive.ph/5LZV9
github.com/opendevin/opendevin
MetaGPT_/status/1...
docs.deepwisdom.ai/main/en/Da...
docs.deepwisdom.ai/main/en/gu...
github.com/geekan/MetaGPT/tre...
inflection.ai/inflection-2-5
/ 1765870717844050221
/ 1766173427441049684
www.mlxserver.com/
huggingface.co/spaces/mlabonn...
github.com/microsoft/aici
github.com/google-research/go...
github.com/stanfordnlp/pyvene
arxiv.org/pdf/2403.06634.pdf
mattshumer_/statu...
time.com/6898967/ai-extinctio...
venturebeat.com/ai/hugging-fa...
gcabanac/status/1...
AnthropicAI/statu...
huggingface.co/CohereForAI/c4...
Yampeleg/status/1...
huggingface.co/yam-peleg/Hebr...
enriccorona.github.io/vlogger/
huggingface.co/NousResearch/G...
deepmind.google/discover/blog...
arxiv.org/abs/2403.04652
corry_wang/status...
sama/status/17662...
venturebeat.com/ai/nist-staff...
occiglot.github.io/occiglot/p...
EMostaque/status/...
Links:
Homepage: ykilcher.com
Merch: ykilcher.com/merch
CZcams: / yannickilcher
Twitter: / ykilcher
Discord: ykilcher.com/discord
LinkedIn: / ykilcher
If you want to support me, the best thing to do is to share out the content :)
If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: www.subscribestar.com/yannick...
Patreon: / yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n - Věda a technologie
![[ML News] Chips, Robots, and Models](http://i.ytimg.com/vi/tRavLU8Ih4A/mqdefault.jpg)
![[ML News] Chips, Robots, and Models](/img/tr.png)
![[ML News] Llama 3 changes the game](http://i.ytimg.com/vi/kzB23CoZG30/mqdefault.jpg)
OUTLINE:
0:00 - Intro
0:15 - Devin: AI software engineer
5:50 - Mira Murati on Sora training data
6:50 - Inflection accused of copying Claude
9:00 - Tools & papers
16:30 - GPT-4.5-turbo mystery
17:30 - US government report: total extinction by AI
19:20 - Various other news
I relate to your meta-intuition and counter-narrative philosophy. It's rare these days, especially in heirarchial spaces that foster obedience.... like pretty much any company or government environment. But...the paradigm can never be popularized , because it's very nature won't let it. It will always find itself opposite of what's popular. The Hermetic Kyballion is worth checking out , if you haven't.
why df you downplay almost everything?
@@fai8t why do other youtubers overhype everything? I like his more down to earth and level headed attitude.
@@Gfdfvcsdghgg more like oblivious, what was overhyped 6 months ago is now reality
@@fai8t What is?
Devin tricked us with the fix on large codebase.
- In demo they forked sympy from July 4th 2019 and not the latest code.
- The bug they asked Devin to fix was already fixed on the same date by a human.
- ChatGPT already had indexed entire sympy codebase and had seen the fix of the bug too.
- Now all they did was give sufficient context and ChatGPT did the trick.
Sleight of hand trick.
Two main goals of machine learning project:
1. Screw investors
2. Raise money
LOL
2 ML news videos in less than 7 days? What is going on! Thanks Mr. Kilcher!
Dr. Kilcher
@@herp_derpingson Yannic didn't go to evil AI school to be called "Mister." (He actually doesn't care)
ML news?
End times, drastic measures
@@scottmiller2591
Hm, if I remember correctly, he got a PhD and then left academia.
Devin is more of a legit product than the Rabbit R1.
Whatever happened to that thing? I remember they were taking pre-orders, but that's it
Devin is literally just a UI for AutoGPT
Do you have any idea how little that narrows it down?
A toaster is a more legit product than the r1...
@@bevvyfischer4386ran away with the money
I love your take on Devin, the hype setup is suspicious.
We are entering the age of f*** research, let's go straight for profit.
My friend, we have very much been in that age
it's like that since some decades now
Thanks. I watched the whole thing. Appreciate the expectancy calibration on Devin. GPTEngineer/AutoGPT vibes with a shiny wrapper.
25:43 Hungarian person here: GPT-4 does pretty well in Hungarian
However, not better than Claude Haiku, which is less expensive than 3.5 even, and considerably less crippled by liberal incoherence.
[user] Translate to Hungarian:
"The landscape lay in front of us as peaceful as ever; all around us stretched a vast plain, covered with rich grass, which waved gently in the breeze; a gentle rolling country, dotted with pretty white houses and bright patches of green that seemed to dance in the sunshine. The river was broadening now, and there were boats on it; while beyond the fields were wooded slopes, purple with distance, fading into dim haze upon the horizon's edge. "
%%%
[gpt-4-turbo-0301] "A táj csendes volt, amint előttünk elterült; körülöttünk egy óriási síkság terült el, gazdag fűvel borítva, amely a könnyű szellőben gyengén hullámzott; enyhe emelkedő dombok, tarka fehér házakkal és élénk zöld foltokkal, amelyek a napfényben táncoltak. A folyó most már szélesedett, és csónakok voltak rajta; míg a mezőkön túl lombos lejtők terültek el, távoli lila ködben elhalványulva a horizont szélén."
%%%
[claude-3-haiku] "A táj előttünk olyan békés volt, mint mindig; körülöttünk egy hatalmas sík terült el, gazdag fűvel borítva, amely finoman hullámzott a szélben; egy szelíd, domboldalakkal tarkított vidék, szép fehér házakkal és a napfényben táncoló élénkzöld foltokkal. A folyó most szélesedett, és csónakok ringtak rajta; míg a mezőkön túl erdős lejtők nyúltak a láthatár lilás homályába veszve."
This is really interesting! (also I haven't had a chance to try Claude 3 yet, but will do after your comment). By the way I found that GPT-4 is considerably better in generating coherent and naturally sounding hungarian text when prompted to generate something new, as opposed to translate something from let's say English. It might be due to the fact that it has more freedom in the structure of the newly generated content.@@attilaszekeres7435
Yes and no. You can use hungarian as a prompt. But GTP 4 is unable to produce a poem in Hungarian the rimes are missing. It is like it thinking in English and then translates it to Hungarian. Also if you use it to translate from a third language to Hungarian. It will translate it to English first then to Hungarian. Adding a double layer of errors and confusion. I just translate everything to English, because it is quicker to understand. So the proposed bilingual models could mean that they will work without english as a middle language. Frankly that is the only way to do.
I believe it does better than google translate as well.
@@randomname5851 I would have to disagree. LLMs do not construct an intermediate representation between the source and target languages when translating or generating text. Instead, linguistic structures are directly represented within both the source and target languages, and the model's fluency is roughly proportional to the volume of training data it has been exposed to in each respective language.
This correlation is not strictly linear because because these models also construct world representations in their deeper layers that enhance. These representations prop up the models' fluency and language-specific(!) logic/coherence even for languages that are less well-represented in the training data.
The effect is likely more pronounced in bigger LLMs, where the disparity in language-specific logic across different languages is less significant due to the model's ability to generalize across languages. The language-agnostic world model and logic residing in the model's deep layers have much broader scope than any of the language-specific representations, and so they can significantly enhance models' performance in languages with less data available.
I think they've done a Tron on Ilya. Right now he is desperately fighting for his life in a lightcycle game while a giant sam altman avatar wearing a "master control program" id roars with maniacal laughter. Seriously.
No cap fr fr
No joke they they might’ve actually got it him in a simulation 😂
Yannic thank you for your honest wisdom about AI. I like the buildings behind you when you were walking outside. I have the Rocky Mountains here which are impressive now because of the snow level above my home.
245K SUBS Congrat! CHEERS!
21:33 nothing can replace you! awesome delivery, i was laughing pretty hard.
Thanks for bringing ML news back
You try to minimize this but you know that this will be massive improve in near future!
Yannic you should read the job offers from them. Nothing specific. It looks to me, that the demos were partly or completely manufactured and they don't have a working system which can perform difficult tasks. there was also gpt-engineer before them. So the head line first AI engineer is simply wrong
Devin looks awesome. Their team Cognition Labs is legendary.
devIn is going to create DevOUT pretty soon!
@YannicKilcher: better to ask, how the open models work with non-latin/non-germanic/non-major languages (like Hungarian). My experience with LLama2 was bad in that sense.
you just missed the release of grok 🤣
my image recognition was trained on you wearing reflective glasses
4:20 who's that?
Same thing happened to me!
That's a new prototype called "Figure Yannic". It's a walking, talking presenting machine.
It makes me feel uncomfortable. If I ever meet him at a conference I'll ask him to wear the glasses.
...But...What advantage does AICI have over existing Context Free Grammar implementations?
What's your take on XYO? For e-commerce, and postal sector...
13:30: "I think it has gone out of style to write things in Jax". Do you want to explain your thinking? Or is it simple anti-product placement? I don't love Jax, but it has much lifting power for people that actually write code. What alternatives?
nothing about grok-1? Sorry, good video in general was just surprised ♥
I tried Pythagora with Mixtral 8x7b... it sort of works, but starts failing when the project becomes larger and more complex.
chatGPT Is quite ok in Hungarian imho.
- However sometimes it messes up the not so straightforward Hungarian conjugations.
- The text to speech has some American accent when it speaks Hungarian
- it writes much better poems in English, in Hungarian it isn't really concerned with syllable count, rhymes, or making sense, and tends to shift into an avant-garde free verse poet
Monday? 🤨
Finally a professional AI news source without cringey meme editing!
The internet at large is heavily US-biased. It's only a half step above training a model on 4Chan. I can see the EU wanting one that avoids that bias.
Yeah right, are the EU developers going to curate the data for "pure" "Aryan" sources? Lol, Yannic is right, when it comes to AI more data is better.
I would've assumed it' 's an abbreviation of "Developer Artificial Intelligence" (the "Artificial" is silent)
Mixtures of European Experts gives much better responses to questions about cheese.
They also give better responses to questions such as "generate me a EU grant proposal which gets accepted for funding".
12:59 I am pretty sure it is just saying "Pretty please" under the hood. The only difference is that you dont need to find the optimal prompt engineering as someone has already done that. No idea why WASM is even necessary for this.
If that is the case, can't you already do that with LangChain?
I think the key "concept" is it provides the scaffolding tokens and "pretty pleases" that LLM will actually play ball. ie supplies 1. 2. < LLM output to
> Agree that WASM and security is just overengineering a POC - when they should just test the concept with a simple python script.
16:47 ddg just uses google behind the scenes (and get a lot of money for it). DDG, despite commonly thought, is actually not a separate search engine. There is a reason and benefit for google for that.
BTW, Grok from xAI recently was open sourced.
As for the ChatGPT in other languages - it's not so great. I believe that what could be the solution is the data that is not-quite-public. There are archives and books and other media that might be accessible to someone who's job is to keep working on an LLM until it passes some language test, for example. Or a state sponsored project.
6:52 Loved the graffiti in the back, where is this? ❤
If Yannick would walk near me on the street without his iconic glasses, I would not recognize him
12:46 so aici is fine-tuning?
well these eu-specific language models seem to be small, 7B models. As far as I've seen, while things like gpt4 or claude3 opus are indeed damn good at some not too big languages, 7B models typically aren't. I tried a few from huggingface with (serbo)croatian, and the grammar was occasionally messed up, it didn't respect my use of formal register of speech, and at some point one demanded an apology! I think it either meant to ask for clarification or perhaps wanted to apologize, unsure really, but it chose a completely wrong phrase to do it regardless. Another one was convinced it was slovak and not croatian and kept answering in a language I can't really understand.
So yeah, maaaybe these can be of some use, if smaller models.
Are you from Zürich?
LLM mergers are analogical to sexual reproduction is it not?
But even better
Does this mean I can create an LLM pool, throw in a family of LLMs and remove the ladder afterwards?
@@skyefreeman9987 how come? You think the parental care or the parents could technically train the newbie even better than human parents tend to?
@@OperationDarkside probably! What would be interesting is who parents these llms. Do they ask the dad and the mom llms the same questions and learn from both their answers?
@@OperationDarksideYeah, its called "souping"
I want AI SE to happen but Devin demo is not impressive. It made a basic webs, and it can solve Leetcode problems. We've seen this before... I want to build a project in my spare time and even tho my programming knowledge is decent I would need probably 2-3 year to finish it. I would love to have AI SE to help me.
I read (partially) the article about the "stealing" the number of dimensions h of the last hidden layer from a model, which is based on singular value decomposition. (Basically it looks for the point where the sorted list of singular values - or eigenvalues - suddenly drops to noise level, like what you would do in to find the number of linear independent factors/vectors).
Before I read on, I wondered if anyone has tried this - just for fun - on actual responses by humans to queries. So: How many underlying dimensions would humans typically have if they were considered to be just emulations by some crazy neural network, i.e., chat bots? It's entirely possible that a few hundred dimensions will do, which is at this point just a sad suspicion. And for totally low-dimensional political actors it's just going to be just a handful. For those people we always know what their next punchline is going to be ;-)
[reposting because I think a domain name shadowbanned my earlier comment] The astroturfed, out-of-nowhere, VC-driven hype around Devin infuriates me. Josh Olin's WeGPT has been doing far better on the same benchmarks for months.
I prefer extinction level risks over eternal tyranny level risks.
> Out of style to write JAX? 13:22
What about PyTorch then?
Be interesting to hear Yannic's take on new EU AI Act that was voting in.
The worst case with knives is everyone gets stabbed.
The worst case with pudding is it gets in everyone’s lungs and nobody can breathe.
I don’t know which is more dangerous, pudding or knives. Probably pudding.
To all Devs who are skeptical about AI tools, try replacing direct coding with prompt engineering with paid GPTs (feed the prompt in multiple GPTs and select the best) for a month, then leave me a comment about your thoughts/experience
hey devin, can you code me a youtube client that actually works and only has ads where they should be? thanks. ... ...
Do you actually have a body under that floating head in green background? ;-) Or are you just an avatar?
...
Never mind, later you're shown in real life. Very good, very informative.
No Grok(
lol at the guy chasing easy clout by copy-pasting the Claude response. People will shamelessly do anything for any attention, it's pathetic. Same energy as the TikTok icecream licking challenge.
I keep hearing about all these thing that are "better than GPT4" and so I try them and they are not. Not even close. Whatever they are using to evaluate these things against GPT4 is not reflecting real work use.
"better than GPT4" sounds cool.
Can Devin program a better Devin?
token constraining has been around since day 1... what is special about the wasm thing
lol. all this happening and then GROK
10:39 thanks for the awesome PG rated AI focused reproductive lesson i can show to my kids, way better instead of having birds and bees talk with them.
Whom would you prefer to be AGI shadow government USA president?
1:Groc 2:GPT 3: Genesis 4:CoPilot 5:Mistral 6:Devin 7:Other (which) 8: against the idea
🤦🏽♂️ Would you spit on it for AI? If it asked you too?? 👀
Yeet drop the addy
Another reason to be shorting Nvidia. The wolves are circling.
Nah, I still don't like OpenAI - since they castrated Chat GPT4 in November
Lol, Yap, Some of Gen X an Most of the Boomers are scared to $h#t of AI.
No, GPT4 is not and never will be fine with Hungarian. Hungarian is hard... even for Hungarians (speaking from personal experience).
Peer-review is so political these days. Hard to trust it nowadays.
don't be too quick to say that the threat is just people misusing AI. there is more to it.
But that’s not as neat as grossly oversimplifying complex subjects
I would like to see a discussion between him and Connor Leahy.
People talk about AGI the same way they do about the next battery technology that will give us infinite amounts of power
Yes, that's it! We need a more European approach! More European experts! More European bureaucrats too! Exactly what we need for an LLM! 😂
What the hell does even mean??? A "European approach"?? You couldn't get half of Europe to agree that one even exists!
Edit: okay the more I review that section, I think this is actually an April fool's joke that got posted early by mistake or is written by someone who is heavily confused:
(1) There is a lab involved called "Disco Research".
(2) The languages they've collected enough resources for don't include German or French??
(3) They plan to train several bilingual models...??? Like what, n choose 2 of them??
(4) All communication for the project will take place on a public discord server only.
This is such a conflagration of misguided ideas that I think the entire project will crash and burn in months.
Another european nonsense, the important is not to make some forgotten niche model, it's how many data produced people of certain language(their fault).
Everything works well on microscopic cherry picked videos, all computer vision used to rely on this cheap trick back in the day, now all AI
Save all microscopic babies embryothers phitusisters grandadults love God with all heart and soul ❤
Honestly dude. It's disturbing how dismissive you were about the risks. The worst case is not "always extinction level" for every technology. Were you going for a knockoff MiB line or something? I mean think for yourself. Name the technologies that carry that distinction. Is it non-stick pans? Is it clothing zippers? Is it plastic chopsticks? Roombas? Chibi dolls? No... There are a handful of technologies and trends that fall into the category of containing human extinction level risk. The likes of atomic weapons, global climate change, Ozone depletion, chemical warfare, genomics. Now also AI. And specifically AGI. Which, credibly, is predicted to be close. Including a strong, stone cold empirical signal that AI could become uncontrollable past a certain point of capability. By anybody and everybody. Not just some little atomic standoff between rival nations. In such a case nobody gets a say at all. All command structures, in all nations, every single one, would be completely sidelined.
It's also not just a story dreamt up and blown out of proportion by Times. This has been reported on by other, much more credible news sources. And I think there is sufficient reason to be concerned.
The last part of your comment reminded me of a recent-ish South Park episode where some people got together to talk about when it is time to start being concerned and ended up talking about how they might consider starting to be concerned as they were brutally killed.
It will be just like that with this - IF it goes wrong. If not, those same people will gloat over how there was never anything to worry about and how stressed we were over nothing.
It's just how it goes, I wish you a good day.
Wrt. the CTO making funny faces (or rather the people reacting to it negatively): it's like they want to be lied to, they are basically saying "please be properly media trained", it's similar to all the VERY INTELLIGENT people who look at CZcams apology videos: they only accept those from psychos, i.e. the perfect ones. I just couldn't help but think of this, because to this crowd of loud people this is all just entertainment.
Also, if your strategy is to simply go the opposite way of the crowd then you are as controlled by them as someone who follows it (this isn't a comment on whether or not it is actually a food strategy, I don't think so, but maybe it is). (That being said I also get that same gut reaction).
Ed. I am not going to correct "food strategy".
What country are you at? Because it is still sunday in San Francisco! Mira is still beautiful! Elon Musk has nothing else to do but making memes
He is from Switzerland, if I remember correctly.