How Developers might stop worrying about AI taking software jobs and Learn to Profit from LLMs
Vložit
- čas přidán 27. 06. 2024
- Right now, the software industry is kind of stuck, where no one wants to hire non-AI developers because the hype is claiming that all the jobs are about to get replaced by garbage like Devin.
But new evidence indicates that the pace of AI growth is slowing down, and that two years of GitHub CoPilot has been creating a 'Downward Pressure on Code Quality.'
The future is never certain, but it looks like there's a path for the next few years to be a new boom in software development, and it might start soon.
00:00 Intro
03:49 Ezra Klein's interview of Anthropic's CEO from April 12th
04:06 There are no Exponentials in the real world
05:10 Research showing LLMs are reaching the point of diminishing returns
06:28 Article on the stagnation of LLM growth
06:57 Stanford AI Index Report
07:18 Research showing that LLMs are running out of training data
07:28 Research on "Model Collapse"
08:13 Research showing AI reducing overall Code Quality
09:04 Quick Recap
09:27 Implications to Software Developers
10:56 Parallels to 2008/2009 and the App boom
11:44 How and when we might know
12:07 Wrap up
Papers and references from this video:
The complexity of the human mind
mindmatters.ai/2022/03/yes-th...
www.psychologytoday.com/us/bl...
Ezra Klein's interview of Anthropic's CEO
www.nytimes.com/2024/04/12/po...
3Blue1Brown Video on Logistics and Exponentials
• Exponential growth and...
Research showing LLMs are reaching the point of diminishing returns
garymarcus.substack.com/p/evi...
arxiv.org/pdf/2403.05812
paperswithcode.com/sota/multi...
Research showing that Data Availability is likely the bottleneck
en.wikipedia.org/wiki/Chinchi...)
www.lesswrong.com/posts/6Fpvc...
www.alignmentforum.org/posts/...
The 2024 Stanford AI Index Report with section on running out of data
aiindex.stanford.edu/report/
Research showing we're running out of LLM Training Data
epochai.org/blog/will-we-run-...
Research showing "Model Collapse" when training data contains LLM output
arxiv.org/pdf/2305.17493
Research showing AI Code Generation reducing code quality
visualstudiomagazine.com/arti...
visualstudiomagazine.com/Arti...
www.gitclear.com/coding_on_co...
Release Hype about GPT-5
tech.co/news/gpt-5-preview-re... - Věda a technologie
Ironically LLMs are taking software jobs not directly by writing code but rather by costing so much that there’s not much money left for other projects
There's definitely an aspect of that. There's only so much R&D investment money to go around, and LLMs are shoveling a lot of it into a furnace at the moment.
I work for a startup whose entire business model is centered around a tool built to analyze data using LLMs (GPT-4 to be exact). It costs less than $100/mo to run and it's extremely profitable. What BS are you spouting?
I'm talking about how the "[private] funding for generative AI surged, nearly octupling from 2022 to reach $25.2 billion [in 2023]" and $1.8Billion in 2023 US Federal Government AI spending (source aiindex.stanford.edu/report/). The losses include OpenAI who lost $540 million last year (source www.theinformation.com/articles/openais-losses-doubled-to-540-million-as-it-developed-chatgpt ), Stability AI, who is burning $8Million/month and making only a fraction of that in revenue (source fortune.com/2023/11/29/stability-ai-sale-intel-ceo-resign/), the (source aiindex.stanford.edu/report/) plus a ton of other companies that don't report losses or don't break out their AI divisions, like Anthropic, Meta's AI division, etc.
I have no doubt there are companies that are making some profit from LLMs, but I seriously doubt it's anywhere close to the $27 billion being spent on it (just in the US).
haha true but the cost will get (eventually) lower that`s a given
@@lattehour I certainly hope so!!
Your content is from a time of CZcams past when people actually cared about delivering quality, original thoughts rather than algorithm optimized drivel. So glad I stumbled across this channel
I agree ...am in my final year in uni and his perspectives has been really encouraging... it's so refreshing to listen to well reasoned arguments
@michaelcoppinger786 I guess it makes sense that my content feels like it's from an earlier time. In case you hadn't noticed - so am I :-)
LOL
Thanks for the compliment. I appreciate it.
Well said
@@InternetOfBugs love your energy though! :-) i've noticed some people also from "an earlier time" aren't as truly analytical as you are about the current state of software. on the contrary, they seem to become stagnant on what they already know and just don't like change (which doesn't make any sense with how software is taught nowadays)
versus you, who clearly take the time to "go the extra thought" and come to a down-to-earth conclusion without coming off as arrogant or "wiseass" lol :p
I found this channel through Jonathan Blow and Casey Muratori but can't remember how exactly.
I've got a degree in biomedical engineering and have done computational neuroscience before. The number of clueless programmers out there talking about how AI is the same as the brain almost makes me regret switching to general software as a career. Thank you for mentioning this.
As one of those clueless programmers (at least when it comes to medical stuff - although I'm arguably clueless about far, far more), I appreciate your expert opinion.
You spread ignorance, I studied neuroscience and computer science in university and have experience with deep learning, and the systems are alike logically, the only difference is that the biological neurons are continuous, and LLMs are clock gated, but the rate is so high that it might've been continuous for that matter. Also, LLMs are NOT frozen if an online (not in terms of the internet, in terms of DL) approach is applied, or if reinforcement (not RLHF) is applied. In general, LLMs are logically almost the same, with some advantages per synapse, but with 10X less "synapses" compared to a healthy young adult. What's holding back the number of synapses is energy budget, and that's about to change
@seventyfive7597 The structure of the neurons might be effectively the same, but the human brain is not just a very large collection of neurons connected at random. The overall systems are vastly different.
Feel free to take it up with Simon Prince. His Book "Understanding Deep Learning" contradicts you. (Book: mitpress.mit.edu/9780262048644/understanding-deep-learning/)
You might want to read it.
I can't link to the relevant section of the book, but here's a condensed explanation from an interview with him on the "Machine Learning Street Talk" podcast:
czcams.com/video/sJXn4Cl4oww/video.html
Also, feel free to argue with Meta's Turing Award winning Chief A.I. Scientist:
“The brain of a house cat has about...the equivalent of the number of parameters in an LLM... So maybe we are at the size of a cat. But why aren’t those systems as smart as a cat? ... A cat can remember, can understand the physical world, can plan complex actions, can do some level of reasoning-actually much better than the biggest LLMs. That tells you we are missing something conceptually big to get machines to be as intelligent as animals and humans.”
observer.com/2024/02/metas-a-i-chief-yann-lecun-explains-why-a-house-cat-is-smarter-than-the-best-a-i/
@@seventyfive7597 I'm glad I did the biomedical stuff before the CS stuff. Helps me have a healthy skepticism when people claim systems that have orders of magnitude more energy requirements than the brain are "logically equivalent". Clearly we're missing something or we could run LLMs on the energy density of bread like our brains do. I'll trust my meat thinker over these tools for now.
@@InternetOfBugsJust going by intuition here but if I were to guess I would say that the human (as well as other biological brains, possibly including octopuses 😉) has many more auxiliary processes that the LLMs don’t have. And I’m not even talking about the whole internal synergies’ landscape among all the specialised centres of the brain. Even mere pruning (and dreaming, no less) is something that is only being researched and not yet applied as a standard process of maintaining LLMs.
My recent candidate for the next hype are KANs (Kolmogorov-Arnold Networks) that apply serious mathematical analysis to transforming the conceptual space of neural networks. Essentially, transformer functions on each perceptron instead of weights and on/off switch functions. The first experiment results look promising and if that turns out to be a breakthrough, my bet would be that we might have some real next-gen creation in our hands. Something that would be beyond the discussion of „human brain has vastly more neurons”… - we would have better neurons that biological brains never evolved to have. That would actually be scary…
I dont understand why were using LLMs for Creative and Programming tasks instead of Administrative work. I feel like that is where it actually makes sense.
Yeah we don't need video generators that cost $1m for 10 seconds, we need an excel AI assistant that does the spreadsheets & the schedule management.
LLMs can make mistakes. They are unreliable. Totally not recommended to use in critical enterprises. Or administrative tools.
They’re being used for everything. Don’t pigeon-hole LLMs. Contract writing, Legal interpretation, Website development, etc.
You will spill that drivel until a LLM powered "payroll clerk" decides this month's salary for you is $7.
I use it for administrative work. Copilot for Microsoft 365 is a crazy time saver. It's not perfect, but it's made my life so much easier.
This was a top-tier take.
All my corporate clients are currently working on using LLMs exactly like you outlined, using them like any other regular API within existing products.
None of them is using AI to generate code or planning to do so.
Corporations were never going to be the ones to use it like this - too risky. Take a smaller business with less to lose and a bigger focus on cost and you will find that code is being written with AI.
@@michaelnurse9089, with all the downsides that come with it.
Great points! I think what a lot of non-programmers don't understand is that actually writing code is a very minor part of a developer's job, with the majority of time spent on "communication" tasks. So even if LLMs could write pristine code, it's still not going to replace developers as only they know the context of what to ask to code, how ot verify it works, and how to implement it.
Communication isn’t really that crazy of a skill though, a LOT of people have communications skills. Coding is hard for many. To me it’s the largest barrier of entry to normal people creating applications and solving software problems.
@@TheExodusLostdisagreed. A lot of people *can* communicate. A much smaller amount can communicate *well*
@@HomeslicedVideos still a much higher number than decent coders.
@SeriousCat5000 Yep. I made a whole video on that here: czcams.com/video/7-f7rPdj6tI/video.html
@@TheExodusLost You are correct. Many organizations separate the systems analysis from the software development, assigning the analysis to personnel having experience with both the problem domain and software development and the programming assigned to a team of programmers who are given specific functional tasks to complete. In my experience, many developers should never be permitted to touch a keyboard, much less to write code.
Dude, your devin video made you my hero, sort of. Not because you debunked Devin, but because you broke down the entire process of approaching the problem and laying out the important points for the conversation that should follow. I love your videos and wish you the best. 🥰
The devin team is formed by gold medals coding and math winners in championships.
@@alonzoperez2470 SO? Whats ur point? Everyone knows that they are LGMs or GM on codeforces. Are u tryna say just coz they are good in competitive coding devin should be real?
@@newbieguy2509 it will most likely be the case. Perhaps. Devin won't straight up replace every coder out there. But it will definitely automate a lot of tasks in the tech industry and many people will definitely be replaced by this AI.
@@alonzoperez2470when it can't do the one they showcased in the demo? Anyways safe to say it will get better though
@@plutack look. The project was unveiled like 2 months ago. In my opinion it will take like at least 2 years or more to launch this project if they aren't only aiming "$" like I said the team is composed by geniuses in the math and coding field. I wouldn't take the project seriously if it wasn't composed by individuals who are experts in the video. But yeah just time holds the answer.
I just want to say how happy I am that you have become a content creator. You've got such a fantastic and well experienced insight into these kinds of topics. Please keep making content - I've been sharing your videos with a bunch of people over the past few weeks!
Thanks. I appreciate it.
I've been following you since your first video. Thank you for bringing much needed nuance to this incredibly difficult conversation.
As an immigrant with mediocre English language skills (verbal), I often find it difficult to communicate these exact notions about AI to my managers.
It seems like everybody is riding high on the AI hype train. I've already implemented some amazing multi-modal search capabilities in our application using CLIP. AI is making a difference in my life already. Imagine, having to deal with an ElasticSearch cluster!
I'm trying to push my company to build more grounded LLM based solutions like these. I wish I was well connected to be an "AI" engineer like you said. Judiciously implementing these amazing features in applications where there is a fit!
I immensely appreciate your clarity sr. you have taken out a creepling doubt out of my heart
I was surprised when I saw you only had 30k subscribers. Your videos are so rich and full of great insights. So refreshing to get a balanced view.
Your sound became really good. I think you produced here the most (good and solid without hype) review that is out there!
Amazing channel, highly appreciated content. Keep up the great no-nonsense approach! No click-baiting, just some interesting food for thought. Big-up.
Although I agree with your general thesis, we must be prepared for significantly better LLMs too. Exponential growth forever is impossible, but how has Moore's law held even as the Dennard scaling that originally drove it died? Stacking sigmoids. They get more expensive to stack, but things can look exponential for surprisingly long. Some potential next sigmoids for LLMs: Training far past chinchilla optimality like Llama 3 (though this increases data hunger) and enabling tree-reasoning with an "evaluator" model like what people suspect the Q* thing is (though compute expensive / demands inference efficient models). There are interesting ideas to address data hunger as well, though I don't want to belabor the point.
It will certainly be interesting, and I'm looking forward to seeing what happens.
And if it turns out I'm wrong, and we do get AGI in the next few years, that will be FASCINATING, although I have no idea what the societal or economic implications of that might be. I'm a functionalist, and I do believe humanity will get there, it just doesn't look to me, knowing what I know now, that it will happen in what's left of my lifetime.
@@InternetOfBugs AGI is a red herring. We don't need AGI to radically transform society. What we have now is sufficient, or will be after one or two years at the most. But yeah, most of the denial stems from the economic uncertainty of an AI driver future.
Q learning, Markov decision models and such have been here for quite some time. They have been used successfully in finance and other fields. Just like LLM's people hear about them because they are new to the field and are under the impressions that OpenAi is revolutionising in ajy way. I used to deploy LLM back in 2021 and google has been using them for at least 5 years if not more. The original paper states exactly what the shortcomings are, shortcomings that have not been surpassed today. People that have been able to optimese these models, or smaller models in the future are the ones that are going to make all the bucks. Large models like ChatGPT are not feasible from an economic perspective, but having only half of ChatGPT be able to run locally will be a huge thing. I currently deploy on 2 X 4090 a 30Billion parameter model. This is tiny compared to any LLM available for general use. If someone finds a way to make it computationally efficient, just like gradient descent did for training, than that will be huge. Until then, Devin is a Reinforcement learning algo, (Q Learning if you want - this comes from gaming and it's a pathfinding algo), and as you can see that one has real limitation also. Just because someone has large computer to better iterate and get better models out there like OpenAi has, does not mean they have come out with something new or revolutionary. They did come with stable products, that are really difficult to build
Moores law isn't exponential lol
Obviously, LLMs are limited by hardware. Does linear increase in compute yield exponential results in LLMs? I'm not sure.
I'm curious as to what the next sigmoid for hardware will look like - I suspect this will take many more years of research, perhaps a decade or more before any meaningful progress is made. On top of that, costs will likely be too high for a market solution for a period of time as well. At least that is how all the previous sigmoids behaved, why would it be any different now?
I always redirect or share link of your videos whenever my friends are demotivated because of the AI hype. Thanks for the researched quality content and your efforts.
"Converges on boring" - yes, I've noticed that in image generation apps. It's all the same colors and faces - it's predictable and boring.
Have you seen any custom models for Stable Diffusion? Tons of variety due to the variety of training data and no corporate DEI finetuning like that at Google creating black nazis.
I haven't experienced that much with Stable Diffusion. Each model you use with it has a different look and feel to it, and you can always train on different styles or characters to get something different.
He was specific on LLM and you are talking about image generation. We create tons of images (or using GAN) and experienced people train the models which are super good at creating specific outputs (like varying human faces).
There is some weird obsession of people training it on Asian faces which makes all the results look like some kpop person, that I agree with.
@@truck.-kun. convergence on atheistic. lol
@@nigel-uno bOrInG iS wHeN bLaCk PeOpLe
You are on point, I actually run into this issue of repeated answers with different LLM models. This AI hype will go away within 2024.
Love your videos
The hype won't die anytime soon.
We had a whole hype cycle over NFTs which has no uses
I don't think it will go away, but it will absorb less capital. Unless results shown are outstanding.
I agree. I can already see the hype dying down as we speak
Woah man. Your channel just popped this morning and I have to say I am really enjoying your content, it comes as a breathe of fresh air in the current context.
Thank you and I hope all is going well for you.
Fascinating, I really like this presentation style, quick with data points. Thanks!
Glad you liked it!
This video earned a subscription from me. Thank you for mentioning the "Model Collapse", because I've been talking to people about this very same thing, but I was calling it "Generation Loss" coming from the audio production world where something is recorded to tape over and over and the artifacts and quality degradations start to compound. I'm glad to know there's an actual name for it!
This needed to be said. When one studies how LLMs work, it’s just statistics, calculus, linear algebra, and trigonometry (cosine, sin, etc) all hacked together. It’s extremely unlikely that this architecture can scale to a human brain. It’s a modern miracle that’s it’s scaled to its current capability. If it scales to human intelligence, then we will have much more to worry about than replacing jobs and programmers. We will need to reevaluate our complete understanding of what it means to be alive and conscious. Replacing jobs and the profit motive is an extremely small minded view of such an achievement.
@@e1000sn Exactly. Well said.
@@e1000snThat's almost as reductive as saying everything is made from atoms. Yes, and?
@@TheManinBlack9054 There is no way you can make such a claim because you have no way of proving it. Right now the only true intelligent things are alive. Perhaps it's a property of the biological mass, perhaps it's a property that comes with consciousness. But the main point is, so far it was never shown otherwise.
I don't care what OpenAI's or NVidia's CEO claim to boost their stock value.
This miracle and the question posed is exactly why its getting so much serious attention
@@TheManinBlack9054At best AI is just a reflection of human knowledge. This is why LLMs cannot solve math ( not just regurgitate a solution that it has seen, or follow a set of instructions ) or understand physics or basic contextual concepts .
Man, your videos are so high quality and insightful.
I wonder why you don’t have at least 100k+ subs yet, oh wait….
This channel is pure gold. Thank you sir, for your videos!
It took me years to find someone that is actually honest about AI. This guy speaks truth, but mainstrem media continues to pump "AI", and the only ones profiting are large corporations.
Great to see that someone that can still think clearly and present ideas in a clear and lucid way.
I appreciate this video so much, it's clear you've put a lot of thought and research into this and it's explained in such a digestible and understandable way. Thank you!
Changed from my TV's CZcams to my phone's just to comment on this video, which I rarely do.
You brought up a few really good points and it was refreshing to see someone taking a more critical stance about the evolution of AI. Usually most people (me included) tend to get carried away too easily by the hype train and promises these AI companies make.
Thanks for challenging these popular opinions and for this thought provoking video!
Your content is just so nice. I hate the current social media tech culture that is just overhyping anything. It still impresses me how you can influence people in their opinion by marketing (like devin and basically every other AI startup).
Thanks.
Overhype and fear-mongering for views
thanks for so eloquently explaining what i’ve been debating my co workers about for months
Glad your video popped up in my feed, you've just earned a new subscriber, great video, nice dissection
This is the first channel I have been grateful for. Just good content, digestible and well planned.
100% agree. I’m a SWE getting a Masters in InfoSys with a concentration in AI, and this is exactly what I’ve been saying for the past semester, AI generated content is ironically poisoning its own well of already limited data. I can’t wait for people to get over the hype so that we can focus on using the existing to tools to make some useful and awesome new and innovative tech
Great insight, subbed to support this kind of researched and well supported content. Keep putting this stuff out!
Gotta say these videos are a breath of fresh air. The VC centric hype over the past 2 years has felt suffocating & it feels great hearing these counterpoints against it. These videos have been really enjoyable to watch.
The crazy part is that this hype cycle also encourages other VCs, founders, and managers to jump on the hype train.
My marketing manager keeps encouraging us to use AI for everything.
Lol. I want to jump off a building whenever I find a technical blog post that I think will give me the answer I want to start with: " in the world of web development."
I see a lot of companies abusing AI to automatically generate technical blog posts or documents that improve their websites' SEO but are garbled when it comes to value and content.
And god forbid someone uses the solution in their code base.
Your videos would be excellent reading but thank god they are videos.
Amazing work! 👍
Glad to hear someone sensible after a long time. In this time of AI hype, it is easy to lose hope. Great video!
Thanks
you got my like and subscribe - probably the best documented explanation for LLMs.
I mean.... I really need to thank you for all effort you put together to create this video... It help me digest a lot of lies big corps want us to believe and give me such a relif. Really appreciate man! Keep it up!
I appreciate you saying so. Thanks for subscribing.
There is an AI bubble coming sometime in next 3 years. If past trends are anything to go by, we will then have less than a decade before actually useful and cost efficient ai assistants will become commonplace.
Make sure to properly rebalance your investment portfolios to profit from the burst.
Anyway, the management quest to replace expensive developers and testers with cheap ai is far from over and will affect hiring decisions even as the bubble will be bursting, but I can't wait for future youtube videos with collapse stories from Cold Fussion and similar channels.
If you are still planning to be in programming for the next ten years, you only have to survive the next 3, so it may be a good moment to pause job changes and start learning something interesting. After that, we are likely to have a ton of bad code to fix or better yet, rebuild from the scratch. I intend to ask double for that job :)
I offer you my Respect. You seem like a genuine intellectual. Hope you get the chance to be heard by many.
I agree 100% with you. Great channel much needed on CZcams
Great video! I'm so happy someone in the youtube space actually drops citations when discussing the topic at hand. The extremely ironic part of data quality decreasing is that in a capitalist system information quality isn't the main focus of how to operate a business but making money is... I mean the channel is called internet is full of bugs which which I feel includes the integrity of data quality published on it.
Yep. The first video I published on this channel was about "information quality isn't the main focus of how to operate a business but making money is" : czcams.com/video/hKqqU1J-WXk/video.html
@@InternetOfBugs Absolutely correct. The most efficient organization (MEO) collects and stores only the minimum amount of information required to support revenue generation. Unfortunately, as a business grows government regulations create requirements to collect and store data which runs counter to efficient management and operations.
More often than not, this extraneous information is not managed in accordance with a records retention policy that results in any data collected for no other purpose than to demonstrate regulatory compliance being purged at the earliest opportunity. Maintaining these records has a cost that increases prices, decreases competitiveness, and negatively impacts profits.
It was noticed by some that the word "delve" was rising in usage. One theory is that it has disproportionately higher probability of being output by OpenAI's LLMs, and people are either adopting this trait, or are just dumping ChatGPT output onto blogs, forums, social media without doing any editing of the content. I've found another possible tell of GPT pollution. Search for the exact term "46,449 bananas" and you'll find all manner of articles, resumes, company brochures and product descriptions that only seem to have one thing in common -- a fun fact about the height of Mount Everest randomly inserted into them. The Internet is not just full of bugs; it is rapidly filling up with bananas! 😂
usually never comment on videos but this video was so fucking incredible. Well researched, coherent and hopeful. Thank you.
This channel and your videos are so spot on and informative. Many thanks!
Keep up to good content. Thanks for not chasing CZcams hype. Your content is actually worth something.
Excellent article! It’s crucial for entrepreneurs to focus on tangible applications of existing technologies to create value today, rather than chasing elusive future capabilities. 🚀
Good video. I'm not worried about the field in general, there will be software engineering roles and I don't think AI will replace programmers. What worries me is that nobody knows where this is going and what will be the steps needed for us to adapt (or retire). Because, like it or not, this is going to affect the way we work. Uncharted territory for us to explore. And we will learn.
I'm an independent contractor (mostly Javascript and Python). Let me tell you that until September last year, I didn't have to look for jobs, jobs came to me. The market is changing and there's no clear path to follow. But overall, I'm optimist.
You are absolutely right - too much hype, I like that you are trying to really dig and find opportunities to take advantage of this
Thank you so much for this video. You came with an original idea and opinion very well based. It’s so good to watch someone being realistic and not blindly jumping into the hype wagon. I will keep a close eye on gpt 5 and what you mention about what we can expect of it.
Great video. I think the focus is always on hype, but there are lots of companies that are already focused on applying LLMs to real world problems.
It feels to me anecdotally (although I don't have any good data) that the amount of effort being done by companies working on applying LLMs is a tiny. tiny fraction of all the effort being spent on LLMs/"AI". I hope that ratio shifts (or has already shifted, and I haven't seen it, yet).
A lot of it seems like tech companies are throwing spaghetti at the wall hoping something will stick. A few months ago Amazon had a LLM ChatGPT hooked up to their review search bar -- it wasn't obvious they did this until people started prompting it to write limericks and Python scripts. What was that all about?
This is fantastic. Well explained and very realistic.
The hype is very dangerous, i live how you logically broke through the hype with facts and papers. Absolutely fantastic you have my subscription
Excellent work, keep these in depth interesting and sober videos coming!
You're folding together a lot of good ideas/info here, thanks for the video!
amazing summary: clear, schematic, sensible. instantly subscribed
This day and age calls for caution splashed with a dash of optimism! Thanks for sharing your thoughts on this, it plagues me daily where we're going and how to leverage it. I also find your thoughts insightful, and it doesn't hurt that it resonates with my own experiences.
🤔🤯 🤤
Great video! I really liked the analogy with the developer environment in 2008 with mobile apps. Could you share some resources on how to start applying the current LLM models into existing internet products?
incredible quality, thank you for sharing your opinions
My concern with AI is the same concern I had with outsourcing jobs. New developers need to start somewhere and if you are a new (L1) can take years to get to an L10 (if ever). AI will replace skill levels L1-L4. Eventually when the good people leave, you will have a lack of skilled talent because they never got those low end jobs that might have gotten them from an L1 to an L5. A lot of other people will think the barrier for entry is too high and they'll get replaced by something new once they get those skills. Eventually there's no talent and they blame kids for not being interested in technology.
Happy to see this channel grows
Sora is getting too crazy. I can't tell if this video is ai generated or not /s
Great video as always!
Very well put. Insightful points
Love how you got to the heart of the matter. Value creation is all about problem solving. AI is just another tool in the box to be used for that purpose.
Just for the MMLU benchmark, it is impossible to keep growing linesrky because it’s a score out of 100 and we’re already at like 90
Your channel name is cool, but somehow I feel that "Hypeslayer" would be more precise description of what you do. Keep up the great work!
an example of that 'photo-copy' effect is how the UIs of the internet (especially mobile) have gotten so tremendously boring and mundane from years of optimizing on the same problem
It's also the tools. Apple has killed (starting with the iOS 7 "flat design", and doubling down with their new SwiftUI library) the interesting interfaces we used to have on the iPhone.
I could do a whole video on the decline of the mobile ecosystem, and my theory about the consequences of Apple's obsession with secrecy.
I should put that on the list.
Great video, this might also explain why these companies are pushing more on the generative AI for images and videos. There's more to do there than what we have with generative text.
That's a possibility. I haven't really been following the economics of Art/Image/Video generation. It's not my area of expertise. But there's a lot going on over there...
AI for images and videos is stolen intellectual property, and is the ultimate betrayal to the artists who made that content with their authentic talent.
@carultch Is that not also true of, say, book authors?
(Not trying to imply artists shouldn't be compensated, I think they absolutely should, but it seems to me - and I could be missing something - that lots of creators are being stolen from, not just artists).
@@InternetOfBugs Book authors too. All kinds of people with creative professions.
@@InternetOfBugs Another issue I find, is that many times, the original author is trying to clear up a misconception on the original webpage. As is expected, the author might start by restating the misconception in the opening paragraph. And then in the next few paragraphs, the author would clear it up with the actual substance of their work.
Guess what makes search result summaries: the misconception, stated as fact.
From my point of view, after a batch of processing data i would like to say, LLM get fill of entropy with data. LLM is just vector which calculate a perfect next vector based on " "avg" score "(just simplify). Thats means most interesting part will disappear because entropy of data and normal distribution. But there is think, LLM what is catch by level of entropy like work by z-score in 0, they will capable to replace a lot people in white shirts, what will significant make impact on industry and cat it. I'm sure people don't understand, but LLM generally speaking is
put hot water in one side, cold in other side, LLM will be line between two sides. But with LLM we will move into stage where is not anymore hot or cold water will be generate, and entropy will increase, in the end we get a situation where is LLM get a everything. I have more explanation about it, because i think about years and years. But i'm kind a dump person, so i just scare my 12 years in IT crumble to dust )
Your videos are absolutely spot on.
While you are looking in front of you, you don't see how llm taking other sides of generative content as images, sounds, science documents, architecture plans.
Such a great talk. Thanks for sharing.
Another great video, this is sort of my current thinking too that LLMs will help developers build things faster and automate aspects of the dev process but won't fully replace a developer any time soon, but who knows and it'll be interesting to see if GPT5 makes a dramatic leap forward. Isn't there a view that quantum computing techniques could result in another dramatic advancement in capability? (btw I liked the Hill Street Blues outro statement whether it was deliberate or not!)
The HSB reference is absolutely deliberate. When talking to early-career developers venturing into the scary world of their early dev careers, I often feel like a grizzled old, bald Sergeant hoping they don't take my advice too seriously (or out of context) and get themselves in trouble with it.
@@InternetOfBugs can you use the HSB intro for your next video intro - I'm sure your GenX following will appreciate it!
@@afai264 Hmmm. I wouldn't want to just drop that in to an unrelated video as a "memberberry." That would feel lazy to me, but let me see if I can figure out a way to work it into a connection with some topic in the a future video.
I watched this video and immediately subscribed. Great content.
Amazing delivery! You, sir, just got a subscriber.
Thanks for this video. I think it was made before GPT-4o so has that changed any of your opinions? I think that doesn't affect software engineers directly but does seem like it could have a big impact on other industries e.g customer service.
I don't pay much attention to employment in the customer service industry, so I don't feel comfortable speculating, but I wouldn't be surprised.
As for GPT-4o, I'm working on a video about that now
Thank you, this is the one of the best and honest videos!
On the point of models trained on LLMs output over and over again result in less variety in the dataset and convergence of outputs as you mentioned. Something, that has been seen to be happening with recommendation engines where over time, the recommendation based feed converges to a cluster of content not allowing for variety in output.
Subscribed. Your thoughts and reasons are so authentic.
Very interesting thank you for sharing your thoughts. I think companies using LLM capabilities already. The fact that new models are coming out doesn’t change anything. Solutions are design model flexible so they can upgrade to those new models in the future
Thank you for sharing a non biased view of AI development
The highlight from "AI Index report 2024" is crazy. This should be the N°1 takeaway from the report and should probably make headlines everywhere.
This is great. A relief. It's really freaking me out, how quickly things seem to be progressing. It's kind of nuts if you extrapolate, this machine that you talk to and it does cognitively what any person can do ... scary stuff. That's my entire livelihood. Let's hope you're right and there are limits to this growth. If not ... brave new world, at least for me ...
the industry always likes to add a layer on the stack. The agent layer sits on top between the UI and the "user" and will be LLM terrific.
Then all the layers below continue with under investment per usual as it bit rots
I so want to believe this isn’t just copium. We’ll have to see 🙏🏻
Also Llama 3 disproved the chinchilla paper. It was trained beyond what anyone thought was the reasonable cut off, yet it performs much better than the previous model from like 8 months before
Congrats! You're the 8th consecutive person in these comments to quote that headline while having no clue what it actually means.
What Llama-3B found was that adding more data tokens to a given amount of compute scales up log-linearly. The IMPORTANT finding from Chinchilla was not that you couldn't improve results from throwing more DATA at a given amount of COMPUTE . The finding from Chinchilla is that you can't improve results by throwing more *COMPUTE* at a given amount of *DATA*, so the amount of quality training data available is still the limiting factor.
These are very interesting thoughts and findings. I dont if they are accurate, only time will tell, but I have many misgivings and uncertainties about AI and these points do fit in very well. Btw, didnt know you had degrees in physics? May I ask what specific degrees you have and how you ended up in software?
Good lord! I feel like you've put into words what I've been thinking about the AI hype lately and even went further to explain what the common investor and general public are missing
Thanks, I appreciate that.
Very thought out video. Love it!
Great video. Love the content 👍
Great points, at same time I wonder if synthetic data can supply a new momentum...however it's sort of intuitive that synthetic data has its own limitations, basically by definition.
The best treatment I've seen for synthetic data is from arxiv.org/pdf/2404.05090 and it indicates that 20% Synthetic is mostly ok, 50% Synthetic still collapses, just less quickly.
Unless that paper turns out to have glaring errors, without a complete breakthrough, the best ratio we're going to get by adding synthetic data is not going to buy us much time.
Thanks for being smart and saying things that make sense. Seriously, there's too much content on this topic that's vapid. But I do think that it will only take a couple more years for another fundamental breakthrough. Btw I also studied physics
Subscribed. This is great content
the only thing that worries me about current AGI hype is how it will probably encourage Mass Surveillance , as a path to data mining
Yep, although we even without AGI, we're headed for more mass surveillance. I talked about that in czcams.com/video/_2rWXVcWMt4/video.html
This is honestly how I learned to stop worrying and love the bomb
I love it when my little human brain and my old school common sense are confirmed by research. In my rudimentary way, I've been thinking that LLMs will run out of good data as soon as the AI-generated content started pouring into the world. It just made a sort of nebulous sense to me that they will run out of human-made data and they'll end up having to train on what they already generated, which will make them less and less imaginative. I'm a failed engineering student, currently a fiction writer. For nearly two years I couldn't write, believing that there's no point, when soon an AI will be able to give people books as easily as ordering fast food. And maybe that will be the case. All I can do right now is to try to use the LLMs to enhance my writing. There's nothing I'd rather do for a living than write books. Maybe this way I'll be able to write more and better books.
I'd love to be proven wrong, but you failed to convince me on diminishing returns.
Do you have actual data that would support any sort of diminishing returns ?
A score to a benchmark is not data.
1) It's not a measurement (it kind of is, but I mean it won't approximate a value that preexisted)
2) It's heterogeneous. You can't know what a x2 (or x100 for that matters) in performance should be expected to look like.
3) Assuming you're correct and this is linear; then what would you expect an exponential to look like ? 50% then 500% then 5000% ?
4) Assuming you're correct about the whole thing. Diminishing returns in scaling transformers is a statement you'd have to base on: either your own expertise, either on published papers. PaperS.
It's called a belief
Scores on benchmarks are, actually, data. In fact, that's the whole point of benchmarks.
If you want to insist on people giving you data while insisting that the data people have to give you doesn't count as data, then having a conversation with you is pointless.
Why do you want to be convinced on something you don't think is a measurement? If you don't like empirical data, you might as well open a church of llm.
@@InternetOfBugs It's not data in the sense a score has no intrinsic value. It can be used to compare LLMs between each other. And you didn't answer to "what would non-linear look like on a benchmark score ?"
> "what would non-linear look like on a benchmark score ?"
It would look like what any increasing-slope exponential graph looks like.
googlethatforyou.com?q=what%20does%20an%20exponential%20graph%20look%20like
This was the first AI/LLM take i havent heard yet in a WHILE! Thank you
You're very welcome.
Your human ability to learn continuously depends on you spending 1/3 of your time offline for information consolidation and training. Your context window for short term memory is one day. Try to learn a hard song on guitar, play it 50 times. Then sleep and play it again. The brain is training when you sleep.
I’ve literally said the same thing about the human brain to many people, and I’ll throw in the fact that people overwhelmingly underestimate the level of complexity that we humans have. It’s why I say that we will not have auto driving cars unless we take humans out of the driving all together, as the moment you introduce a human into a system you introduce a level of unpredictability people refuse to acknowledge.
Great video. As much as i do think AI is the future, as an assistant, i love these videos from the skeptics perspective. It keeps us all grounded
I also think AI is the future. I just haven't seen any evidence that it's going to be revolutionary any time soon.
I'm guessing that in the next 3-5 years, it will have something similar to the impact that smartphones had. Which is much bigger than nothing, but not a societally disruptive shift.
I think (or worry) you’re right. The moments I thought of underpants gnomes about AI was a) when Microsoft offered a GenAI as a feature of a search engine and b) OpenAI opened the GPT store before proving any clear runaway profitable use of it.
To me,
a) meant they were leaning into how the AI appeared to work (which works only so long as users aren’t trying to use it how it should work) and b) meant they were throwing working out how to use it to users.
Both suggesting (I think) the creators didn’t really know what to use it for….
I think the industry was really just surprised that throwing more data at LLMs created results as impressive as it did, and they got caught up in their own hype before they realized the limitations (although the cynical side of me says there was a good chance the knew and it was just a money-grab).
Nice one. Might be worth diving deeper on synthetic data, as I believe that's the reasoning for why Llama 3 is so strong at its size ( 8B / 70B models , 15T tok corpus ). Meta didn't mention what portion of that 15T is synthetic, but I imagine a fair amount.
I'm curious if the "convergence / photo copy chain" problem can be mitigated through increasing the diversity of sampling parameters and methods of sampling (dynamic temperature , min_p instead of top_p, etc). If you consider what percentage of all generative AI output is created through rather vanilla sampling with quantized to hell variants of gpt-3 and gpt-4, there is some hope for diversity and further improvements if open weight models and creative sampling eat away at O aye I's share
The best treatment I've seen for synthetic data is from arxiv.org/pdf/2404.05090 and it indicates that 20% Synthetic is mostly ok, 50% Synthetic still collapses, just less quickly.
Unless that paper turns out to have glaring errors, without a complete breakthrough, the best ratio we're going to get by adding synthetic data is not going to buy us much time.