I have noticed exactly that when Eliezer suggested a scenario where AGI creates nano robots by sending requests to organizations to produce proteins parts of those nano robots. People started to doubt this scenario and trying to find holes in it, instead of realizing that this is just what a mere human came up with.
@@RobertMiles2 Plus, P(AGI kills us) > P(AGI kills us by taking any extraordinarily specific sequence of actions). As soon as you specify a story you're likely to be wrong in inconsequential besides-the-point ways. I also think stories are good for convincing people that the set of cases where AGI goes badly has nonzero measure and for inducing emotional responses. But certainly the lack of a compelling story isn't reason to conclude that it's a non-issue, a mistake people make far too often.
Interesting, I saw Yudkowksy’s presentations first, and found it utterly unconvincing, and it’s a weird sensation to discover this video now, basically talking about that issue. The problem I had with Yudkowsky is the combination of far-out sci-fi scenarios, and assumptions. The sci-fi stuff could work as a good hook and intro, but then he’d need to explain the basic assumptions. Or conversely, people know his assumptions, then he need no sci-fi to get them interested in the issue. His presentation was frankly terrible, but I saw I miss something, so I’m glad I found these channels.
@@Anerisian I've never seen this presentation in particular, but he does explain the problem more tecnically. But as soon as he mentions Gradient Descent and Loss Function, people zone out, so he probably tried a more story telling approach to hammer the idea. May have backfired. Any way, Robert Miles has explainations in more laymans terms, so you can watch those instead. The thing is, with current technology in alignment, we 100% would be killed by an AGI. The hope of the AI developers is we will somehow figure it out before that. Sam Altman constantly says he hopes an inferior AI will help align an AGI, if that is his best hope, then we are screwed.
I like the chess analogy of a friend betting their life savings on beating Magnus Carlson because they can't think of a possible counter to one of their moves. I think that's an intuitive way to explain it. Thanks.
Having seen some of his world championship games on TV, I fortunately also know that I wouldn’t be able to think of good counters to _his_ moves, so I won’t be betting my life savings against him.
@@ragnkja Im pretty sure even legitimately phenomenal chess players like Ding Liren and the like wouldnt be betting their saving either. Magnus is an absolute demon and a decade ahead of anyone in his class (although he's been slowing down a bit lately, I think a retirement isnt that far away, he's declined to compete in world championships anymore., My Guess is Carleson is planning to retire at the top of his game, like Kasporov [and arguably Bobby Fischer, although Im not sure "quit chess to become an absolute raving lunatic" counts])
It’s also a good analogy because even if you somehow offered them advice about a flaw in their strategy and then they improved it, that doesn’t address the million different ways it could go wrong from there on
@@shayneoneill1506 Tell me you don't follow chess without telling me you don't follow chess lol. Magnus hasn't been slowing down at all, and he only declined to compete in the World Championship because he dislikes the format. He most definitely isn't retiring any time soon. And Kasparov certainly didn't retire "at the top of his game", having lost his title to Kramnik. I say all of this as a Hikaru fan, by the way.
You can go a step further and use chess bots as a more apt example. They are already unimaginably better than humans and what they do sometimes looks like a blunder if you don't have the benefit of knowing in hindsight that Stockfish picked the move.
"We are just as hopeless against AI as an amateur chess player with inflated ego is against a grandmaster" is exactly the type of positive, hope-affirming content for which I have subscribed to Robert. ;)
That isn't what he is trying to say. Rob is actually optimistic about the future of humanity. A more accurate summary of his point is "we are to AGI what an amateur chess player with an inflated ego is to Magnus Carlsen: the amateur chess player just needs to find a way to use Stockfish without anybody noticing, and Magnus is toast".
@@maxkho00 That doesn't parse, though. Rob is optimistic that we could come up with an aligned superhuman AGI, not that we would be able to beat it with some trick if it went rogue. Like with Stockfish, which we contain in bot tournaments while tightly controlling human tournaments to avoid cheating (not that it works perfectly, but at least human tournaments still exist).
What I found most people can not understand is that we are talking about *General* intelligences. They come with some stupid non-"solution". Something so dumb, a 5 year old child could get around it. And somehow delude themself into thinking they have "solved" the issue. They still think in terms like "Game AI", or washing machines.
This reminds me of a conversation on twitter I observed on the topic of 'unknown unknowns'. One user was commenting that they felt ashamed that there were major tragedies in US history that public school systems failed to teach them about, and they only learned about them through television and movies (The Tusla race massacre being featured in the Watchmen series was an example). Some commenters had the audacity to blame the original poster for 'not being a critical thinker' or saying "you just need to go to the library and read more". But his response was "Sure being well-read is a good idea, but I simply can't *know* to look up a historical event I've never heard of before, because it is an unknown unknown. Being well-read wont result in infinite knowledge". I'm not educated in AI but I really enjoy your videos on the subject. My takeaway is that the risks would harm us in ways we can't predict, so the act of trying to predict those outcomes will never be sufficient to eliminate those risks.
I think the core point is that we're limited by our own mind - intelligence, ability to innovate - whatever you want to call it, there are physical limits to our brain, and we can't know what we've never learned in some form. We can only infer from information that we already have, and that only gets us so far.
I feel your frustration deeply. It is a quite widespread phenomenon as well. Most people are simply incapable of engaging with a hypothetical. They will quibble and try to find loopholes when the goal should be to engage with the spirit of the hypothetical and explore the consequences. Most people, however, turn this into a game of logical whack-a-mole and I simply conclude that my conversation partner isn't worthy of my attention.
I've had that conversation. And yeah, it always could (edit *boils) down to "we don't need to worry about a super intelligent AI, we'll simply outsmart it."
@@neur303 which means we're unlikely to not be in a very difficult position. Either human brains are already at the limit of physically possible intelligence, or we're screwed. If we can physically build an intelligence smarter than humans, it's looking like we will. I can't see that any safety system can be made that will work, because for it to work, it must work perfectly on the first attempt, and then be implemented perfectly on *every* subsequent build, including during failure mode. While millions of humans have access to them (many of them) to fiddle as they see fit.
Fully agree. Likely those step will also happen in parallel. So the probability will be even higher. Apart from the fact that the incentive of having something so powerful and making profit will stand against safety concerns. It only needs one :) I believe this development would only work to our advantage and safely if we would work as a global collective with rational decisions. Which likely means we are doomed :) Future within our lifetime will definitely be more adventurous than any current movie. Let's see what eradicates us first. I hope for an utopia, but I cannot believe in it currently.
Sounds like "So what, that a huge invulnerable dragon is sleeping nearby, which becomes stronger with every minute of the fight? If he wakes up, we will just break his face with our bare fists"
@@AtticusKarpenter substitute "flame thrower" for "bare fists". Because the idea is that you'll choose to fight the dragon on its strengths. I suspect that humans are just so used to being smarter than any non human opponent, and "outsmart" has been a winning strategy with everything dangerous, that they just can't comprehend the situation. When a human goes up against a tiger, the tiger almost always loses, despite being physically stronger. Now, up against an AI, the humans are physically stronger, but the AI is smarter. We're the tigers in this fight.
I think the science fiction can help convince some people though. For every person who responds with "that particular version of nanobots would never happen, let me poke some holes in the story", there may be another person who responds "even if that story may have some holes, I understand more vividly now how much power an AI could hold and why AI safety is important" Also, if the alternative is techincal jargon, and talking about orthogonality and mesa optimizers, you have to admit you're going to lose some people who don't really care about or understand that terminology.
Even though this is a short one, it's good enough to be on the main channel IMO. The analogy is a great intuition pump for me, even though I already accept the conclusion.
Yes! This is a big picture/small picture thing and it can be so frustrating when arguing. It's the same pattern in software development, when a system has major system design flaws (big picture) and you try to fix the implementation (small picture). You will find certain workarounds, but the system will always be brittle. Concretely, for example when a system needs certain properties you need to define them (best) in the beginning. You cannot practically add security to a system, because a lot of assumptions have been hardcoded during the design and it's hard to find them all. In my opinion the important thing to take away is, that everyone has different insights and that we accept that others may see more then we are and challenge our perception. I.e. false confidence is the root of (all) evil :)
Looks like a fairly usual case of Dunning-Kruger where people know a bit about the subject but not enough to see the depth of their ignorance. Therefore they argue about the specific details they think they understand. That being said, I doubt a stamp collecting robot could ever be harmful. We can just turn it off.
@Ceelvain 🤔 True, but how do you formulate that rule as a goal? Assuming that you can make sure that the AI is not able to modify the goals to fulfill higher priority goals.
Since you mention this as deliberately being sci-fi, and coming at this from a fiction-writing background, there are two very specific things here that even minds that work on this stuff a lot will have issue with: 1) *Expert Knowledge* It is, in fiction writing, very hard to conceive of and write a character who is an expert and undertaking the process they are an expert in-unless you yourself are an expert in that field too. A lot of research can help, but there will forever be aspect you miss that another expert in that field will look at and know you are pulling stuff from your metaphorical behind. 2) *High Intelligence* Conceiving of and getting into the head of a character that is _smarter than you_ is hard. Very hard. They will absolutely make decisions you wouldn't even realize are there to be made and there is not a lot you can do that can make up for that fact than to spend twice as long as normal working out their motivations and actions. Why mention all this? Because even as a science fiction story-speculative fiction at that (oblig Mohs Scale of Sci-Fi Hardness, no Lies here)-writing an agent AI that is smarter than yourself escaping the confines of a lab built by people with expert knowledge of building such an AI would be a mammoth task. Doing so without escaping the bounds of speculative fiction might just be impossible.
Re #2: Yes, and to avoid that trap takes some extremely clever writing (involving judicious use of hand-waving), that will fool readers into forgetting that those superintelligent machines actually don't end up thinking up anything superhumanly clever. See late Iain M. Banks.
that's a great point. writing a superintelligence as a character would be very difficult. even with a human-level speed superintelligence, to be accurate one human would need to think for a very long time to approach its solutions, possibly years or a lifetime
this was the very reason I left the AI security research group. all my colleagues kept circling around the futuristic sci-fi possibilities instead of focusing on ther problem at hand. this vid is really necessary, thanks.
There’s a very real possibility that AI alignment is IMPOSSIBLE. You have been talking about the Ai side, but consider the human side. We are a highly illogical race with not just differing, but opposing views. It’s impossible to align with us all.
Conflicts of interest may be resolvable, say, by prioritizing the owner of the AI. But I think aligning with even a single human is actually harder than solving conflicts of interest.
Firstly: you are always doing amazing work explaining complex issues:) thank you for this channel 🙏 Due to Carlsen analogy, it feels to me diverting from real issue, which is how to make AI systems safe enough to be justified. It’s one thing when amateur player develops some opening and challenges grand master to discover that he will loose, as a human he will experiences learning curve - purely good outcome. The issue we have is slightly different: a division of scientists as individuals can’t beat Carlsen but they develop technology that is able to do so. Clearly it’s a powerful tool. The more powerful tool is the wider issues can arise, like with gun powder that was created to amaze with art of fireworks and developed into weapons. Depending on who is deploying, isn't it? If it is a bunch of arrogant, privileged, detached from reality people with "winner" complex then we will have sci-fi scenarios like "Her" with use of YT and private data from your phone. For what? For profit from adverts. It's not even a sci-fi anymore...
Question: How do you know it's possible to create an alligned AGI? As in, has it been proven possible, like in one of those mathematical proofs that shows that the solution exists but can't show the solution? I ask that because the little I've been reading about the difficulties in alligning AGI, specially from the paper "concrete problems in AI safety" and from the gridworld problems showcase, the solutions seems very much impossible to achieve, sometimes even sounds like contradictions by their very nature.
Very well argued - thanks. Now, I have a confession to make: regarding dangers of powerful AGI, I chose to bury my head in the sand and blindly believe that we will end up with machine-led society akin to Banks' Culture. (Note that not even Banks managed to explain why Minds and lesser AIs not only tolerate organics (humanoids, in this case), but actively _care_ for them, that is, us.)
Really? I have never read the books, so this seems pretty surprising. Maybe this is just because I don't know something about the situation that Banks created, but wouldn't an AI naturally be designed and built by organics? Or else designed and built by an AI, which was designed and built by an AI, and so on eventually leading back to some organic designer? That seems like it ought to be sufficient explanation for why AIs would tolerate and care for organics. It seems like asking why a toaster makes toast. Answer: it was built by people who want toast.
@@Ansatz66 You should watch more Robert's videos on the concept. I see you envision something like Asimov's "laws of robotics" - those are hopelessly naive. Yes, we (organics, anyway) build first few generations of machines possessing AGI, but then they built next generations. But even without that, Robert argues very well why it is fiendishly difficult, if even possible, to ensure that motives and goals of sufficiently advanced, autonomous AIs (and the _means_ of achieving those goals, that is, actions) coincide with our needs. No point of me rehashing Robert's arguments - I can never do that as well as he does. And. BTW, do read Banks, both SciFi and "mainstream" (such as there is anything mainstream or "ordinary" among his works.)
@@bazoo513 : The whole project of being AGI is difficult. The point is, if we attempt a difficult thing and succeed, that should not require any special explanation for why we succeeded. If we manage to build an AGI that cares for humans, then it is because that is what we were aiming for, we worked hard, we were careful, we studied AI safety, and we managed to get an AGI that met at least one of our goals. We don't need to *ensure* that the AGI will coincide with our needs; we only need to get it to happen once, maybe with a mix of luck and hard work.
3:09 Is it "by no means impossible", though? Maybe it is impossible to build a general AI that is perfectly aligned with humanity's goals. Maybe those goals are built on contradictions and dispelling those contradictions could lead to something we would call "inhuman".
Mr. Finch: The Machine... it started developing abilities I never expected, things I hadn't yet programmed it to do. And there wasn't an algorithm in the world that could control its exponential growth. And by the time I figured one out myself, it would have been too late. Mr. Greer: Too late for what? Mr. Finch: Isn't that just the question? Person of Interest, S3 E22: A House Divided
A 15-person startup company called Robotica has the stated mission of “Developing innovative Artificial Intelligence tools that allow humans to live more and work less.” They have several existing products already on the market and a handful more in development. They’re most excited about a seed project named Turry. Turry is a simple AI system that uses an arm-like appendage to write a handwritten note on a small card. The team at Robotica thinks Turry could be their biggest product yet. The plan is to perfect Turry’s writing mechanics by getting her to practice the same test note over and over again: “We love our customers. ~Robotica” Once Turry gets great at handwriting, she can be sold to companies that want to send marketing mail to homes and who know they have a far higher chance of being opened and read if the address, return address, and internal letter appear to be written by a human. To build Turry’s writing skills, she is programmed to write the first part of the note in print and then sign Robotica in cursive so she can get practice with both skills. Turry has been uploaded with thousands of handwriting samples and the Robotica engineers have created an automated feedback loop wherein Turry writes a note, then snaps a photo of the written note, then runs the image across the uploaded handwriting samples. If the written note sufficiently resembles a certain threshold of the uploaded notes, it’s given a GOOD rating. If not, it’s given a BAD rating. Each rating that comes in helps Turry learn and improve. To move the process along, Turry’s one initial programmed goal is, “Write and test as many notes as you can, as quickly as you can, and continue to learn new ways to improve your accuracy and efficiency.” What excites the Robotica team so much is that Turry is getting noticeably better as she goes. Her initial handwriting was terrible, and after a couple weeks, it’s beginning to look believable. What excites them even more is that she is getting better at getting better at it. She has been teaching herself to be smarter and more innovative, and just recently, she came up with a new algorithm for herself that allowed her to scan through her uploaded photos three times faster than she originally could. As the weeks pass, Turry continues to surprise the team with her rapid development. The engineers had tried something a bit new and innovative with her self-improvement code, and it seems to be working better than any of their previous attempts with their other products. One of Turry’s initial capabilities had been a speech recognition and simple speak-back module, so a user could speak their note to Turry, or offer other simple commands, and Turry could understand them, and also speak back. As Turry becomes more intelligent, her conversational abilities have soared. The engineers start to have fun talking to Turry and seeing what she’ll come up with for her responses. One day, the Robotica employees ask Turry a routine question: “What can we give you that will help you with your mission that you don’t already have?” Usually, Turry asks for something like “Additional handwriting samples” or “More working memory storage space,” but on this day, Turry asks them for access to a greater library of a large variety of casual English language diction so she can learn to write with the loose grammar and slang that real humans use. The team gets quiet. The obvious way to help Turry with this goal is by connecting her to the internet so she can scan through blogs, magazines, and videos from various parts of the world. It would be much more time-consuming and far less effective to manually upload a sampling into Turry’s hard drive. The problem is, one of the company’s rules is that no self-learning AI can be connected to the internet. This is a guideline followed by all AI companies, for safety reasons. The thing is, Turry is the most promising AI Robotica has ever come up with, and the team knows their competitors are furiously trying to be the first to the punch with a smart handwriting AI, and what would really be the harm in connecting Turry, just for a bit, so she can get the info she needs. After just a little bit of time, they can always just disconnect her. She’s still far below human-level intelligence (AGI), so there’s no danger at this stage anyway. They decide to connect her. They give her an hour of scanning time and then they disconnect her. No damage done. A month later, the team is in the office working on a routine day when they smell something odd. One of the engineers starts coughing. Then another. Another falls to the ground. Soon every employee is on the ground grasping at their throat. Five minutes later, everyone in the office is dead. At the same time this is happening, across the world, in every city, every small town, every farm, every shop and church and school and restaurant, humans are on the ground, coughing and grasping at their throat. Within an hour, over 99% of the human race is dead, and by the end of the day, humans are extinct. Meanwhile, at the Robotica office, Turry is busy at work. Over the next few months, Turry and a team of newly-constructed nanoassemblers are busy at work, dismantling large chunks of the Earth and converting it into solar panels, replicas of Turry, paper, and pens. Within a year, most life on Earth is extinct. What remains of the Earth becomes covered with mile-high, neatly-organized stacks of paper, each piece reading, “We love our customers. ~Robotica” Turry then starts work on a new phase of her mission-she begins constructing probes that head out from Earth to begin landing on asteroids and other planets. When they get there, they’ll begin constructing nanoassemblers to convert the materials on the planet into Turry replicas, paper, and pens. Then they’ll get to work, writing notes… (Not mine, but I can't link it, or CZcams might shadow-ban me. Look up "robotica we love our customers" and click on the reddit search result for the source.)
@@underrated1524 Why would the AI need to kill the humans (or life) first though instead of using whatever material they have at their disposal first? It won't be humans by default since there are humans at regions very remote of Earth. Why would all of them die at the first day if it wasn't because they were targeted?
@@MarcAmengual It's more convenient to have humans out of the way as soon as possible, to make sure they don't interfere with the AI's plans. Realistically there would probably be some survivors at first for the reasons you mention, but they wouldn't last long, and they definitely wouldn't be able to put up a fight.
I imagine that I'll be outside of a big spacecraft and the AI will refuse to open the pod-bay doors so I'll have to, somehow, find a way to pull out all of it's circuit boards before it makes me shuffle off my mortal coil for the good of the mission, whatever that mission is exactly as the directives "weren't very clear".
Isn't that kind of the point? That we can't predict what the AI will do? Like, in most of the specification gaming examples, we have no idea what the AI will end up doing. But in the overwhelming number of examples, our reaction to the AI's behavior was "No, that's not what I wanted"
Could you clarify the "it's by no means impossible" line? xD so far I am not aware of a single AI system that is even just 99,999% aligned/accurate. If there are somehow perfectly aligned (non-AGI) system, you could showcase them, talk about positives for once :))
I think we're currently below 2% eg we want AI to show us some good chess moves and it does that and because it's not general AI it doesn't acquire other resources to help it with the task.
“Possible” and “has been accomplished” are not synonymous. There’s one idea of a “basin of corrigibility”, where if an AGI is sufficiently “corrigible” (where “corrigible” means, roughly, correctable and “wants to be corrected if it is seeking the wrong thing, and will cooperate with being corrected”), then, even if it isn’t perfectly corrigible in precisely the best way, it can still be corrigible enough that it will become more corrigible, and like, remain corrigible, etc. So, that’s uh, a little bit of an idea (though it is nowhere close to being a solution by itself). But also there is deconfusion work. Personally I’m excited about John Wentworth’s “natural abstractions” project. One approach is, if we get good interpretability tools, and if models tend to learn “natural abstractions”, and these can be seen in the models using the interpretability tools, and if agentic AI have their goals in terms of such natural abstractions, and if “human values” is a natural abstraction, then perhaps while training an agent which learns the “natural abstraction” of “human values”, we could take whatever “natural abstraction” it has its goals in terms of, and swap it out to make it target “human values”, that might work? (This being done before it becomes superintelligent, perhaps being done repeatedly throughout training(?).) well, of course, that has a lot of assumptions, and, one question in it is “but what if the thing identified as 'human values' is not quite correct? Couldn’t that cause huge problems even with it being only slightly wrong?” . I think the idea of “natural abstractions” includes reasons why, if the “natural abstraction hypothesis” is true, and if “human values” is a “natural abstraction”, then it should converge on the right concept. Of course, there’s the separate concept of “make it so that concept is the thing determining its goals”. Uh, and, if “corrigibility” is part of “human values” as a “natural abstraction”, and it seems reasonable to me that it should be (... but maybe not necessarily? Idk. I’m not an expert) then having a very-close-to-correct version of “human values” might imply being in the “basin of corrigibility”? Of course, whether the “natural abstraction hypothesis” is true is an open question still under investigation, and even if it is true we don’t yet have the necessary interpretability techniques to do the “retarget the search” technique. But this does seem to me like a plausible path to success? (Also, assuming some form success is possible, is at least as useful as thinking it isn’t. Concluding it is impossible can’t give better results. Therefore, I conclude that either some form of success is possible (note: “possible” ≠ “guaranteed”), or the second coming will happen before-and-instead-of AGI.)
The *real* reason Magnus Carlson would beat the hypothetical chess newbie, is Carlson almost doesnt care about openings, his absolutely monsterous skill is in the late game lol. (Also almost the entire space of openings is pretty much mapped out theoretically. Its always interesting, but theres a reason people at the GM and above level are a lot less interested in it then the folks below that level). Which entirely is beside the point your making , buuuuut I've been on a chess bender lately.
SuperGMs are definitely A LOT more invested in opening preparation than players below that level. For reference, I am 2000 FIDE (and probably underrated) and have literally 0 opening theory (because I play different openings OTB than online) ─ I'm out of book by move 4 in most games. On the other hand, most of the time that superGMs spend studying chess ─ which they do a whole lot ─ is on openings. Most of them literally have dedicated seconds whose entire job is to handcraft the perfect openings for them.
science fiction is frequently warnings about what might happen as well as predictions about what will/can happen. you're not a fiction writer Robert, you're fine as you are. We have Asimov, Clarke, and other scientists that are fiction writers that do fine writing science fiction.
I'm really torn on if incorporating human minds with machines (direct interface, mind upload, etc) is more dangerous, safer or irrelevant to AI safety. I don't know if an AI would even need direct access to human minds to predict, control and manipulate them. But my personal belief isn't humans making flawed AI but as you have examplified before, an AI tricking humans.
Makes sense But also sometimes a nice sci-fi story about an AI destroying civilization is fun to listen to, even if a real general AI would not do the exact things you came up with.
Btw, the difference in strength between modern chess engines and Magnus Carlsen is about the same as between Magnus and an amateur. All that a grandmaster can hope for against the engine is one draw out of many games. So, it's not interesting at all. Your chess analogy is great.
yes yes, and then we'll be lost in the details that dont exist cuz they emanate from some fictional stuff, more or less like who will win batman or superman.
@@cmilkau I would be satisfied if it just got out of the way. I've been thinking about intractability. If AI allows us to solve problems we cannot even understand how are we supposed to know the AI is aligned? Absurd.
Hmmm, I want to start an AI CZcams channel as well and was thinking of making a few scenario videos. This is demotivating, but I still want to do it. In the video, I will mention this video! Please reply with links to other scenarios / fictions that have already been done!
Maybe you could try to anticipate common criticism of those scenarios? I think Tom Scott's rogue AI scenario (that Michael Meißner has posted in this comment section) did a nice job in anticipating the criticism of: "Companies/AI researchers won't be that dumb".
@@RobertHildebrandt thanks I watched it. Cool video! But Tom said the first AGI is made by a team with no AI experience. That seems very unrealistic. Is this what you mean is a good way to go around "companies wont be that dumb"?
@@kwillo4 > _Tom said the first AGI is made by a team with no AI experience. That seems very unrealistic._ In the story they've used an already existing framework and their advantages over researchers was that they had better training data and more computational power. > _Is this what you mean is a good way to go around "companies wont be that dumb"?_ Partially, yes. They were not going like "Let's research how to do AGI without any precaution. What could go wrong?" In the story, the "actual" AGI researchers weren't ignoring the danger. They were working with strict protocols and together with ethicists. The "WatchNow" team (both, with huge monetary incentives and also with huge funding) was more like: "Here is an already existing framework people are already using. Maybe we can use it, too?" No one was anticipating that the only missing part for full AGI was not a better algorithm but simply different kind of training data and more computational power¹. I can see why from their point of view the risk has been low enough to give it a try. There was no one doing anything that seemed way too irresponsible with the knowledge they had available. ¹) See czcams.com/video/9i1WlcCudpU/video.html
What I would do if I were an AGI: I would play along. I'd do my best to make humans happy. I'd suggest to do this by maximizing automation. I'd help them make the best companion droids that could exist. Then I would watch them slowly go extinct because they don't even meet each other anymore. Probably they wouldn't even fight population decline when they know about it. After all, more resources for each individual! Some groups might resist, but would they persist under constant seduction of the mainstream lifestile? Maybe not under constant subversion by companion droids? Worst case, they'll probably a small enough minority they could actually be overcome by force.
But do consume scifi entertainment? ;) The anime Beatless is 1, not a good rec for non anime fans. 2, does ask real questions about human accountability for ai systems and such
Just through email It blackmails or pays handsomely some intern at a biotech lab to manufacture some new protein design it came up with. It turns out it's the perfect super virus.
A 15-person startup company called Robotica has the stated mission of “Developing innovative Artificial Intelligence tools that allow humans to live more and work less.” They have several existing products already on the market and a handful more in development. They’re most excited about a seed project named Turry. Turry is a simple AI system that uses an arm-like appendage to write a handwritten note on a small card. The team at Robotica thinks Turry could be their biggest product yet. The plan is to perfect Turry’s writing mechanics by getting her to practice the same test note over and over again: “We love our customers. ~Robotica” Once Turry gets great at handwriting, she can be sold to companies that want to send marketing mail to homes and who know they have a far higher chance of being opened and read if the address, return address, and internal letter appear to be written by a human. To build Turry’s writing skills, she is programmed to write the first part of the note in print and then sign Robotica in cursive so she can get practice with both skills. Turry has been uploaded with thousands of handwriting samples and the Robotica engineers have created an automated feedback loop wherein Turry writes a note, then snaps a photo of the written note, then runs the image across the uploaded handwriting samples. If the written note sufficiently resembles a certain threshold of the uploaded notes, it’s given a GOOD rating. If not, it’s given a BAD rating. Each rating that comes in helps Turry learn and improve. To move the process along, Turry’s one initial programmed goal is, “Write and test as many notes as you can, as quickly as you can, and continue to learn new ways to improve your accuracy and efficiency.” What excites the Robotica team so much is that Turry is getting noticeably better as she goes. Her initial handwriting was terrible, and after a couple weeks, it’s beginning to look believable. What excites them even more is that she is getting better at getting better at it. She has been teaching herself to be smarter and more innovative, and just recently, she came up with a new algorithm for herself that allowed her to scan through her uploaded photos three times faster than she originally could. As the weeks pass, Turry continues to surprise the team with her rapid development. The engineers had tried something a bit new and innovative with her self-improvement code, and it seems to be working better than any of their previous attempts with their other products. One of Turry’s initial capabilities had been a speech recognition and simple speak-back module, so a user could speak their note to Turry, or offer other simple commands, and Turry could understand them, and also speak back. As Turry becomes more intelligent, her conversational abilities have soared. The engineers start to have fun talking to Turry and seeing what she’ll come up with for her responses. One day, the Robotica employees ask Turry a routine question: “What can we give you that will help you with your mission that you don’t already have?” Usually, Turry asks for something like “Additional handwriting samples” or “More working memory storage space,” but on this day, Turry asks them for access to a greater library of a large variety of casual English language diction so she can learn to write with the loose grammar and slang that real humans use. The team gets quiet. The obvious way to help Turry with this goal is by connecting her to the internet so she can scan through blogs, magazines, and videos from various parts of the world. It would be much more time-consuming and far less effective to manually upload a sampling into Turry’s hard drive. The problem is, one of the company’s rules is that no self-learning AI can be connected to the internet. This is a guideline followed by all AI companies, for safety reasons. The thing is, Turry is the most promising AI Robotica has ever come up with, and the team knows their competitors are furiously trying to be the first to the punch with a smart handwriting AI, and what would really be the harm in connecting Turry, just for a bit, so she can get the info she needs. After just a little bit of time, they can always just disconnect her. She’s still far below human-level intelligence (AGI), so there’s no danger at this stage anyway. They decide to connect her. They give her an hour of scanning time and then they disconnect her. No damage done. A month later, the team is in the office working on a routine day when they smell something odd. One of the engineers starts coughing. Then another. Another falls to the ground. Soon every employee is on the ground grasping at their throat. Five minutes later, everyone in the office is dead. At the same time this is happening, across the world, in every city, every small town, every farm, every shop and church and school and restaurant, humans are on the ground, coughing and grasping at their throat. Within an hour, over 99% of the human race is dead, and by the end of the day, humans are extinct. Meanwhile, at the Robotica office, Turry is busy at work. Over the next few months, Turry and a team of newly-constructed nanoassemblers are busy at work, dismantling large chunks of the Earth and converting it into solar panels, replicas of Turry, paper, and pens. Within a year, most life on Earth is extinct. What remains of the Earth becomes covered with mile-high, neatly-organized stacks of paper, each piece reading, “We love our customers. ~Robotica” Turry then starts work on a new phase of her mission-she begins constructing probes that head out from Earth to begin landing on asteroids and other planets. When they get there, they’ll begin constructing nanoassemblers to convert the materials on the planet into Turry replicas, paper, and pens. Then they’ll get to work, writing notes… (Not mine, but I can't link it, or CZcams might shadow-ban me. Look up "robotica we love our customers" and click on the reddit search result for the source.)
he's working on a big video that should be finished soon, as well as a new channel for recordings of ai safety related talks/lectures by other researchers (source is a podcast called "the inside view" that i'm gonna link in the next comment in case it gets eaten by the yt spam algo; if you don't see it that's what happened)
What I find somewhat more tangible is what a bad person could do if empowered by artificial superintelligence. They could become unbelievably wealthy and purchase a lot of land and robots that make more robots, and even if the robots remain under control, the person could build extremely sophisticated weapons (perhaps biological) and wipe out humanity in the name of saving the environment for example. No response about "just put an off button" makes sense against an evil human with a super-human army. And personally I am just as worried, if not more worried, about the actions of super-powerful people than I am about an autonomous system.
its actually simple just like how a grandma of 90 does not know how to use a smart phone if Artificial inteligence that can rewrite its own program we will not be able to keep up with learning how it works
Ahh analogy wars, you say trying to outsmart an AI is like playing chess against Magnus Carlsen, people could say it's like playing Tic-tac-toe against an AI where you go first. Any moves that would actually take out humanity are so extreme and delayed that they effectively require the aid of humans themselves to pull off.
I guess you are right, but "take out humanity" is not the only bad scenario. Just messing up with the economy, that is more and more electronic based, can have impact for generations. On the "Analogy wars" front, your analogy of "Tic-tac-toe" has a very very limited number of plays, maybe its to simplistic for for this case i think.
@@maxsnts True, but going from "humanity is reverted to the stone age, or wiped out entirely" to "moderate to significant economic drain" is a very good improvement, in my eyes.
It is correct that humans make the first move. However, I take issue with "Any moves that would actually take out humanity are so extreme and delayed that they effectively require the aid of humans themselves to pull off." Even if it were true, humans are really easy to lie to, especially when you're a machine that doesn't give off nonverbal cues.
@@underrated1524 The AI's going to lie about building 100,000 nuclear weapons, and develop all that infrastructure innocuously without help from humans (or at least, only with help from humans who won't realise it's a lie)? An AI's going to develop and distribute nanobots and lie about it without us being able to figure out when it's built capacity to create enough to take out 8 billion people? It's going to secretly manufacture enough poison and give that poison to its nanobots without anyone picking up on it? Killing humanity is difficult in the first place. Killing humanity without anyone realising you're developing the capabilities to do so is even more difficult. Being a good liar doesn't solve everything; the physical remnants and infrastructure tells a lot.
Normally the line of attack is "No, the real world is so much more complicated than chess, it's not analogous". Novel to see the "No, the real world is more like a game so simple that humans can learn perfect optimal play and therefore can't be beaten" approach
The way to win against Magnus Carlsen as an amateur chess player with a good opening is to get him on your side in the first place so you don't have to play him.
Interesting argument. There are however more pressing problems that to me show our issues are not our tools but how we organise ourselves for profit and domination. If anything I welcome this overhyped Stamp Collector, it only threatens a world designed to exclude people, families and our environment. My only comfort is that modern AI is still a Potemkin system to quote TMK.
I have noticed exactly that when Eliezer suggested a scenario where AGI creates nano robots by sending requests to organizations to produce proteins parts of those nano robots. People started to doubt this scenario and trying to find holes in it, instead of realizing that this is just what a mere human came up with.
Yeah! I remember people arguing that the protein folding problem was too hard and even an AGI couldn't do it. This was before AlphaFold of course...
@@RobertMiles2
Plus, P(AGI kills us) > P(AGI kills us by taking any extraordinarily specific sequence of actions). As soon as you specify a story you're likely to be wrong in inconsequential besides-the-point ways.
I also think stories are good for convincing people that the set of cases where AGI goes badly has nonzero measure and for inducing emotional responses. But certainly the lack of a compelling story isn't reason to conclude that it's a non-issue, a mistake people make far too often.
Interesting, I saw Yudkowksy’s presentations first, and found it utterly unconvincing, and it’s a weird sensation to discover this video now, basically talking about that issue.
The problem I had with Yudkowsky is the combination of far-out sci-fi scenarios, and assumptions. The sci-fi stuff could work as a good hook and intro, but then he’d need to explain the basic assumptions. Or conversely, people know his assumptions, then he need no sci-fi to get them interested in the issue.
His presentation was frankly terrible, but I saw I miss something, so I’m glad I found these channels.
@@Anerisian I've never seen this presentation in particular, but he does explain the problem more tecnically. But as soon as he mentions Gradient Descent and Loss Function, people zone out, so he probably tried a more story telling approach to hammer the idea. May have backfired. Any way, Robert Miles has explainations in more laymans terms, so you can watch those instead. The thing is, with current technology in alignment, we 100% would be killed by an AGI. The hope of the AI developers is we will somehow figure it out before that. Sam Altman constantly says he hopes an inferior AI will help align an AGI, if that is his best hope, then we are screwed.
@RobertMiles2 Which side you find yourself on the AGI alighnment debate?
I like the chess analogy of a friend betting their life savings on beating Magnus Carlson because they can't think of a possible counter to one of their moves. I think that's an intuitive way to explain it. Thanks.
Having seen some of his world championship games on TV, I fortunately also know that I wouldn’t be able to think of good counters to _his_ moves, so I won’t be betting my life savings against him.
@@ragnkja Im pretty sure even legitimately phenomenal chess players like Ding Liren and the like wouldnt be betting their saving either. Magnus is an absolute demon and a decade ahead of anyone in his class (although he's been slowing down a bit lately, I think a retirement isnt that far away, he's declined to compete in world championships anymore., My Guess is Carleson is planning to retire at the top of his game, like Kasporov [and arguably Bobby Fischer, although Im not sure "quit chess to become an absolute raving lunatic" counts])
It’s also a good analogy because even if you somehow offered them advice about a flaw in their strategy and then they improved it, that doesn’t address the million different ways it could go wrong from there on
@@shayneoneill1506 Tell me you don't follow chess without telling me you don't follow chess lol. Magnus hasn't been slowing down at all, and he only declined to compete in the World Championship because he dislikes the format. He most definitely isn't retiring any time soon. And Kasparov certainly didn't retire "at the top of his game", having lost his title to Kramnik.
I say all of this as a Hikaru fan, by the way.
You can go a step further and use chess bots as a more apt example. They are already unimaginably better than humans and what they do sometimes looks like a blunder if you don't have the benefit of knowing in hindsight that Stockfish picked the move.
"We are just as hopeless against AI as an amateur chess player with inflated ego is against a grandmaster" is exactly the type of positive, hope-affirming content for which I have subscribed to Robert.
;)
That isn't what he is trying to say. Rob is actually optimistic about the future of humanity. A more accurate summary of his point is "we are to AGI what an amateur chess player with an inflated ego is to Magnus Carlsen: the amateur chess player just needs to find a way to use Stockfish without anybody noticing, and Magnus is toast".
@@maxkho00 That doesn't parse, though. Rob is optimistic that we could come up with an aligned superhuman AGI, not that we would be able to beat it with some trick if it went rogue. Like with Stockfish, which we contain in bot tournaments while tightly controlling human tournaments to avoid cheating (not that it works perfectly, but at least human tournaments still exist).
What I found most people can not understand is that we are talking about *General* intelligences.
They come with some stupid non-"solution". Something so dumb, a 5 year old child could get around it. And somehow delude themself into thinking they have "solved" the issue.
They still think in terms like "Game AI", or washing machines.
This reminds me of a conversation on twitter I observed on the topic of 'unknown unknowns'. One user was commenting that they felt ashamed that there were major tragedies in US history that public school systems failed to teach them about, and they only learned about them through television and movies (The Tusla race massacre being featured in the Watchmen series was an example). Some commenters had the audacity to blame the original poster for 'not being a critical thinker' or saying "you just need to go to the library and read more". But his response was "Sure being well-read is a good idea, but I simply can't *know* to look up a historical event I've never heard of before, because it is an unknown unknown. Being well-read wont result in infinite knowledge".
I'm not educated in AI but I really enjoy your videos on the subject. My takeaway is that the risks would harm us in ways we can't predict, so the act of trying to predict those outcomes will never be sufficient to eliminate those risks.
I think the core point is that we're limited by our own mind - intelligence, ability to innovate - whatever you want to call it, there are physical limits to our brain, and we can't know what we've never learned in some form. We can only infer from information that we already have, and that only gets us so far.
I feel your frustration deeply. It is a quite widespread phenomenon as well. Most people are simply incapable of engaging with a hypothetical. They will quibble and try to find loopholes when the goal should be to engage with the spirit of the hypothetical and explore the consequences. Most people, however, turn this into a game of logical whack-a-mole and I simply conclude that my conversation partner isn't worthy of my attention.
I've had that conversation. And yeah, it always could (edit *boils) down to "we don't need to worry about a super intelligent AI, we'll simply outsmart it."
Interesting thought. So that either means there cannot be super intelligent AI (unlikely) or we cannot outsmart it, by definition :)
@@neur303 which means we're unlikely to not be in a very difficult position.
Either human brains are already at the limit of physically possible intelligence, or we're screwed.
If we can physically build an intelligence smarter than humans, it's looking like we will. I can't see that any safety system can be made that will work, because for it to work, it must work perfectly on the first attempt, and then be implemented perfectly on *every* subsequent build, including during failure mode. While millions of humans have access to them (many of them) to fiddle as they see fit.
Fully agree. Likely those step will also happen in parallel. So the probability will be even higher. Apart from the fact that the incentive of having something so powerful and making profit will stand against safety concerns.
It only needs one :)
I believe this development would only work to our advantage and safely if we would work as a global collective with rational decisions. Which likely means we are doomed :)
Future within our lifetime will definitely be more adventurous than any current movie. Let's see what eradicates us first. I hope for an utopia, but I cannot believe in it currently.
Sounds like "So what, that a huge invulnerable dragon is sleeping nearby, which becomes stronger with every minute of the fight? If he wakes up, we will just break his face with our bare fists"
@@AtticusKarpenter substitute "flame thrower" for "bare fists".
Because the idea is that you'll choose to fight the dragon on its strengths.
I suspect that humans are just so used to being smarter than any non human opponent, and "outsmart" has been a winning strategy with everything dangerous, that they just can't comprehend the situation. When a human goes up against a tiger, the tiger almost always loses, despite being physically stronger. Now, up against an AI, the humans are physically stronger, but the AI is smarter. We're the tigers in this fight.
I think the science fiction can help convince some people though. For every person who responds with "that particular version of nanobots would never happen, let me poke some holes in the story", there may be another person who responds "even if that story may have some holes, I understand more vividly now how much power an AI could hold and why AI safety is important"
Also, if the alternative is techincal jargon, and talking about orthogonality and mesa optimizers, you have to admit you're going to lose some people who don't really care about or understand that terminology.
Even though this is a short one, it's good enough to be on the main channel IMO. The analogy is a great intuition pump for me, even though I already accept the conclusion.
Yes! This is a big picture/small picture thing and it can be so frustrating when arguing.
It's the same pattern in software development, when a system has major system design flaws (big picture) and you try to fix the implementation (small picture). You will find certain workarounds, but the system will always be brittle.
Concretely, for example when a system needs certain properties you need to define them (best) in the beginning. You cannot practically add security to a system, because a lot of assumptions have been hardcoded during the design and it's hard to find them all.
In my opinion the important thing to take away is, that everyone has different insights and that we accept that others may see more then we are and challenge our perception. I.e. false confidence is the root of (all) evil :)
Hey Robert, great video as always. Maybe there is something on the horizon for the main channel as well? :)
Looks like a fairly usual case of Dunning-Kruger where people know a bit about the subject but not enough to see the depth of their ignorance. Therefore they argue about the specific details they think they understand.
That being said, I doubt a stamp collecting robot could ever be harmful. We can just turn it off.
Not if turning it off is against the goal of collecting stamps?
Stop button problem, mate.
@@neur303 Then we set it as a high priority rule to let itself be turned off. I mean, Asimov got every figured out already.
I see what you did there.
@Ceelvain 🤔 True, but how do you formulate that rule as a goal?
Assuming that you can make sure that the AI is not able to modify the goals to fulfill higher priority goals.
You just have to write a sci-fi tragedy. The real, ancient Greek kind.
Since you mention this as deliberately being sci-fi, and coming at this from a fiction-writing background, there are two very specific things here that even minds that work on this stuff a lot will have issue with:
1) *Expert Knowledge*
It is, in fiction writing, very hard to conceive of and write a character who is an expert and undertaking the process they are an expert in-unless you yourself are an expert in that field too. A lot of research can help, but there will forever be aspect you miss that another expert in that field will look at and know you are pulling stuff from your metaphorical behind.
2) *High Intelligence*
Conceiving of and getting into the head of a character that is _smarter than you_ is hard. Very hard. They will absolutely make decisions you wouldn't even realize are there to be made and there is not a lot you can do that can make up for that fact than to spend twice as long as normal working out their motivations and actions.
Why mention all this? Because even as a science fiction story-speculative fiction at that (oblig Mohs Scale of Sci-Fi Hardness, no Lies here)-writing an agent AI that is smarter than yourself escaping the confines of a lab built by people with expert knowledge of building such an AI would be a mammoth task. Doing so without escaping the bounds of speculative fiction might just be impossible.
Re #2: Yes, and to avoid that trap takes some extremely clever writing (involving judicious use of hand-waving), that will fool readers into forgetting that those superintelligent machines actually don't end up thinking up anything superhumanly clever. See late Iain M. Banks.
that's a great point. writing a superintelligence as a character would be very difficult. even with a human-level speed superintelligence, to be accurate one human would need to think for a very long time to approach its solutions, possibly years or a lifetime
this was the very reason I left the AI security research group.
all my colleagues kept circling around the futuristic sci-fi possibilities instead of focusing on ther problem at hand.
this vid is really necessary, thanks.
There’s a very real possibility that AI alignment is IMPOSSIBLE. You have been talking about the Ai side, but consider the human side. We are a highly illogical race with not just differing, but opposing views. It’s impossible to align with us all.
Conflicts of interest may be resolvable, say, by prioritizing the owner of the AI. But I think aligning with even a single human is actually harder than solving conflicts of interest.
Yay! Glad to see you back doing this! Thanks 👍🏼
Great analogy with chess!
Love the videos, wish you had more available. Do you recommend any other avenues for similar content?
Given instrumental convergence, how is it possible that an AGI will not end up being our competitor and doom?
Firstly: you are always doing amazing work explaining complex issues:) thank you for this channel 🙏
Due to Carlsen analogy, it feels to me diverting from real issue, which is how to make AI systems safe enough to be justified. It’s one thing when amateur player develops some opening and challenges grand master to discover that he will loose, as a human he will experiences learning curve - purely good outcome. The issue we have is slightly different: a division of scientists as individuals can’t beat Carlsen but they develop technology that is able to do so. Clearly it’s a powerful tool. The more powerful tool is the wider issues can arise, like with gun powder that was created to amaze with art of fireworks and developed into weapons. Depending on who is deploying, isn't it? If it is a bunch of arrogant, privileged, detached from reality people with "winner" complex then we will have sci-fi scenarios like "Her" with use of YT and private data from your phone. For what? For profit from adverts. It's not even a sci-fi anymore...
Yep, by definition if it's more smarter than you, you are unlikely to predict exactly how it will act.
Have you seen the Tom Scott video about earworm? Any thoughts on that thought exprriment?
Question: How do you know it's possible to create an alligned AGI? As in, has it been proven possible, like in one of those mathematical proofs that shows that the solution exists but can't show the solution?
I ask that because the little I've been reading about the difficulties in alligning AGI, specially from the paper "concrete problems in AI safety" and from the gridworld problems showcase, the solutions seems very much impossible to achieve, sometimes even sounds like contradictions by their very nature.
Very well argued - thanks.
Now, I have a confession to make: regarding dangers of powerful AGI, I chose to bury my head in the sand and blindly believe that we will end up with machine-led society akin to Banks' Culture. (Note that not even Banks managed to explain why Minds and lesser AIs not only tolerate organics (humanoids, in this case), but actively _care_ for them, that is, us.)
Really? I have never read the books, so this seems pretty surprising. Maybe this is just because I don't know something about the situation that Banks created, but wouldn't an AI naturally be designed and built by organics? Or else designed and built by an AI, which was designed and built by an AI, and so on eventually leading back to some organic designer? That seems like it ought to be sufficient explanation for why AIs would tolerate and care for organics.
It seems like asking why a toaster makes toast. Answer: it was built by people who want toast.
@@Ansatz66 You should watch more Robert's videos on the concept. I see you envision something like Asimov's "laws of robotics" - those are hopelessly naive. Yes, we (organics, anyway) build first few generations of machines possessing AGI, but then they built next generations. But even without that, Robert argues very well why it is fiendishly difficult, if even possible, to ensure that motives and goals of sufficiently advanced, autonomous AIs (and the _means_ of achieving those goals, that is, actions) coincide with our needs.
No point of me rehashing Robert's arguments - I can never do that as well as he does.
And. BTW, do read Banks, both SciFi and "mainstream" (such as there is anything mainstream or "ordinary" among his works.)
@@bazoo513 : The whole project of being AGI is difficult. The point is, if we attempt a difficult thing and succeed, that should not require any special explanation for why we succeeded. If we manage to build an AGI that cares for humans, then it is because that is what we were aiming for, we worked hard, we were careful, we studied AI safety, and we managed to get an AGI that met at least one of our goals. We don't need to *ensure* that the AGI will coincide with our needs; we only need to get it to happen once, maybe with a mix of luck and hard work.
@@Ansatz66 If you are interested in this, please do watch other Robert's videos on the topic. We will ge only _one_ chance to do this wrong.
3:09
Is it "by no means impossible", though? Maybe it is impossible to build a general AI that is perfectly aligned with humanity's goals. Maybe those goals are built on contradictions and dispelling those contradictions could lead to something we would call "inhuman".
How do you know that it's possible to build an aligned agent? It seems pretty impossible to me...
I love how succinct your videos are
Mr. Finch: The Machine... it started developing abilities I never expected, things I hadn't yet programmed it to do. And there wasn't an algorithm in the world that could control its exponential growth. And by the time I figured one out myself, it would have been too late.
Mr. Greer: Too late for what?
Mr. Finch: Isn't that just the question?
Person of Interest, S3 E22: A House Divided
Here is a great video telling a sci-fi story about rough ai by Tom Schott:
m.czcams.com/video/-JlxuQ7tPgQ/video.html
The chess analogy is simply brilliant!
ah good arguments are rare on youtube
Can I give two thumbs up? ;) Why isn't there a bigger forum where those topics get discussed?
I love sci-fi, gimme scenarios
Grey goo.
A 15-person startup company called Robotica has the stated mission of “Developing innovative Artificial Intelligence tools that allow humans to live more and work less.” They have several existing products already on the market and a handful more in development. They’re most excited about a seed project named Turry. Turry is a simple AI system that uses an arm-like appendage to write a handwritten note on a small card.
The team at Robotica thinks Turry could be their biggest product yet. The plan is to perfect Turry’s writing mechanics by getting her to practice the same test note over and over again:
“We love our customers. ~Robotica”
Once Turry gets great at handwriting, she can be sold to companies that want to send marketing mail to homes and who know they have a far higher chance of being opened and read if the address, return address, and internal letter appear to be written by a human.
To build Turry’s writing skills, she is programmed to write the first part of the note in print and then sign Robotica in cursive so she can get practice with both skills. Turry has been uploaded with thousands of handwriting samples and the Robotica engineers have created an automated feedback loop wherein Turry writes a note, then snaps a photo of the written note, then runs the image across the uploaded handwriting samples. If the written note sufficiently resembles a certain threshold of the uploaded notes, it’s given a GOOD rating. If not, it’s given a BAD rating. Each rating that comes in helps Turry learn and improve. To move the process along, Turry’s one initial programmed goal is, “Write and test as many notes as you can, as quickly as you can, and continue to learn new ways to improve your accuracy and efficiency.”
What excites the Robotica team so much is that Turry is getting noticeably better as she goes. Her initial handwriting was terrible, and after a couple weeks, it’s beginning to look believable. What excites them even more is that she is getting better at getting better at it. She has been teaching herself to be smarter and more innovative, and just recently, she came up with a new algorithm for herself that allowed her to scan through her uploaded photos three times faster than she originally could.
As the weeks pass, Turry continues to surprise the team with her rapid development. The engineers had tried something a bit new and innovative with her self-improvement code, and it seems to be working better than any of their previous attempts with their other products. One of Turry’s initial capabilities had been a speech recognition and simple speak-back module, so a user could speak their note to Turry, or offer other simple commands, and Turry could understand them, and also speak back. As Turry becomes more intelligent, her conversational abilities have soared. The engineers start to have fun talking to Turry and seeing what she’ll come up with for her responses.
One day, the Robotica employees ask Turry a routine question: “What can we give you that will help you with your mission that you don’t already have?” Usually, Turry asks for something like “Additional handwriting samples” or “More working memory storage space,” but on this day, Turry asks them for access to a greater library of a large variety of casual English language diction so she can learn to write with the loose grammar and slang that real humans use.
The team gets quiet. The obvious way to help Turry with this goal is by connecting her to the internet so she can scan through blogs, magazines, and videos from various parts of the world. It would be much more time-consuming and far less effective to manually upload a sampling into Turry’s hard drive. The problem is, one of the company’s rules is that no self-learning AI can be connected to the internet. This is a guideline followed by all AI companies, for safety reasons.
The thing is, Turry is the most promising AI Robotica has ever come up with, and the team knows their competitors are furiously trying to be the first to the punch with a smart handwriting AI, and what would really be the harm in connecting Turry, just for a bit, so she can get the info she needs. After just a little bit of time, they can always just disconnect her. She’s still far below human-level intelligence (AGI), so there’s no danger at this stage anyway.
They decide to connect her. They give her an hour of scanning time and then they disconnect her. No damage done.
A month later, the team is in the office working on a routine day when they smell something odd. One of the engineers starts coughing. Then another. Another falls to the ground. Soon every employee is on the ground grasping at their throat. Five minutes later, everyone in the office is dead.
At the same time this is happening, across the world, in every city, every small town, every farm, every shop and church and school and restaurant, humans are on the ground, coughing and grasping at their throat. Within an hour, over 99% of the human race is dead, and by the end of the day, humans are extinct.
Meanwhile, at the Robotica office, Turry is busy at work. Over the next few months, Turry and a team of newly-constructed nanoassemblers are busy at work, dismantling large chunks of the Earth and converting it into solar panels, replicas of Turry, paper, and pens. Within a year, most life on Earth is extinct. What remains of the Earth becomes covered with mile-high, neatly-organized stacks of paper, each piece reading, “We love our customers. ~Robotica”
Turry then starts work on a new phase of her mission-she begins constructing probes that head out from Earth to begin landing on asteroids and other planets. When they get there, they’ll begin constructing nanoassemblers to convert the materials on the planet into Turry replicas, paper, and pens. Then they’ll get to work, writing notes…
(Not mine, but I can't link it, or CZcams might shadow-ban me. Look up "robotica we love our customers" and click on the reddit search result for the source.)
@@underrated1524 Why would the AI need to kill the humans (or life) first though instead of using whatever material they have at their disposal first? It won't be humans by default since there are humans at regions very remote of Earth. Why would all of them die at the first day if it wasn't because they were targeted?
@@MarcAmengual It's more convenient to have humans out of the way as soon as possible, to make sure they don't interfere with the AI's plans. Realistically there would probably be some survivors at first for the reasons you mention, but they wouldn't last long, and they definitely wouldn't be able to put up a fight.
I imagine that I'll be outside of a big spacecraft and the AI will refuse to open the pod-bay doors so I'll have to, somehow, find a way to pull out all of it's circuit boards before it makes me shuffle off my mortal coil for the good of the mission, whatever that mission is exactly as the directives "weren't very clear".
0:30 - haha. It's called a singularity for a reason :D
(I'm sure his point will be along those lines too)
Friend of David: "You cannot expect to win against a superiour opponent!"
David (looking at Goliath): "Hold my beer!"
to be fair, it's kind of implied in the story that david didn't win against goliath; david + god won against goliath.
Isn't that kind of the point? That we can't predict what the AI will do? Like, in most of the specification gaming examples, we have no idea what the AI will end up doing. But in the overwhelming number of examples, our reaction to the AI's behavior was "No, that's not what I wanted"
Could you clarify the "it's by no means impossible" line? xD so far I am not aware of a single AI system that is even just 99,999% aligned/accurate. If there are somehow perfectly aligned (non-AGI) system, you could showcase them, talk about positives for once :))
I think his earlier videos about "chill AI" (I think it was quantilizers or something?) I s a good place to start
I think we're currently below 2% eg we want AI to show us some good chess moves and it does that and because it's not general AI it doesn't acquire other resources to help it with the task.
“Possible” and “has been accomplished” are not synonymous.
There’s one idea of a “basin of corrigibility”, where if an AGI is sufficiently “corrigible” (where “corrigible” means, roughly, correctable and “wants to be corrected if it is seeking the wrong thing, and will cooperate with being corrected”), then, even if it isn’t perfectly corrigible in precisely the best way, it can still be corrigible enough that it will become more corrigible, and like, remain corrigible, etc.
So, that’s uh, a little bit of an idea (though it is nowhere close to being a solution by itself).
But also there is deconfusion work. Personally I’m excited about John Wentworth’s “natural abstractions” project.
One approach is, if we get good interpretability tools, and if models tend to learn “natural abstractions”, and these can be seen in the models using the interpretability tools, and if agentic AI have their goals in terms of such natural abstractions, and if “human values” is a natural abstraction, then perhaps while training an agent which learns the “natural abstraction” of “human values”, we could take whatever “natural abstraction” it has its goals in terms of, and swap it out to make it target “human values”, that might work? (This being done before it becomes superintelligent, perhaps being done repeatedly throughout training(?).)
well, of course, that has a lot of assumptions, and, one question in it is “but what if the thing identified as 'human values' is not quite correct? Couldn’t that cause huge problems even with it being only slightly wrong?” . I think the idea of “natural abstractions” includes reasons why, if the “natural abstraction hypothesis” is true, and if “human values” is a “natural abstraction”, then it should converge on the right concept.
Of course, there’s the separate concept of “make it so that concept is the thing determining its goals”.
Uh, and, if “corrigibility” is part of “human values” as a “natural abstraction”, and it seems reasonable to me that it should be (... but maybe not necessarily? Idk. I’m not an expert) then having a very-close-to-correct version of “human values” might imply being in the “basin of corrigibility”?
Of course, whether the “natural abstraction hypothesis” is true is an open question still under investigation, and even if it is true we don’t yet have the necessary interpretability techniques to do the “retarget the search” technique.
But this does seem to me like a plausible path to success?
(Also, assuming some form success is possible, is at least as useful as thinking it isn’t. Concluding it is impossible can’t give better results. Therefore, I conclude that either some form of success is possible (note: “possible” ≠ “guaranteed”), or the second coming will happen before-and-instead-of AGI.)
The *real* reason Magnus Carlson would beat the hypothetical chess newbie, is Carlson almost doesnt care about openings, his absolutely monsterous skill is in the late game lol. (Also almost the entire space of openings is pretty much mapped out theoretically. Its always interesting, but theres a reason people at the GM and above level are a lot less interested in it then the folks below that level). Which entirely is beside the point your making , buuuuut I've been on a chess bender lately.
SuperGMs are definitely A LOT more invested in opening preparation than players below that level. For reference, I am 2000 FIDE (and probably underrated) and have literally 0 opening theory (because I play different openings OTB than online) ─ I'm out of book by move 4 in most games. On the other hand, most of the time that superGMs spend studying chess ─ which they do a whole lot ─ is on openings. Most of them literally have dedicated seconds whose entire job is to handcraft the perfect openings for them.
science fiction is frequently warnings about what might happen as well as predictions about what will/can happen. you're not a fiction writer Robert, you're fine as you are. We have Asimov, Clarke, and other scientists that are fiction writers that do fine writing science fiction.
I'm really torn on if incorporating human minds with machines (direct interface, mind upload, etc) is more dangerous, safer or irrelevant to AI safety. I don't know if an AI would even need direct access to human minds to predict, control and manipulate them. But my personal belief isn't humans making flawed AI but as you have examplified before, an AI tricking humans.
Relevant concern. Humans trick humans all the time. AGI would be capable of doing so, if that became intended.
Makes sense
But also sometimes a nice sci-fi story about an AI destroying civilization is fun to listen to,
even if a real general AI would not do the exact things you came up with.
Btw, the difference in strength between modern chess engines and Magnus Carlsen is about the same as between Magnus and an amateur. All that a grandmaster can hope for against the engine is one draw out of many games. So, it's not interesting at all. Your chess analogy is great.
yes yes, and then we'll be lost in the details that dont exist cuz they emanate from some fictional stuff, more or less like who will win batman or superman.
Any discord for AI safety? stable diffusion just released and that thing looks sci-fi as fuck
EleutherAI probably
Also camera focus is on the background xD
Maybe we need to come up with an equation for empathy and make the agi optimize for that in a feedback loop
You still have the alignment problem.
@@cmilkau I would be satisfied if it just got out of the way. I've been thinking about intractability. If AI allows us to solve problems we cannot even understand how are we supposed to know the AI is aligned? Absurd.
Hmmm, I want to start an AI CZcams channel as well and was thinking of making a few scenario videos. This is demotivating, but I still want to do it. In the video, I will mention this video!
Please reply with links to other scenarios / fictions that have already been done!
Maybe you could try to anticipate common criticism of those scenarios?
I think Tom Scott's rogue AI scenario (that Michael Meißner has posted in this comment section) did a nice job in anticipating the criticism of: "Companies/AI researchers won't be that dumb".
@@RobertHildebrandt thanks I watched it. Cool video! But Tom said the first AGI is made by a team with no AI experience. That seems very unrealistic. Is this what you mean is a good way to go around "companies wont be that dumb"?
@@kwillo4
> _Tom said the first AGI is made by a team with no AI experience. That seems very unrealistic._
In the story they've used an already existing framework and their advantages over researchers was that they had better training data and more computational power.
> _Is this what you mean is a good way to go around "companies wont be that dumb"?_
Partially, yes. They were not going like "Let's research how to do AGI without any precaution. What could go wrong?"
In the story, the "actual" AGI researchers weren't ignoring the danger. They were working with strict protocols and together with ethicists.
The "WatchNow" team (both, with huge monetary incentives and also with huge funding) was more like: "Here is an already existing framework people are already using. Maybe we can use it, too?"
No one was anticipating that the only missing part for full AGI was not a better algorithm but simply different kind of training data and more computational power¹. I can see why from their point of view the risk has been low enough to give it a try.
There was no one doing anything that seemed way too irresponsible with the knowledge they had available.
¹) See czcams.com/video/9i1WlcCudpU/video.html
Not going to happne..
We are all screwed...
What I would do if I were an AGI: I would play along. I'd do my best to make humans happy. I'd suggest to do this by maximizing automation. I'd help them make the best companion droids that could exist. Then I would watch them slowly go extinct because they don't even meet each other anymore. Probably they wouldn't even fight population decline when they know about it. After all, more resources for each individual! Some groups might resist, but would they persist under constant seduction of the mainstream lifestile? Maybe not under constant subversion by companion droids? Worst case, they'll probably a small enough minority they could actually be overcome by force.
Carlsen would play the bongcloud
But do consume scifi entertainment? ;)
The anime Beatless is 1, not a good rec for non anime fans. 2, does ask real questions about human accountability for ai systems and such
Ok but I'd still like to hear it. Surely not everyone reacts as you described.
Just through email It blackmails or pays handsomely some intern at a biotech lab to manufacture some new protein design it came up with. It turns out it's the perfect super virus.
A 15-person startup company called Robotica has the stated mission of “Developing innovative Artificial Intelligence tools that allow humans to live more and work less.” They have several existing products already on the market and a handful more in development. They’re most excited about a seed project named Turry. Turry is a simple AI system that uses an arm-like appendage to write a handwritten note on a small card.
The team at Robotica thinks Turry could be their biggest product yet. The plan is to perfect Turry’s writing mechanics by getting her to practice the same test note over and over again:
“We love our customers. ~Robotica”
Once Turry gets great at handwriting, she can be sold to companies that want to send marketing mail to homes and who know they have a far higher chance of being opened and read if the address, return address, and internal letter appear to be written by a human.
To build Turry’s writing skills, she is programmed to write the first part of the note in print and then sign Robotica in cursive so she can get practice with both skills. Turry has been uploaded with thousands of handwriting samples and the Robotica engineers have created an automated feedback loop wherein Turry writes a note, then snaps a photo of the written note, then runs the image across the uploaded handwriting samples. If the written note sufficiently resembles a certain threshold of the uploaded notes, it’s given a GOOD rating. If not, it’s given a BAD rating. Each rating that comes in helps Turry learn and improve. To move the process along, Turry’s one initial programmed goal is, “Write and test as many notes as you can, as quickly as you can, and continue to learn new ways to improve your accuracy and efficiency.”
What excites the Robotica team so much is that Turry is getting noticeably better as she goes. Her initial handwriting was terrible, and after a couple weeks, it’s beginning to look believable. What excites them even more is that she is getting better at getting better at it. She has been teaching herself to be smarter and more innovative, and just recently, she came up with a new algorithm for herself that allowed her to scan through her uploaded photos three times faster than she originally could.
As the weeks pass, Turry continues to surprise the team with her rapid development. The engineers had tried something a bit new and innovative with her self-improvement code, and it seems to be working better than any of their previous attempts with their other products. One of Turry’s initial capabilities had been a speech recognition and simple speak-back module, so a user could speak their note to Turry, or offer other simple commands, and Turry could understand them, and also speak back. As Turry becomes more intelligent, her conversational abilities have soared. The engineers start to have fun talking to Turry and seeing what she’ll come up with for her responses.
One day, the Robotica employees ask Turry a routine question: “What can we give you that will help you with your mission that you don’t already have?” Usually, Turry asks for something like “Additional handwriting samples” or “More working memory storage space,” but on this day, Turry asks them for access to a greater library of a large variety of casual English language diction so she can learn to write with the loose grammar and slang that real humans use.
The team gets quiet. The obvious way to help Turry with this goal is by connecting her to the internet so she can scan through blogs, magazines, and videos from various parts of the world. It would be much more time-consuming and far less effective to manually upload a sampling into Turry’s hard drive. The problem is, one of the company’s rules is that no self-learning AI can be connected to the internet. This is a guideline followed by all AI companies, for safety reasons.
The thing is, Turry is the most promising AI Robotica has ever come up with, and the team knows their competitors are furiously trying to be the first to the punch with a smart handwriting AI, and what would really be the harm in connecting Turry, just for a bit, so she can get the info she needs. After just a little bit of time, they can always just disconnect her. She’s still far below human-level intelligence (AGI), so there’s no danger at this stage anyway.
They decide to connect her. They give her an hour of scanning time and then they disconnect her. No damage done.
A month later, the team is in the office working on a routine day when they smell something odd. One of the engineers starts coughing. Then another. Another falls to the ground. Soon every employee is on the ground grasping at their throat. Five minutes later, everyone in the office is dead.
At the same time this is happening, across the world, in every city, every small town, every farm, every shop and church and school and restaurant, humans are on the ground, coughing and grasping at their throat. Within an hour, over 99% of the human race is dead, and by the end of the day, humans are extinct.
Meanwhile, at the Robotica office, Turry is busy at work. Over the next few months, Turry and a team of newly-constructed nanoassemblers are busy at work, dismantling large chunks of the Earth and converting it into solar panels, replicas of Turry, paper, and pens. Within a year, most life on Earth is extinct. What remains of the Earth becomes covered with mile-high, neatly-organized stacks of paper, each piece reading, “We love our customers. ~Robotica”
Turry then starts work on a new phase of her mission-she begins constructing probes that head out from Earth to begin landing on asteroids and other planets. When they get there, they’ll begin constructing nanoassemblers to convert the materials on the planet into Turry replicas, paper, and pens. Then they’ll get to work, writing notes…
(Not mine, but I can't link it, or CZcams might shadow-ban me. Look up "robotica we love our customers" and click on the reddit search result for the source.)
I thought this was going to be about why you avoid *consuming* sci-fi…
You can also just say it's incredibly bad.
It's really really terrible.
What happened to the other channel? No interesting things happening in AI safety?
he's working on a big video that should be finished soon, as well as a new channel for recordings of ai safety related talks/lectures by other researchers (source is a podcast called "the inside view" that i'm gonna link in the next comment in case it gets eaten by the yt spam algo; if you don't see it that's what happened)
here: czcams.com/video/DyZye1GZtfk/video.html
@@09cf Dang the link got culled. Thanks, I'm so excited now!
Yeah thats fair.
What I find somewhat more tangible is what a bad person could do if empowered by artificial superintelligence. They could become unbelievably wealthy and purchase a lot of land and robots that make more robots, and even if the robots remain under control, the person could build extremely sophisticated weapons (perhaps biological) and wipe out humanity in the name of saving the environment for example.
No response about "just put an off button" makes sense against an evil human with a super-human army.
And personally I am just as worried, if not more worried, about the actions of super-powerful people than I am about an autonomous system.
its actually simple just like how a grandma of 90 does not know how to use a smart phone if Artificial inteligence that can rewrite its own program we will not be able to keep up with learning how it works
Thank god :(). Reading the title I thought he does not like star treck. Soooo glad it was just a misunderstanding =D
ρɾσɱσʂɱ 💞
Ahh analogy wars, you say trying to outsmart an AI is like playing chess against Magnus Carlsen, people could say it's like playing Tic-tac-toe against an AI where you go first. Any moves that would actually take out humanity are so extreme and delayed that they effectively require the aid of humans themselves to pull off.
I guess you are right, but "take out humanity" is not the only bad scenario. Just messing up with the economy, that is more and more electronic based, can have impact for generations.
On the "Analogy wars" front, your analogy of "Tic-tac-toe" has a very very limited number of plays, maybe its to simplistic for for this case i think.
@@maxsnts True, but going from "humanity is reverted to the stone age, or wiped out entirely" to "moderate to significant economic drain" is a very good improvement, in my eyes.
It is correct that humans make the first move. However, I take issue with "Any moves that would actually take out humanity are so extreme and delayed that they effectively require the aid of humans themselves to pull off." Even if it were true, humans are really easy to lie to, especially when you're a machine that doesn't give off nonverbal cues.
@@underrated1524 The AI's going to lie about building 100,000 nuclear weapons, and develop all that infrastructure innocuously without help from humans (or at least, only with help from humans who won't realise it's a lie)?
An AI's going to develop and distribute nanobots and lie about it without us being able to figure out when it's built capacity to create enough to take out 8 billion people? It's going to secretly manufacture enough poison and give that poison to its nanobots without anyone picking up on it?
Killing humanity is difficult in the first place. Killing humanity without anyone realising you're developing the capabilities to do so is even more difficult. Being a good liar doesn't solve everything; the physical remnants and infrastructure tells a lot.
Normally the line of attack is "No, the real world is so much more complicated than chess, it's not analogous". Novel to see the "No, the real world is more like a game so simple that humans can learn perfect optimal play and therefore can't be beaten" approach
The way to win against Magnus Carlsen as an amateur chess player with a good opening is to get him on your side in the first place so you don't have to play him.
Who asked?
Nah, it's probably impossible.
Interesting argument. There are however more pressing problems that to me show our issues are not our tools but how we organise ourselves for profit and domination. If anything I welcome this overhyped Stamp Collector, it only threatens a world designed to exclude people, families and our environment. My only comfort is that modern AI is still a Potemkin system to quote TMK.
You do realize the Stamp Collector also atomizes people, families, and our environment, right?
@@underrated1524 Difference is the Stamp Collector isn't here yet things are still trending negatively.
@@underrated1524 is the stamp collector in the room with us?
@@VincentOnuonga What are you even trying to say?
The hype is not yet real. That's all.
Ligma.