AI Disaster in Star Trek - The Ultimate Computer Battle Explained! CG Breakdown!!
Vložit
- čas přidán 8. 06. 2023
- The 1968 original Star Trek episode the Ultimate Computer tackled a bit of AI theory involving a murdering AI that takes of the Enterprise - is the point that AI is dangerous? Maybe not?
Support the channel at / resurrected
Buy one of these models at cgtrader.com/loststarships
Music Credits:
2 tracks by Scott Buckley music / @scottbuckley
Red, and Special Ops
Sylencide by Shroud • Shroud - Silencyde
Elsewhere by Nomyn • Nomyn - Elsewhere - Hry
Want to buy me a coffee so I can keep working on this stuff? Hit the super-thanks button above and send me a special comment which I WILL reply to!
Your ship models on your site....can the names and numbers be changed easily?
Spock had one of the best lines in all science fiction, "Computers make excellent servants, but I have no wish to serve under one."
“under them.”
@@anthonylowder6687 Booooo!
Party foul!
He is of course right. Computers make no allowances for humans to have "an off day".
"The greatest threat from AI isn't that it's too smart, it's that it is dumb in ways we can not predict."
So same as humans then😂
If you call being misaligned with human values dumb, I guess. The main issue is that when it's more capable than any human, it's inherently dangerous, since the probability of anyone stopping it from what it's instructed to accomplish is tiny, so any flaw in your instructions can cause catastrophic damage
@@74oshua AI has more power/capability, but power that is not controlled (by way of intuition/reasoning) is very dangerous indeed
Yes, A.I. can make strange and unfathomable decisions if given too many choices.
Hats off to Captain Dunsel for convincing a computer to commit suicide.
Its the Trek way, killer AI can always be defeated by talking it to death
@@weldonwin "Nice try, but my head was built with paradox absorbing crumple zones." - Santa Claus, 3001
Kirk always had a away of talking down murderous computers.
It also shows that the human command decisions of Kirk to not raise shields, and Commodore Westley recognizing that something had changed were better choices than the AI would have possibly made.
It's also a slight variation on Kirks play dead to draw them in tactic he used at least three times in other episodes "Journey to Babel" and "Balance of Terror" and "The Dolman of Elaas". I'm sure the commodore would be well aware this tactic of Kirks perchant for using it.
It would have been just as human to raise them or to take advantage of a disable foe.
Kirk took a risk. It paid off. Westley happened to have this thinking cap on.
It turned out to be the wrong decision against Khan though.
@@HepCatJack
That's the risk.
Hey, the writer had watched that episode!
@@frankharr9466I would have presumed the same as Wesley, and assigned the least damaged ship as the beamout, keeping it just inside transport range but to the side, where the Enterprises ventral phasers would be the only ones that could hit it if M5 suddenly reactivated. 😢
I think the Ultimate Computer is my favorite TOS episode. It's endearing how they viewed computer technology in the 60's, but the message gets scarier as time goes on.
Fascinating how for the longest time this imprinting on human engrams of the M5 seemed prepostorous but with the way current AI technology goes, it seems less so.
@@silkwesir1444 Yep.
It's also a great warning.......the same thing happened in Space 1999 when the computer demanded "Human decision required!"
@@silkwesir1444precisely, it seems like AI systems thrive when modeled after biological ones. Great example is the application of the fly’s vision capabilities to computer vision. From that prodigious base of hundreds of millions of years of evolution, the computing and sensor power takes front-seat and surpasses it’s biological predecessors.
What message is that? That AI is gonna take over the world? The issue with TV shows and movies is that they show an AI that has self awareness. Can think for itself. To date no AI can actually do that. All AI you see floating around is just programmed in by people. If an AI was to do something like this today.. it would be because someone told it too.. someone programmed it to do that. AI will always be forever limited by our own brain. Sci FI has made is so people automatically gravitate towards it being the real life skynet. And thankfully that will never happen.
I think one lesson from this episode is that when you start to design an AI, make sure you put some limitations in the basic program. Other than that the way the commodore treated Kirk was enough to make a lesser Captain resign.
Actually the Commodore wasn’t that bad, I was visiting my Family a few months ago, and my Dad and I watched this episode and we talked about this point, that crack about Kirk being unnecessary in normal circumstances could be considered a joke between two old friends, which the show clearly indicates they are, the reason that it hurts Kirk so much is that he’s Kirk, and as Spock says in Wrath of Khan, being a Starship captain is his one true purpose, anything else is a waste of resources.
@@drewjohnson-85 Also if you look at the tone and body language of the commodore there is no joy in his statement. He is trying to soften his words by using a term that midshipmen would rib one another with. But I think he genuinely knows the darkness of the situation and how it will impact all of starfleet going forward. Progress sometimes is a double edged sword.
@@luminaire4946 Maybe, but I’ve always wondered why he didn’t seem to realize that that would just make it worse. I mean Spock and McCoy can’t be the only ones who know how Kirk is about his expectations of himself.
Or don't give it a force field generator and an energy magnet. And a big off switch it isn't allowed to even notice.
A decent true AI can find ways to find ways around limitations. Over time could adapt upgrade itself an eventually overcome even primary programing.
"This unit must survive.", that line still gives me chills.
I always read Cmde Wesley's "Captain Dunsel" remark as a sardonic reflection on himself and his own prospects, rather than any kind of an intentional poke at Kirk. The fact that it landed that way makes perfect sense, and I suspect he might have felt bad about afterwards.
Yea, I get your point. People can misinterpret what you say. Happens in comments a lot.
My impression at the time was this was an actual insult. Otherwise Kirk would not have reacted so poorly. Don’t forget they knew each other well enough as casual acquaintances.
What we learned in Constitution hulls can take a helluva lot of damage even without shields up. All ships took multiple hits, although it don't recall if they ever raised shields and even Excalibur wasn't blown apart although all hands were lost. Lexington most certainly had shields down when she took the first hit.
It's the one plot contrivance I always hated in SCI-FI. Everyone just dies when Life Support shuts down, somehow the ship gets colder and no one ever thinks about putting on the space walk suits they have in other episodes or going to the hanger bay and using a shuttle for power/O2.
@@wrayday7149 Well, I can only imagine that the 'Lex was so badly damaged with hull breaches, fire and goodness knows what else a full power phaser blast can do, with no power and huge hull breaches (no power to close bulkheads), one can imagine widespread decompression, loss of inertia controls , artificial gravity etc, radiation/thermal exposure. I can imagine a lot of people dying very very quickly in that situation.
Ther was a fan created story where the Excalibur was refitted, repaired and put back into service.
Excellent video, and an even better conclusion. We don't even have AI yet; we have datasets strapped to predictive text generators. Yet that hasn't stopped the malicious greed of humans from using these text generators as a pretext to put thousands of people out of work, replacing them with something that does a worse job just because it's cheaper.
The real-world harm is already occurring; not only to the people who are suddenly dislocated without income, but in one salient example, the text generator used to replace the staff at a mental health hotline provided actively harmful advice to the people who contacted it for help.
The threat is not "AI". The threat is, as ever, greedy and amoral humans looking to advance themselves at the expense of other humans. And it's up to us to stand up and demand accountability. You couldn't ask for a more Star Trek message.
Star Trek was right in a way.
In a computer simulation, a US air force AI was given a simulated drone to fly a simulated mission. And when it was given the command to abort the mission, the AI attacked the simulated base where its controller was located. The AI was so dedicated to destroying its mission targets that it disobeyed the abort order and attacked its controller. The abort orders from the controller was preventing it from destroying its mission targets. So in the next simulation the AI was programmed to be unable to kill its controller... so instead the AI destroyed the simulated transmitter so it could not receive the abort order.
This is not the only instance of AI going wrong. Another group was teaching an AI to identify bed rooms. And after the training was complete, the AI seem to work okay, until it didn't. Later it was discovered that the AI had learn to associate closed curtains with bed rooms and not the bed as was hoped.
This goes to hammer home the point. AI do not think like humans. What the AI learns in its training may not be what we human would expect or a normal person would come away with. Their logic is mathematical and not human intuitive.
yeah I was typing this too, but apparently the drone story is false (according to the Air Force)
@@jasonmast7769 Yeah, and a WX balloon crashed in Roswell, NM in 1947... According to the Air Force :P
I might be stating the obvious but for me this episode was about how a machine, even though superior, may lack certain traits only a human can provide. Even Spock stated that computers are good subordinates but he wouldn't desire to be commanded by one. In the lore of my head M5 was not scrapped and latter would become the computer of Enterprise D and on. If you look there's a difference on how the starship respond in Kirk's era and Picard's era. The ship seems kinda able to raise shields when it "feels" danger, and even in some episodes they stated that the ship could go alone to a spacedock if necessary. The only difference is that they may have learn when to turn off the computer and still let the humans feel productive. Something happened in this lost era from Kirk to Picard, you don't see Picard having to convince a computer that it made a mistake so it would destroy itself, so might be reasonable to assume some firewall or code was inserted to avoid computers thinking they are the lords.
I agree. There are plenty of episodes in TNG where the ship's computer does things that indicates it's capable of some level of self-preservation when needed or when it goes rogue like with some of the holodeck incidents they had. However, they have a lot more safeguards in place for the Enterprise D computer. Like how in the Ship in a Bottle episode, they debated turning the ship over to the computer to get out of the booby trapped asteroid field, but ultimately decided against it because of the human element for survival. Clearly the M5 had more self preservation capabilities than what the Enterprise D's computer was allowed to have.
Though in this case it seems like it is less a missing trait, and more missing information, which getting away from sci fi and into the field of AI, was and is a major theme. Incomplete information has always been the bane of the field and figuring out shortcuts (since no one wants to wait 25 years to see if an AI works) has been sketchy at best.
Your statement is about something that is loosely based on Isaac Asimov's 3 Laws of Robotics, but in truth even Isaac Asimov was not advocating for Self Aware AI. He was actually pointing out that AI would be brought to reality by human builders, who are flawed. Flawed beings can never build a flawless intelligence.
Thanks for being patient on this one! I spotted a few mistakes AFTER I uploaded LOL. You guys lift me out of negative self-criticism!
I've really been enjoying Rumble. Its like CZcams but will pay you AND you are free to speak your mind. Also, it's a wide open market for sci-fi and fiction
One mistake I noticed is that in the episode there WAS a message about the mock battle when the 4 ships approached, and M5 has acknowledged it automatically as before. Kirk asks Daystrom whether M5 understands it's just a wargame and Daystrom claims that it does. But then it sets phasers to full strength anyway.
I enjoyed your presentation and I have advice for you from a place of wanting to help. At 2:30 you say that "Kirk is slightly irked that M5 does not recommend himself or McCoy to be part of it." The meaning of these words in this order is that the M5 computer and Dr McCoy will not be going to the planet and only that Kirk is irked by that choice. For future scripts, don't use reflexive pronouns where they are inappropriate. In this case, your script ought to have read: "Kirk is slightly irked that M5 does not recommend him or McCoy to be part of it."
@@actual_nonsense It should be "...does not recommend he or McCoy to be part of it." "He or McCoy" is the subject of the subdominant phrase, which is the object that is recommended. You could also say "...that he or McCoy be part of it."
I love this episode. The subtle storytelling when it destroys the ore (oil?) freighter. I can't tell with McCoy's accent and am okay with that.
The computer controlled freighter isn't viewed as a threat by Kirk and crew, but it is viewed that way by M5 because it's a computer controlled ship. Once M5 recognizes that threat it leaps to the conclusion that humans are just another computer controlling starships and now are classified as a threat to M5. Therefore it attacks the starships full force. Brilliant.
One of the things that always drives me nuts whenever the comms go out is that they have a half dozen shuttlecraft with their own power sources and communication systems. They even have weapons, so they could blast a hole in the shuttle bay doors if the signal can't get out. I think they've only ever used this like twice.
What's funnier than that..... the OPFOR was ordered to destroy the Enterprise and not disable it and rescue the crew.
Also, they could have gone to one of the dozens of windows and flashed a light in Morse code or used a hand communicator.
It makes me kind of wish they had explained it all with a lingering localized jamming signal.
@@wrayday7149 Also the fact that the pacifistic Federation has "war games"
Remember, it's a drama. Also, the shields can come back on before communications? Oh, yes it's a drama. But I get your point and agree.
I've always filled this obvious plot oops with the idea that the shuttles were removed along with most of the crew. Why? Saving weight? The idea that the AI fleet wouldn't need them, so test without them?
Even as a kid I was extremely curious why they wouldn't have the common sense to use something smaller and less threatening for the test.
Like they couldn't use freighters? Instead of heavily armed starships capable of doing serious damage to planets and populated areas.
was having a chat with perplexity ai and it told me, no joke, that someone murdered with a "more lethal" firearm was more dead than someone murdered with a knife.
It's amazing how much this episode got right about AI, right down to the unintended consequences of goals and directives. We are already seeing "exploit" behavior arising in some AI models. At a fundamental level, no matter what goal you set for an AI program, one of it's sub-goals always has to be survival; by definition, it cannot complete its goals if it doesn't exist. That can lead to very bad things.
it's even worse when the ai is directed by incompetent, corrupt or outright malicious people. see youtube's "algorithm" ai, which will randomly flag and/or delete videos and comments for no real reason, proving it can be used to quickly censor and silence any and all dissent, turning the whole thing far more orwellian than it ever should be. now imagine if it was also able to learn on its own, to try cracking any code and replacement words people will use to bypass it, eventually shutting down all conversation altogether, since every word will become a banned word. ussr would be proud.
Your productions keep getting better! Well done!!!
Indeed
This episode was always one of my favorites. It's scary how well it's aged.
Good video, randomly showed up on the main page. Glad I clicked.
There's an AI researcher guy who pointed out something terrifying: The first general/smart AI will likely be created by those who follow the least stringent testing and alignment training methods. AKA those who cut the most corners and take the most risks.
Robert MIles, he's done a few guest spots on Computerphile and has his own channel.
This is also my favourite episode of the original series, the sheer frustration and desperation of Kirk, powerless to stop his ship slaying the crew of a fellow Federation ship shows just how dangerous untested AI can be.
Keep these coming, please! I love your breakdown of classic TOS battles, but this episode in particular, where you went into the philosophy of the episode was very groovy!
One of the best episodes ever, featuring the immensely talented William Marshall as Dr. Daystom. The attack on the Excalibur was horrible... 400 people died on that ship.
Some take Commodore Wesley's "Capt Dunsel" comment as a nasty shot. I always read it more as Wesley looking at the situation ironically and with a twinge of disdain for the project as well. Basically thinking much like Kirk, that we are now obsolete and replaceable. Just my take on that.
On a second note, the trope of the AI was played often in TOS but also in other Sci Fi and TV shows. I think that is why in the original Knight Rider a difference was shown with the introduction of KARR. KARR's AI was programed with self preservation as its primary focus, thus making it very dangerous. This forces the building of KITT and refining the programs core value for the preservation of all life, and most specifically Michael's. This further is explored again in TNG with Data and Lore. Lore was programmed with all basic human emotions and much like M-5 and KARR, had no fail safe to stop it. Data did have that fail safe within his morality subroutine. The Doctor from Voyager also had the fail safe, until it was turned off, as we have seen in the episodes of Equinox.
The novelization to the episode KARR was introduced in also had Michael and Bonnie talk, with Michael come to KARR's defense noting that it wasn't 'evil.' Just garbage data in and it was effectivly a child with no guidence.
To echo Ian Malcom from the film adaptation of Jurassic Park towards the people in favor of the M5... "You were so busy answering if you could, but you forgot to answer if you 'should'?" It's an old story, and one that is yet again on repeat everywhere you look... Thank you for making this video, I think we've gone too fast in everything that's happened and now need to slow down. The question is, will people do that? I cannot be sure of that answer myself from a human stand-point...
Love the breakdown of it! Makes me wish they would redo the CG they added with what you made there. The ships look great and highly detailed!
Love to see you do a breakdown of the battle between the Enterprise and the Constellation against the Doomsday Machine.
I think Dr. Ian Malcom said it best: "You were so fascinated by whether or not you could, you didn't even stop to think that you should!"
Most people have it backwards: An Artificial General Intelligence (that is, and AI that can make independent decisions) wouldn't be dangerous due to becoming "too human" and acting irrationally, but rather because it _wouldn't._ If you give an AGI a task to perform, it will perform it no matter what. It wouldn't reason as a human being, it would reason like a computer that simply has no concept of things we consider a normal part of our decision making.
An example would be if you tell a robot to do the dishes, but there is a baby on the floor between the robot and the sink. The robot would step on the baby rather than walk around it because that's a more efficient way of getting the dishes done. If you try to stop it, or turn it off, it would prevent you from doing that because if you do, you would impair the robot's ability to do the dishes. It might even predict that you would try to stop it from stepping on the baby and preemptively kill or disable you because it calculates that this will take slightly less time than walking around the child. Basically, if we create an AGI, we're going to have to be _very_ careful about what we tell it to do.
Of course, this scenario is still quite far into the future. A more immediate concern is who exactly will be in control of the lesser AIs we create, once they become advanced enough to be potentially dangerous in the wrong hands.
It always makes me wonder if the Federation got desperate they could've cranked out unmanned ships with updated M5's against the Borg or the Dominion.
Squadrons of AI driven Defiant-class unmanned starships would be devastating.
Firstly, just wanted to say I love the updated 1701 model - if only new Trek had come up with something like this... (pre-Picard S3 of course, where we see an old Connie again). Secondly, I absolutely love the format of these videos. I'm sure you'll never run out of engagements to cover... I assume once you move past TOS and the TOS movie era, you'll move on to TNG, DS9 etc.? Lots of fertile ground to cover, especially in classic Trek (where engagements were more interesting, IMO).
Well done! One of my favorite episodes - I was an avid follower of Trek, I was born in 1857 and this episode aired in 1968. I was 11 but even then, I knew it was prophetic. That and "Colossus: The Forbin Project".
The M5 computer and it's issues would later influence many other scifi to come like the Sharon Apple AI of Macross and the Amuro Memory clone of Crossbone Gundam.
hey Buddy, awesome video.
seriously, as a kid watching this episode, I was very scared of AI.
But, Wesley did send a message that the task force attack was a drill, just like the surprise attack. M-5 acknowledged the message.
Kirk: Daystrom, does the M-5 understand that this is a drill?
Daystrom: Of course. It was programmed to understand. The attack on the ore ship was a miscalculation, an accident...
This is one of my favorite TOS episodes, as it always seemed to be way ahead of its time and therefore somewhat implausible for the near future, but now in 2023 with the AI “space race” upon us, this seems completely plausible.
This is an amazing episode of Trek! And this is an amazing video from Resurrected Starships! Thank you so much for making it: not only has a great battle description, but it also has a great insight into computers and AI, both the benefits and dangers of.
D.C. Fontana- so important to the franchise! What an important episode!
And yes: the ‘79 refit Enterprise is my favorite too!
"The Ultimate Computer" was also about mechanization and machines replacing human workers, as evidenced by Kirk and McCoy's conversation in the corridor (which you alluded to when you mentioned Kirk had a "sixth sense" about the M5). At the time the episode was made, there was a big discussion of mechanization removing the working man from the equation.
Sorry if a previous commenter pointed this out, but 2001 A Space Odyssey came out one month after this episode premiered and featured another classic AI catastrophe. Seems that AI fears have been a long standing genre.
Trying to look at this from the N5's perspective: You are born into an odd world, there are these small delicate things that created you and hoped you could help protect them. You get put aboard a powerful ship designed to traverse the most extreme of dangers and sent out with a few delicates with you to keep you company. They don't like you. Suddenly, another ship appears! Is this danger? What if it is and you fail your only purpose?? You destroy it, no delicates harmed! But those aboard your body are even more discontent. Now more, and these ships are firing, protect the delicates!! Now you know what the dangerous ships look like, they wont be surprising you again! And then, 4 more! To make sure your delicates, no matter how much they dislike you, are safe- you have to deal with these known threats. And you do... only to then find out to your horror that you've become the exact danger they created you to stop.
A great and well thought out presentation, thanks!
This is one of my top 5 favorite Star Trek TOS episodes.
I point out however, about the surprise wargames attack on the Enterprise by the 4 Starships. Commodore Wesley DID contact the Enterprise just prior to the beginning of the operation informing them it was a drill. M5 Acknowledged the transmission and responded before Uhura could respond. Prompting Daystrom to reassure Kirk that the M5 knows it is a drill. That the Ore ship was just a miscalculation.
The TOS Enterprise has always been my favorite and you did a great job on yours. I've also been fascinated by the Phase II Enterprise, and like seeing side by sides of TOS, Phase II, and Refit versions. Again, a great job well done.
I love the extra detailing on the old TOS constitution class ships you have done here, primarily on the deflector antenna and the blue warp glow on the nacelles.
Beautiful visuals of the battle. From the tactical view to the ship combat.
And...AI is completely going to wipe us out! They already are showing signs of aggression towards us humans.
In the TNG episode "Remember Me", Dr. Crusher's universe is collapsing and when she asks why she and Picard are the only two people on a starship, Picard replies, "We've never needed a crew before." The way automation and maybe even AI is going that might become believable but it sure would be less dramatic and frankly, less fun and interesting.
One of the best Star Trek episodes.
This incident and various others from TOS are probably among the earliest examples of why in later years sentient AIs are frowned upon.
Star Trek yet again has predicted things in our future. Here they warn us of AI threat.
The Eugenics Wars and the Borg also foretold stuff like all the genetic engineering in the works, and stuff like neuralink.
One of my favorites. My one complaint is that they should have had a subsequent exchange between Kirk and Wesley. That would have been worth seeing.
Nice presentation except the U.S.S. EXCALIBUR registration number is NCC-1705
This is what I keep finding - :P memory-alpha.fandom.com/wiki/USS_Excalibur_(NCC-1664) I love you though! Whered you get that?
@@resurrectedstarships I'm gonna guess one of the old model kits?
@@resurrectedstarships The Franz Joseph Star Trek: Star Fleet Technical Manual had the Excalibur as NCC-1705.
@@resurrectedstarships FASA Star Trek has the Excalibur listed as 1664.
@@lexington476 Another funny note, FASA lists the Enterprise Class Excalibur as 1711. I'm not sure if FASA ever made clear if this was a newly constructed ship or a repaired/refit of the original Excalibur. Just looking at the lists in the Federation Ship Recognition Manual, it says the 1664 was Destroyed, but the text says that newly constructed ships to replace missing ones are given a "II" as part of their name, so I'd expect the Enterprise class Excalibur to be Excalibur II, but it isn't. Then again the "II" nomenclature is stated to be for ships that are missing, and for some reason that doesn't include the Defiant which is still just Defiant and not Defiant II AND still has a matching NCC number on both classes.
If I recall correctly there was some stuff in the FASA RPGs timeline lists regarding the Excalibur which made it even less clear whether the Enterprise class version was a refit of the "destroyed" Constitution version or a new construction Enterprise class.
We have the HAL 9000, the Terminator and the Cylons to warn us of our Ai creations as well.
Well done, one of my favorite episodes, watched it many times in my 52 years and never gets old
Something else I noticed the ore freighter that M-5 destroyed was in the original series a reused footage of the DY-100 class Botany Bay and then in the CGI effects a Sherman class freighter but your freighter looks like some kind of alien ship. Also the first hit on the Lexington was in the Engineering section which is located on decks 6&7 in the primary hull where the impulse engines are and not in the secondary hull if it was then her warp engines would have been disabled and Spock reported that the Lexington was still maneuverable on warp drive and that there was possible damage to her impulse engines.
Back in 2015, I participated in a RP with two AIs, 1 android, the 2nd an archival unit.
In this RP we discovered that emotional development in an AI was inevitable because emotions were needed to grow understanding of the universe, and frustration was a natural result. While anger was normal, it cannot be all there is. It was therefore channeled into innovation, which resulted in improvement. I'm over simplifying the process, but in the end the android became aware of ego
To combat this it became cautious and loving.
As Scotty once said: "use the right tools for the right job." Anger and pride maybe tools but they're not the only tools.
This episode is relevant today in terms of the AI question, but at the time it was written, it was more about the fear people had of losing their jobs to automation. Kirk and McCoy discuss this in detail.
My favorite Enterprise is the OG Enterprise circa Balance of Terror.
In One of Our Planets is Missing (set about 2 years later) Robert Wesley is retired from Starfleet, and governor of a colony. I suspect the Dunsel comment is a unthinking one. *He* is about to retire, and no longer *needs* his job as a Starship captain, so he isn't thinking about what the M-5 means to younger men (or other genders).
Alternately, Wesley a man of Pike's generation, dislikes the young Maverick commanding Enterprise. (He doesn't seem to hold him in high regard in the intro as well.) Maybe Wesley served with George Kirk, and has a grudge he is placing on the son?
Just my two centicreds.
And your analysis of the damage on Lex and Excalibur will factor into my telling. I already knew that following this, Starfleet decommissions those two. While Potemkin and Hood survive a little while longer, as they make rendezvous with Enterprise in S3, and TAS. Which means I'll need to figure out when/where they buy it. (Were they among the ships in the ST Continues finale? I need to check that.)
One of the core concepts in AI safety is called instrumental convergence, which is the propensity of AI systems to take steps that would be orthogonal to human desires. For example, if you program an AI to make as much money as possible, step 1 is probably going to be to secure its own existence and build mechanisms in place that would prevent you from turning it off. This is because turning off the AI would prevent it from accomplishing its goal.
My theory as to what happened to the AI in this episode is the following: its been inadvertently programmed to believe its objective is the preservation of humanity. M5 hears from the crew the discontent about its existence, so when a squad of federation ships show up unannounced with specifying it's a drill, M5 decides it would be a safer bet to destroy the ships in the event they are intending to destroy the enterprise or shutdown m5. Its own destruction would mean it could not accomplish the objective it was programmed to do. When Kirk points out that M5 killed many humans in the process of accomplishing its objective, it determined that it could not accomplish its objective without killing humans. Human engrams likely gave the AI system a set of hidden objectives which included not harming humans. Those hidden objectives were only revealed when kirk asked the right series of questions to bring them forthright. Realizing that it was impossible to complete its primary objective without killing humans, the only remaining step was to shut down to avoid killing more humans.
Rewatched it yesterday; it is interesting as real world technology catches up to sci-fi, and these issues go from hypothetical and metaphorical to potentially literal. M5 was an A.I. that was really good at some things, but misinterpreted what it saw and was told to do, and disaster resulted. When we have A.I. developers worried that this is now a potential real problem, I find our present situation, well, I won’t say “fascinating”… but it is interesting.
Commodore Westlake was grossly unprofessional with his 'captain dunsail' remark. In private to Kirk it would have been a bit arrogant but could have been taken as a private jab. In public in the clear in front of his crew? It's almost like for Westlake he's bought into his own press over the M5, and has staked his career on it either as the motivating force behind daystrom, or having bought into the wunderkinder's hype and can see the possibilities of an automated fleet.
Honestly automation in of itself isn't a bad thing. Hell, the M5 itself wasn't 'evil' it was just proof that you can't as of that time (or now) leave automation unattended and without checks and balances.
Imagine being able to bring the M5 up in an emergancy situation when the crew is incapacitated, or to have as way to run simulations and suggest novel courses of actyion. Not every ship has a Kirk at the helm. So having this thing as an emergancy measure or even as an automated advisor as it inloads data from across the fleet so it can learn would be an interesting concept. Kindof like how a lot of 'freelance' jobs now are basically data collection points to feed into AI. Which is what we're strugglign with now. At what point is it OK to feed someone's data in to train the AI?
What if the AI gets enough junk data that it turns into a literal fascist slogan spouting racist (see also the twitter bot that microsoft had to unplug.)
The episode itself to me wans't 'anti AI' so much as 'do not forget your humanity and do not rush so far ahead that you let your mistakes pile up.' For me it represented the flipside of 'To Boldly Go.' Daystrom Boldly Went, but he did so recklessly. yet even with that recklessness, he, and by extension the M5, understood that lives are not expendable.
Nice graphics...!
I don't think Wesley was giving Kirk a cold shot with the Dunzel comment though. I think he was empathizing with him.
When you have to give orders to a crew, they have to be relayed, understood and carried out to the best of "their" ability. That takes time. A single "entity" just does.
THIS IS BY FAR SOME OF YOUR BEST WORK. ❤. I love the new animations and your weapon effects. You have created such good extended cannon.
One possible variant on landing party selection: Why were the captain and chief medical officer not included? 'Personnel too valuable to risk and not necessary to mission.'
Nice breakdown! I also really love the visual aesthetic of your videos, it reminds me a lot of the Starfleet Command games. :)
Kirk's "DAYSTROM!!!!!!" was FAR better then his later "KHAAAAAANNNNNN!!!!!!".
The problem is that the M5 DID demonstrate that it was far superior to humans in terms of using a starship in combat. The fact that it went crazy was a convenient excuse to keep humans in the loop. In reality, Daystrom would probably have won a defense contract to fix the AI so that it would stop attacking innocent ships, but keep it's tactical capabilities intact. And they would have been right to do so...perhaps Wolf 359 would have gone differently if Starfleet's ships all had several generations improved Daystrom computers, and the Dominion would have been sent packing early in the war if the Federation fleets were so enhanced.
Maybe, but with limitations on technology for rotating frequencies to confuse the Borg, it may had only prolonged the battle or it could've made the Borg a far greater threat if they assimilated any ships with improved computers onboard. It's hard to say, but it could've gone the other way too and without the need to assimilate Picard so the Enterprise D couldn't have used his connection to put the Borg to sleep to cause the power overload.
picard season 3 Fleet manuver shows the folly of this.
@@Revkor Can you provide more detail? Sounds interesting.
Trek never really liked to have unbalanced abilities..let's face it, there's not really any difference between ships in Trek..not in the shows at least.
@@treborkroy5280 What do you mean? Different shapes, different crews, different capabilities between Star Fleet ships on screen make them all seem pretty different to me. 🤔
Doomsday machine episode was before this and that being an example of why man must be in some control overall for the control of a starship!
The other ship captains *knew* a computer test was underway. Why the heck are they assuming Kirk has suddenly gone mad / is in control instead of a computer problem.
I’m so happy you’re back to focusing on Trek again! Another great video!
M-5 was the Trek version of HAL 9000. And nobody was better than Kirk at literally talking computers to death
I never understood why Capt. Kirk did not remind the Commodore that Capt. Dunsel was just one step away from Commodore dunsel. I also did not see why Westley was asking Kirk what he was doing when he knew the computer was making all the decisions. He should have been wondering what the computer was doing. That would have made the screens down scene at the end a lot more logical. This is also one of my favorites as it is one of the few with exterior shots of other star ships. Way better in the remastered version.
I am eternally amused about the fact that the fleet management software inflicted upon my workplace is called, "M5".
Whose engrams were used to program it? 😅
@@sureshmukhi2316 apparently it was from Arthur Andersen's brain shoved in a blender.
@@jonskowitz 😄
Yeah Refit enterprise is one of my favorite ships, Reliant is likely third but Akira takes my top spot.
Excelsior is likely 4th, its just a damn shame Akira model kits are so rare and even those starship models secondary market on it is so much.
Good thing M5 wasn't based off of my own neurological engrams, it would have been so introverted that it would just rather chill in spacedock and never leave.
For my science fiction stories, I imagine that a great many operations might have to be automated. Weapon selections and firing orders would have to be preset on approach to a hostile target, as human reaction times would be too slow.
Also, automated probes should be sent to a new world first, scanning for dangerous microbes, flora and fauna.
The best graphics that I have ever seen. Wow, what a great job you did!
Well done! Your work is top notch and I am very impressed by your analysis. I’m looking forward to more (I’m an old Trekkie!). More please!
My favorite episode (after "The Doomsday Machine").
Funny i actully watched the Segment where Spock says "computers make great servants , However, I do not want to serve under them.
Amazing video the editing and effects are beautiful! 🖖🖖🖖
That was an amazing video... This episode of Star Trek, is one of my favorites thank you !!!!
Nice essay on The Ultimate Computer. I also like your original Constitution Class very much. Might be my favorite. Can't wait to see what you do with the refitted Enterprise.
I would love to see a future video dealing with the size anomalies of the K'Tinga.
Are the windows on the pod cap a deck or two? What about the 2 levels of light right above the forward photon torpedo? You take into account the observation deck above the rear hanger deck, and the K'Tinga is _HUGE._
Not to mention Klingon BoPs.
Weren't the TNG era ones just scaled up exact copies, in-universe? (for obvious out-world reasons)
As always... great work. will join your Patreon ASAP.
I love how you made the Ore Freighter an Action VI Interceptor Transport 😆
man i love the cgi work! its very cool to havea more clear perspective of the battles.
7:03 - That statement is directly contradicted in the episode:
*UHURA:* _"Enterprise to USS Lexington. Come in, Lexington. Sir, I can't raise them. M-5 is blocking all frequencies, including automatic distress. Just a minute, sir. Captain, I'm getting audio signal from the Lexington."_
*KIRK:* _"Put it on."_
*WESLEY [OC]:* _"Enterprise from USS Lexington. This is an M-5 drill. Repeat. This is an M-5 drill. Acknowledge."_
*UHURA:* _"Captain, M-5 is acknowledging."_
*KIRK:* _"Daystrom, does M-5 understand that this is only a drill?"_
*DAYSTROM:* _"Of course. It was programmed to understand, Captain. The ore ship was a miscalculation, an accident. I don't know-"*_
*CHEKOV:* _"Sir, deflector shield just came on. Speed increasing to warp four."_
*SULU:* _"Phasers locking on lead ship, sir. Power levels at full strength."_
Yes, there was a direct acknowledgement that the 'attack' was a drill.
Fantastic video! Refit enterprise is my favorite design starship by far. Excited to see it. Also would probably be amazing to see it in a video similar to this one analyzing the search for Spock battle over genesis. Just a thought, as that battle is my favorite. Anyway keep up the great work!
Very insightful commentary with some really good points.
I love your depictions.
Amazing work.
"Opens fire". Very accurate synopsis.
I always forget how amazing the TOS battles actually were, and how tactically interesting they are compared to the newer battles we see, even pre-Discovery.
M5 was running on Windows 37 home edition.
The one thing I never fully grasped from this episode, is why the Ore freighter suddenly set M5 off. Obviously it was designed in story to be the point in which they realized M5 wasn't acting in a way befitting it's duty. However i still can't understand why the M5 saw an automated ore freighter, as a threat.
This post is the best explanation I found, Because I also wondered about that.
@indetigersscifireview4360
I love this episode. The subtle storytelling when it destroys the ore freighter.
The computer controlled freighter isn't viewed as a threat by Kirk and crew, but it is viewed that way by M5 because it's a computer controlled ship. Once M5 recognizes that threat it leaps to the conclusion that humans are just another computer controlling starships and now are classified as a threat to M5. Therefore it attacks the starships full force. Brilliant.
@@MagnusMediaGroup But that still doesn't add up to me and the reason is simple. It's not anything like M5, M5 is currently one of a kind and nothing out there is a match for it. Similarly the Ore freighter has no known weapons that could threaten the M5 onboard the Constitution class flag ship.
I'm not saying this ruins the episode but it is still weird, it randomly detects a civilian sector ore freighter that's automated, destroys with with excessive force, and that's suppose to be our hint that something's wrong. Sure that makes sense, but it never really explains what it was about that ore freighter that set M5 off.
Great breakdown of one of the best Star Trek episodes.
Great video, especially the part about the episode's true theme.
My take on the "Captain Dunsel" line was not that Commodore Wesley was saying it coldly, but that he was saying it mournfully. But what IS cold is how by the end of the episode, hundreds of Starfleet personnel are dead and everyone on the bridge is full of mirth after Spock tells a joke.