Did Particles From SPACE Almost Kill 315 People? | Qantas Flight 72
Vložit
- čas přidán 15. 05. 2022
- Donations are never expected but appreciated: paypal.me/miniaircrash
Join My Discord: / discord
A330 Image: Masakatsu Ukon
This is the story of qantas flight 32, At 9:32 am local time on the 7th of october 2008 an airbus a330 was on its way from singapore to perth australia with 303 people on board. The flight so far had been normal the autopilot was set the plane was cruising at the right speed and height. At 4:40 am UTC, the autopilot suddenly disconnected and the captain took control of the plane. This was weird but what happened next was weirder, a series of error messages started to pop up on the planes ECAM system, the ECAM or the Electronic centralized aircraft monitor is a system that displays important things about the plane, when something goes wrong it displays error messages that you can then troubleshoot. But the warnings this crew were getting made no sense they were getting stall and overspeed warnings at the same time. In addition to all of this the captains altitude and speed indications were all over the place, nothing was making any sense about this plane. The captain realized that he could not trust his own instruments and so he fell back onto the standby instruments and the first officers instruments. Not knowing what was happening they started to debug the errors that they were seeing but unknown to them, things were about to get a lot worse.
As the second officer was in the cabin asking talking to the cabin crew member the plane started to pitch down. The captain pulled back on the stick to arrest the planes descent but at first nothing happened. His inputs had no effect on the plane whatsoever. But slowly after a few seconds the plane started to respond and the dive started to ease and the plane climbed back up to 37000 feet. In the cabin people had been thrown around and people had been injured but the a330 was now under control and the pilots were looking to put the plane down the first opportunity that they got.
As the a330 flew on the pilots were working through the huge backlog of ECAM messages as they did that the plane inexplicably went into another dive. The captain, like before, pulled back on the stick but the plane initially did not respond. For some reason the a330 just wouldnt respond to the pilots inputs for a few seconds and like before the plane started to bottom out again. After the second upset the pilots noticed a new ECAM message at the top of the list it read “ flight control alternate law protection lost”. In simple terms the plane was telling them that the usual software protections that the A330 had which protected the plane from stalls and overspeeds were now gone.
The pilots had no idea what was happening, they were being hit with a barrage of warnings and they had no way of silencing them, it was at this point that the pilots decided that they werent going to risk it and that they'd divert to learmonth. As they made their way to learmonth the captain realized that the planes auto trim system was not working and that he had to trim the plane normally, but the captain wasnt taking any chances he disconnected the auto throttled and autopilot he did not know if the automation would cause another dive and he did not want to find out. Now that the plane was kind of stable the captain enquired about the state of the cabin, some people had sustained some serious injuries and so the captain declared a mayday.
Even though the plane was stable the pilots were still working through the huge list of errors on the ECAM. They contacted technicians on the ground over satellite phone to see what was - Věda a technologie
Major props to the crew here, for following the three guiding principles of airmanship: aviate, navigate, communicate.
Sometimes when I watch recaps of crashes caused by computers I wonder why the pilots where even there.
Not to mention the fact that they actually knew how to fly the plane without the automation. As simple as it sounds to apply air brakes, thrust reversal and wheel brakes, ground accidents have happened (and been covered on this channel) when some change of setting (or something) disabled one of these things and it caught the crew off guard.
@@glocke380 sorry what do you mean?
Particularly interesting one. It underscores the importance of keeping pilot stick and rudder skills up to date when automation fails. Great job by the flight deck crew. 👍
Layman question: Is it not weird to switch between left hand and right hand when switching seats?
@@bluecoffee8414 Sorry, D JSV. That's not a layman question. It's a moronic question. Do you know who invented the statement "There's no question that's stupid". It was created by a person who wanted to weed out the morons quickly. Very successful strategy.
@@bluecoffee8414 Don't pretty much every plane have the same "issue", captains have to fly with their left hand, and control the throttle with right hand.
except when the aircraft is fly by wire only
@@bluecoffee8414 I have wondered same. I can't imagine driving on the left side of the road using a manual transmission.
I also question 100% fly by wire. Is as bad as self driving cars. As much as programmers think they covered everything, something always slipped through the cracks.
Edge cases, man. Even exceptionally good QA testing can't find everything. Unfortunately there will always be bugs that customers find first.
I'm curious how they actually test those units - are all cases designed manually, or do they introduce some level of automatically generated / randomized tests? I remember reading about a case where a really subtle bug was identified in an automotive system by modelling it and running property-based tests on it using QuickCheck. If I remember correctly, it had something to do with an event bus protocol extension. As a result, for example volume control input events would get higher priorities then, let's say, braking... 😳 I wonder if that kind of approach would be able to catch edge cases like this. Or at least catch more of them - there is no such thing as a 100% bug-free system...
We can’t resign testing failures to accepting this inevitability. Lessons learned need to be fed back to QA and software engineering such that processes may be changed mean more instances like this one may be caught early.
If you’ve ever worked for AB you’ll know that there’s a culture problems being documented and forgotten. It’s just not good enough.
@@RingaMinga much like aviation safety, it is a never ending process of improvement built on the back of previous failures. Of course, that basically applies to all human knowledge and progress.
@@przemysawotarzewski557 it would be really interesting to see how their automated testing works and what sort of mock systems they use. One thing is for sure, after this incident I bet they really stepped up the scope of their fuzz testing.
In computerized railroad signaling systems, radioactive decay and cosmic radiation are a known hazard.
The joy of fully computerized planes and saving costs replacing a flight engineer. Thank God, some people are able to fly by hand and human senses without computerized aid.
Your CPUs’ caches have ECC (en.m.wikipedia.org/wiki/Error_correction_code), your HDDs and SDDs have them, the CD-ROM has them, your cable/ADSLmodem have those, digital TV has it.
These things would not even work without them, as they do produce errors while reading data as part of their normal operations.
Consumer PC RAM does not, early on there were parity bits, but these cost an extra bit per byte and thus went away. Servers have ECC RAM, because reliability is worth the small extra cost.
I would expect any critical component to have both ECC checks and a second/third copy of the item…
I
@@advorak8529 the question isnt "when will an error occure, how do we prevent it", it's "so, why would this even work given the sea of shit that can go wrong?"
I've traced one computer crash to a cosmic ray shower that was powerful enough to make the evening news. As I was on the phone with the OS developer, I mentioned what I had just heard and the uncanny timing. That allowed him to zero in on an area of code that assumed, rather than a simple binary flag value, an illegal state that lead to the OS controller to begin eating its own filesystem.
Where I'm currently living, we get other faults from the solar environment - geomagnetic storms that alter ground currents, generating power transients. I invariably end up having to reset one of my computers when one strikes. My tracking geomagnetic storms come from when I had to rely on satellite communications in a military environment, watching the space weather and planning communications dropping out, usually when we could least afford to lose communications.
@@spvillano Where do you live? Arctic circle?
I knew another software bug in the A330/A340 concercing the flaps calculator : the flaps calculator is suposed to perform a periodic routine test. But one day, in flight accros the Atlantic ocean, an AF crew had to adjust the main clock which was late. They didn't knew that the periodic flaps calculator self test was just about to start nor that it can be impacted by the clock adjustment. So since the time was adjusted forward, that periodic test was never triggered. Consequence: the test wich was set to faulty by default, stayed faulty. Upon the arrival at Paris CDG the calculator refused to deploy the flaps since it was in falty state. The plane has to land at 200knts without flaps. Fortunately the track is 4km+ long, but the brakes can still catch fire, but it went fine at the end. The investigation was quite complicated. At first nobody unterstood why the test was maked faulty and not any defect was found on the flaps. It tooks more than 2 weeks of investigation on the grounded plane and simulation before they could reproduce and understand the problem as a side effect of the clock forward adjustment. Not any developer had anticipated such a clock adjustment nor an alternate clock source in case of failure. Then the bug was corrected on all the A330/A340.
Someone obviously did a bad job during design and/or code review or neither was performed (shame on the manufacturer).
lmao, I can't believe they didn't use UTC time in their code.
@@lurkingstarit was the case. But now they use it.
That is IAF, thanks for mentioning the incident.
@@lurkingstar A UTC clock would still have this problem if it's adjusted mid-flight. It's more appropriate to use a monotonic clock to trigger the test. A monotonic clock is like a stop watch - a clock that just ticks forwards and counts time from boot-up. It is disconnected from wall-clock / calendar time.
They’re called “single event upsets”, or SEU’s, and they happen when either a cosmic ray or a high energy electron/proton goes through an integrated circuit and changes an internal state. And satellite hardware/software deal with them routinely. For the most part, the Earth magnetic field protects lower altitude electronics from the effects. But not 100%, which is why critical electronics uses the same mitigation strategy as is used on satellites. I’m shocked that it is not used on aircraft systems, particularly highly automated ones.
Just so.
The technology for detecting a flipped bit in memory is well known, but you have to design around chips that support it.
Once the error has been detected, recovery may or may not be straightforward. As SEUs are rare events, and any given SEU should affect only one module in what we hope is a redundant system, just having the affected module reset itself might be the right approach. A reset means that it's not sitting there spewing nonsense....
Actually testing SEU detection and recovery in the lab is another matter entirely; zapping the unit under test with sufficiently energetic particles to flip bits requires some nontrivial equipment, and there's not a lot of control over which bits get flipped, nor when.
Then, too, SEUs get blamed for things they almost certainly didn't cause; some years ago, there was that outbreak of sudden unintended acceleration of newfangled cars, and there was much speculation that the effect of cosmic rays on modern electronics could be the cause. Just one little detail: the events correlated strongly with the age of the driver, and not at all with elevation.
it's those not quite 0% possibilities that befuddle us, as they are so unlikely as to not seemingly warrant attention.
@@ericwilner1403 I think the two strongest correlations for unintended acceleration were close placement of the accelerator and brakes pedals and a driver unfamiliar with the car. As in, most cases involved both.
But the same thing happened in the same area with a different aircraft a bit later on.
@@prudencepineapple9448 Hello Prudence. I quote you "same thing happened in the same area with a different aircraft a bit later on.". I hope you realise who incredibly vague that statement was.
The phenomenon that some have hypothesised as causing this issue is also known as "bit flips". If you'd like to know more once you've finished this video, Veritasium has an excellent video on the topic titled "The Universe is Hostile to Computers" where reference is also made to this Qantas incident.
High energy particles from space cause way more issues than we may realise. For instance it is estimated that each gigabyte of RAM experiences four bitflips per month due to high energy particles. This figure is for ground based computers. I can only imagine that it is even worse at cruising altitude, seeing how the dose rate is quite high for a flight due to cosmic particles.
don’t servers use ECC memory that kinda mitigates the random bit flips?
And that is why ladies and gentleman, we do need pilots in a cockpit. Its gonna take a while to go full autopilot
This underscores the importance of skillful, well-trained pilots who can identify faulty computerized actions and shut them off, and then fly the plane manually. Given the complexity of present-day aircraft that rely on computers to juggle all of the tasks, this is no simple matter. In this case, I applaud the pilots for saving the lives of so many people.
What happens in a fly by wire only aircraft which has no connections between cockpit inputs and control surfaces?
What a great job by the crew after such a stressful incident! Not only is pilot error one of the most common reasons for incidents/accidents/incidents becoming accidents, but it is very rare where you hear of pilots landing a plane with no fatalities after a mechanical/technical failure (especially an unknown/undiagnosable one). Qantas training is top notch, it's no wonder as Australia's largest airline (est. 1920) they have never had any fatal jet airliner accidents!
Qantas has a huge focus on proper training, proper maintenance and safety; it’s why so few of their planes have crashed and why a significant portion of their air crews are ex military. There’s concerns that privatisation would destroy the safety culture in favour of profit but instead they just charge more for flights. It’s also why there’s only really 8-9ish major airports in the eastern part of the country.
Craziest accident series of 3 planes with identical faults off the west coast of Australia. Great video.
I was going to bring that up too. I think one was a Singapore B777 out of Perth. That was a flight computer corruption as well from what I recall. Can’t remember what the third was.
Wow, that was an incredibly subtle problem. I’m honestly amazed they figured it out.
Cosmic rays causing the problem is incredibly unlikely, but this is the only potential case I know of so that fits. They can flip bits in memory and things like that. So the data could be corrupted while in RAM.
ECC-RAM should prevent that and I guess they use it?
If the timing is exceptionally bad, could a particle somehow flip a bit as its traveling between the ram and cpu?
@@Aquatarkus96 yeah if the timing is exceptionally bad, it could theoretically happen in Cache or even in the CPU registers, there's no error correction there as far as I know of (if it's not in the execution logic of the code).
Yep. They're called bit flips and they're surprisingly common in space operations. Planes fly high enough to get a fair amount of radiation exposure, so it's not implausible that this might happen to them too.
Ahh the old cosmic ray bit flip. Made for one of the best n64 mario speed runs ever caught on video. Kinda scary when it happens to a plane though...
Yeah. Crazy what one random neutron can do.
I always wondered how electrical equipment could work in space safely bombarded by the Solar Wind etc. Aircraft are told to avoid the South Atlantic anomaly where the Solar wind is stronger because it can affect electrics.
It's not the solar wind, but the inner Van Allen belt following the magnetic field lines of the earth in that region. Those particles are a bit more energetic than plain solar wind particles, as those particles are zipping along the earth's magnetic field lines like in a particle accelerator.
Flight crews who routinely fly northern flights are also documented to have elevated radiation exposure, with cataract development being much more frequent in those crews.
Seriously, was the plane traveling through the "hole in the ozone" ?
Always wear your seatbelts during flight at all times...
. ....or your head is going to make the kinda holes in the compartments above you - just like in this video.
Some of the more dangers of automation
I'm a software engineer in a safety critical industry, and we all understand that Murphy's Law is a very real thing.
We program with the highest safety standards there are in our industry, but every time we release a software we just have to let go and hope to the engineering gods that we will not kill anyone. As much effort as you put into making your software bulletproof, there can always be some unforeseen. It is absolutely a constant stress in the back of our minds thinking that maybe a bug that we program could one day kill someone. I do feel for the people who programed that bug in the A330, I know they had the best intentions and I'm pretty sure they did it to the best of their abilities.
I'd be willing to put money on the code responsible for recognizing those spikes and switching to/from memoization mode having full condition coverage too, but no fuzzing.
I mean, space particles messed up all sorts of stuff, so it's entirely plausible
Jace, you've nailed it. Something happened somewhere, from something, maybe, so someone can post a video about it. Of course they include "not clickbait" somewhere.
The american pilot retired soon after this, too much stress
I know these pilots well, have flown with one of them. They did an incredible job to get the aeroplane down safely, and were absolutely terrified of another pitch down on short final right until touchdown.
They were then treated rather badly, with suggestions especially from Airbus, that they had actually caused to problem.
Kudos to the Captain for getting this plane on the ground immediately. Back in the day before pilot unions pilots were actually reprimanded for declaring an emergency. Today pilots are encouraged by the unions and by company to get it on the ground first and deal with the paperwork later.
@@donwilson4618 After a 30 year career flying heavy jets up to and including 747s, I’ve never heard of anyone being “reprimanded for declaring an emergency.”
@@FutureSystem738 30 years ago puts you in the 1990's which is not "in the day" I was referring. In the 50's - 70's the FAA enjoyed a reputation as the bad guy looking to violate anyone they could. You did not want to attract their attention. Also company policies at the time resulted in firings and demotions at will. It was a whole different ball game and I used reprimanded as an all inclusive term. Congratulations on your career.
@@FutureSystem738 Did you fly with Captain Sullivan? 😍
@@leonadittrichs3604 I flew with the first officer literally dozens of times up to as recently as a year or so before this incident, but not with Capt Kevin Sullivan at least in the last fifteen years or so, as we were both Captains and on different aircraft fleets- (though pretty certain that I did back in the 1990s).
They were both extremely thorough and professional pilots.
Man, I’m glad I found this channel! Love and appreciate how you present data and situations. REALLY appreciate how you research and deliver your content. Top notch, man! Keep it up. Thanks! ✈️
Thanks Aaron!
@@MiniAirCrashInvestigation could you do qantas flight 32
@@MiniAirCrashInvestigation great video.
Bit flips caused by energetic particles coming from space is one of the craziest factors you need to take into account with computing.
I’ve been waiting forever for a channel to cover this incident!! Thanks for covering this, it’s such a fascinating event
Also please could you bring attention to the fact the crew were never compensated for what happened and Qantas refuses to acknowledge what happened
Even in general purpose computing there are people strongly advocating the use of ECC memory, I wonder if the ADRU make use of that
I can't believe that essential systems didn't already use ECC. For IT professionals, ECC is a requirement for production systems, and its hard not to see how important this is
@@kurtwinter4422 on a multi milion aircraft would be worthy having also RadHard for critical stuff, gotta check if there is documentation available
@@fabiosemino2214 it's expensive and not worth putting for one in a million problem or even a crash.
Maybe (just maybe) ECC wasn't available at the time of development? A330 is from early 1990s, so development was probably made in the 80s. Was ECC memory even a thing back then ? Not so sure.
And if you add it later, I can't imagine the nightmare to re-certify the whole thing..
@@etiennec8139 ECC was a thing back in the 60's.
While unrelated to this case, it is in fact possible for an aircraft to be in both a stall and overspeed condition at the same time. There are two ways that this can happen. The first is if the aircraft is pulling high "G". Stall speed increases in proportion to the square root of the "G" being pulled - e.g. stall speed doubles when pulling 4G and triples at 9G. Therefore with enough "G" force, the stall speed could equal or exceed the aircraft's max speed (Vne). The second case is if the aircraft exceeds it's maximum altitude (ceiling). As the air gets less dense (lower pressure), while the stall speed stays at the same *indicated* airspeed (IAS), the aircraft's *true* airspeed (TAS) increases. At very high altitudes, the true airspeed at the point of stall can equal or exceed the speed of sound. Unless designed for supersonic flight, the aircraft must not exceed the speed of sound, and doing so will thus be an overspeed situation (even though the *indicated* airspeed will be much lower). In fact a non-supersonic aircraft's absolute maximum ceiling (the highest altitude it can fly) is determined by the height at which its stall speed (in true airspeed) equals the speed of sound. It is also why supersonic aircraft (such as Concord) are able to fly much higher than non-supersonic aircraft. At high cruising altitudes an aircraft's stall speed can be very close to the speed of sound, and so the pilot must be careful to fly only within the narrow range of speeds between stall and mach 1
When this was first happened there was talk of interference caused by a nearby Military surveillance center. Another incidence occurred in the same area a bit later.
In software engineering it is called the Sherlock Holmes fallacy.
How do one know "when one had eliminated all the impossibility" given it is impossible to know all impossibility.
That's why we still need human intervention, in this case, pilots.
There was also the case of a radiation therapy machine with faulty programming. Imagine you're getting cancer treatment but the machine is faulty and kills you.
Therac 25? I hadn't heard that it was a radiation-induced bit-flip, more a case of not accounting for certain cases of user input that caused unexpected behavior. In one case, the bug was caused by the removal of certain mechanical safeguards in the previous system without adequate replacements in the updated software, causing users to believe the requested dose was a lower value than the actual dose.
Or is this a more obscure series of incidents? As a little baby programmer in university, the story of Therac-25 rather stuck with me, as it was no-doubt intended. It tends to leap to mind whenever I hear "malfunctioning radiation therapy device".
@@Amanda-C. Yeah, I meant that one, I just summarized it as "machine faulty", basically the programming practices of the company that built it were poor as you described. I thought it sounded a bit similar to the problem on the airplane.
The Therac accident was due to badly designed and badly tested software, along with a lack of fallback hardware safety features. It wasn’t caused by cosmic rays etc.
@@Sashazur Succinct and accurate. I commend you!
Veritasium has an interesting approach on bit flipping through radiation.
Since many have pointed out ECC data protection, it is also interesting that high safety level CPUs like the Hercules ARM Cortex-R do the same calculations twice some cycles apart. In theory the radiation won’t affect both calculations, which means a difference in the solution is detectable.
Thank you, as always!
I remember reading incident report once it was out.
There is a lot of geeky details in the report, but what is very curious is that the event was triggered by the faulty value for AoA, which had a value matching current altitude.
This type of problems (when something is where it is not supposed to be) in software usually indicate a bug in code. Something in general direction of memory corruption, buffer overrun, uninitialised variable or humble logic error.
The problem is that the bug in code should mean that this type of an in- flight upset should be a lot more frequent
I remember a flight having to land at a remote military airstrip with all this damage. The poor grounds people, normally see nothing for ages, then suddenly a large planeload of injured passengers - little resources and a really small terminal building.. a 'fun' day at the office....
Back in the '70's, when I was a partner in a small microcomputer engineering firm, we read about random errors in the 1k RAM chips of those days. Technicians eventually realized the packaging for the chips was emitting occasional gamma rays which could switch a single bit from 1 to 0, or 0 to 1. I think the packaging was plastic, made from crude oil... when they switched to ceramics the errors were gone.
So it's not just airplanes and satellites. It was a real issue on the ground too, one that almost killed the micro industry in its infancy.
SM64 TTC Upwarp Qantas Flight 72
🤝
Unexpected code behavior
due to cosmic rays
Bits getting flipped by cosmic rays are a known phenomenon, and the probability increases with altitude. It's fairly easy to deal with by using ECC memory.
The processors that run these are not your state of the art core i9 12th gen. The A320 still uses Intel 80186 chips for example. Why? Because they work
@@tomstravels520 ECC isn't a cutting-edge feature. It's been used since the 1950s.
@@pseudotasuki Well you may as well ask Northrop Grumman or Honeywell or anyone else who makes the ADIRU's why they didn't use ECC. There is basic information about the processor and its composition in the report but not very detailed
I knew of this incident and I was wondering why the title is what it is? Thanks for explaining it in the video ,love your channel mate
ADIRU, a marsupial accountant
GREAT VIDEO
LOVE THIS :)
Thank you for your great videos! I've been interested in Aircraft my whole life. My Grandparents owned a home on the Bay of Quitie, close to Trenton Ontario. The Town of Trenton is home to Canada's largest Airforce Base called Canadian Forces Base, 8 Wing Trenton. It's home for the C-130J. I believe it still has a number of F/A-18C's, they've been their since 9/11. It's also home to a Search & Rescue SQN. When I was a kid I can remember the old DC-9's, such a noisy Aircraft! My Father would always take me to the Air Show, we both enjoyed it! Keep up the good work!
This is fascinating
Muons are a real possibility. During my internship at DESY, Hamburg we had generated a few images displaying stray streaks of particles that haven't been caused by electron-positron collisions.
You normally have streaks originating at the collision chambers extending toward the outside of the detector, but sometimes you have streaks entering from the outside, then travel through the detector while at the same time avoiding the reaction chamber and leaving the detector at some other spot.
Now, if particles manage to induce a charge in the detection elements, they can certainly do so in computers as well, especially since the latter aren't even protected against radiation!
Taking that into account the theory of high-energy particles (muons are exactly that. Even though they are decaying in mere microseconds, muons traveling at relativistic speeds do exist long enough to have an effect) cannot be dismissed at all. However, that begs the question why this corrupted data hasn't been discovered and trapped (normally a parity check is done on the data in RAM whenever it is retrieved, and if a problem is detected, an NMI is raised to handle it).
Two things are interesting. The Frequency of the Australian Power Grid is 50 HZ. 50 cycles per second. Or 24 ms per cycle.
So 1200 ms (1.2 seconds) divided by 24 ms per cycle equals 50 cycles. Also 50 cycles at 60 Hz is 1 second.
There is huge base full of large numbers of very powerful broadcast and reception antennas that cover a wide range of frequencies. Including a huge amount of EMF processing equipment and computers, memory storage and servers.
Many are made in countries with a 60 Hz power grid like the US. Most probably use switching power supplies that would need to changed to operate on the Australian power grid. Or single large inverters could be used to convert 50 Hz to 60 Hz or Vice versa. The base have its own separate power generation (probably as a back up, at least). This power may operate on completely different frequencies to improve noise and conversion frequencies.
Many of the antennas probably operate on much higher or lower frequencies. And higher or lower voltages and currents. That's a lot of switching that would produce a lot of harmonic transients. The interference frequencies would heterodyne into many complex waveforms of varying power.
The Australian power grid is essentially a very long antenna. This could act over long distances to broadcast or amplify the 50 Hz / 60 Hz switching noise or harmonics. Wavelengths: 50 Hz ~6000 km (~3700 miles) 60 Hz ~ 5000 km (~3100miles). Determining the broadcast powers and their interference behavior is incredibly complicated. But with enough computational power, it can be approximated vary closely.
Since this flight was approaching the coast of NW Australia I think that the above simple numbers and calculations may have had something, or a lot,j to do with it?
Or not. It's just food for thought. Your milage may vary.
I am aware of this case. And there is a lot of info on that including a documentary film on it. Anyway amazing job.
Maybe its me...elevator trim, AOA sensors and faulty data being fed to critical flight control computers. Did AB have an "MCAS" problem before Max? Maybe a ROM/EEPROM became corrupt?
Great job on the series!! Always look forward to new notifications
The data from the AOA sensors was correct. It was the ADIRU that was corrupting it.
the cause was,
data from the air data computer output to flight control computer was send in serial ,speed ,aoa,... next to each other. when the clock signal to those computer did not align ,the fcc receive the speed data as aoa data ,the data is in binary of 255 so the high cruise speed data was understood as a very high aoa data.
Cool. Thanks.
Excellent video, captain Kevin Sullivan did good work that day!😸
Semiconductor devices can have impurities in their makeup that can affect the way they function. If the frequency of malfunction is low then the device can pass test and be utilized where it can cause serious problems later on. This kind of defect can be difficult to track down. A system that acts erratically should be quarantined until a replaceable component that harbors the defect can be replaced. Airline maintenance, being expensive to accomplish, often overlooks these kinds of hazardous malfunctions. Over dependence on automation and intermittent malfunctions cause planes to crash and people to die. Add to that Boeing's contribution of faulty design, maladministration, mendacity .......
Computer engineer here. Bit flip due to cosmic radiation seems like the most likely explanation IMO. It's a known phenomenon and occurs more often than you'd think, but usually doesn't cause any major problems or any problems at all.
The Apollo program had all manner of reset events due to SEU's accumulating. Or in one case, buffer overrun due to the radar being left on while landing.
At 9:30 it explains that the bug was introduced during flight testing. They redesigned the algorithm that calculated angle of attack values to fix other AOA problems. This is not uncommon for a fix to create additional bugs.
AOA problems are not uncommon. Of course the most notable AOA problem was with the 737 MAX. We have seen it rear its ugly head on several planes. Airbus is currently dealing with an AOA bug on the A350.
I would like to see see all airliners with at least two pitot tube based AOA and two synthetic AOA sensors. This would reduce the reliance on cleverness needed to get a good AOA reading in the face of sensor failure.
If you’re referring to the recent EAD that is nothing to do with AOA
AOA has nothing to do with the pitot tubes, they have their own sensors. On the A380/A350 those sensors are combined into one probe but the sensors are still separate.
fascinating.................well done...............
This is the second time space particles turned up in a discussion about one of my hobbies. The first time was when a particle caused a never repeated glitch in a video game. I think it's possible, especially because the higher in altitude we are the more influence space has on us.
haha same here! SEUs are super weird. i remember a SEU caused a bit flip in a belgian election but since the vote count it had an impact on was so obviously incorrect they recounted it and got the correct result
@@ExperimentIV Yes I heard of that! It's fascinating what a big impact a single bit can have!
this is why we should always have a human in control whether it be aircraft automobile or anything that could cause fatalities should a malfunction occur.
Wow! These guys are heros! Great video
Because they were pilots and flew a plane with zero damage?
@@brucebaxter6923 not cool. you say zero damage but I would expect a plane with little to be just as hard to fly as one with physical damage. and is it really that bad to praise somebody for doing their job right, especially in a stressful scenario like that? for example, would you respond the same way to an ambulance's power steering going out, when the driver still gets the patient to the hospital? and let's add in the fact that a plane can have hundreds of people on board, and can seriously damage infrastructure, housing, and other things wherever it may crash. I think those pilots did an amazing job. (oh yeah and also imagine the ambulance is screaming warnings at you, and you are trying to get directions to the hospital over the phone) no offense, just try to think as if you had the fate of potential hundreds of people in your hands, without the wonders of computers to help you. TLDR, plane flying is tough, give them a break
@@xcharke3126
Tell your plumber he is a hero for unblocking your toilet
@@brucebaxter6923 not what I meant and I think you know it. bad stuff was happening, people were hurt, pilots were stressed, still did their job well. thats it. being a pilot is probably a hard job anyways. I would certainly applaud them. why wouldn't you?
@@xcharke3126
Nope.
I find it horrendous and cruel to put pilots in planes where these situations can occur.
Really interesting listening to all these videos. Do you have a pop filter for your mic? There’s a lot of popping in your audio and it’s painful with headphones.
just check the btrfs mailing list, there are around one report per month that, some memory bitflip caused problems and was detected by btrfs tree checker. Although most of them are bad memory hardware and can be reproduced by memtester, rarely we can hit obvious bitflip which can not be reproduced. Hardware problems are not that rare and we should be aware of them. The old days of 100% trust on hardware is long gone.
How much trust in pilots?
What's interesting is that back in the days before btree and other more advanced filesystems with error detection and correction (eg zfs) errors would just cause silent corruption that would usually go undetected, be written to disk and could potentially cause data loss that isn't detected until long after all backups are rotated through. In the days of plain ufs I recall that I had a single binary that should not have changed in any way since install get flagged as having a different hash than a db made from its identical cousin (both were DNS servers installed and patched to the same version), when I investigated in hexdump a 00 became an 0F. Freaky.
And god said 'let there be a spare electron in the memory unit'.. and there it was.. he saw it and said it was good.. meanwhile pilots - oh crap this plane is unflyable 😂
Lets just say there is a strong reason to have space / airspace computers in large old-fashioned circuit nodes; maybe use S-RAM instead of DRAM etc to get less “weird physics” into your computing. And error correction for paranoia. And have great shielding around chips. You cannot design away every failure mode, but you have to do your best when hundreds of lives on the line. Software barely works without hardware issues, you sure don’t want space making it even worse. Veratasium has a great video on these space events, if I understood it correctly NASA has documented that it is fairly common problem for “normal” electronics in space :)
Thumbs up as usual! But i didn't hear anything on the theory of the high power military transmissionstation closeby... Are spaceparticles more likely to be the cause than radiointerference?
It might not be very probable, but cosmic rays have done some strange things to computers before.
I remember a story about a video game speed runner breaking a record because his game system was struck by a cosmic ray.
In Exmouth has a US naval base which sends ultralow frequency for deep sea submarine communication since the incident the flights rerouted
No they didn't MH132 just flew directly over the base.
wonderful video! I'm not so sure about the whole space thing, but something that could have caused the problem is a physical defect in the CPU of the ADRU. that could've allowed for corrupted data to be made or used, not very likely, as aircraft computer chips are probably tested for everything. but so are softwares, supposedly. and there was that 1.2 second glitch. I thimkkk this scenario was just everything that could have gone bad did, other than the plane's structural or hydraulic components failing. Very glad the pilot/copilot got the plane down safely.
It's a fairly probable event, given that at cruising altitude, the radiation environment is substantially higher. The particles themselves typically wouldn't be the problem, but the bremmstrung radiation generated when they'd impact the skin of the aircraft.
Add in transients in the power supply, life can get kind of complex designing for an environment. I'm considering picking up a saturable reactor, as geomagnetic storms tend to cause system crashes where I'm currently living. They're just a big transformer that's designed to operate in saturation, which then fails to transmit transients very well.
I did trace one weird system error years ago, my computer, using a specialized SCSI controller, started to eat its filesystem. I had the developer on the phone, the news playing on the TV when a report of a record setting cosmic ray shower had struck right when the error occurred. From that, he zeroed in on the cause. A binary flag suddenly toggled to an illegal value, which caused the OS to go a bit wild. He wrote a patch quickly for the issue and pushed it out immediately.
Environmental problems can strike in extremely unusual ways. I've had multiple failures from flooding - in a Middle Eastern desert. Temperature excursions of a server room due to the AC freezing solid with ice - in that same desert. No, we're not going to pressurize the wet pipe fire suppression system inside of a 100 million dollar communications facility that carries all of both wars communications... The installation commander and CENTCOM commander both told that fire chief where to get off and no, we didn't run water into that pipe.
Oh, the flooding was due to a ruptured underground water service pipe, which flooded manholes with our telephone service cables for the entire base in them and for more merriment, also went unobserved into the diesel fuel tank for the emergency generator, displacing the fuel. Timing being of great importance, the flood occurred a week after our monthly generator test. Two weeks post flood, a power transformer blew out and the generator wouldn't start...
@@spvillano I can't tell whether to call your life cool or say oof for those problems.
@@xcharke3126 a bit of both, really. Had to pay for excellent good luck sometime.
@@spvillano well then, oof, and yo, you got a cool life.
@@xcharke3126 I did, then my wife of over 40 years died.
This has always been a very strange event. I think from memory this sort of thing has only happened twice and both times off the coast of WA.
IMO ALLEGEDLY
Veritasium did a video on this very topic, titles 'The Universe is Hostile to Computers', and even mentioned this flight.
very very interesting bit of science.
Yeah, the ability for cosmic rays to cause a "bitflip" (flipping a zero to a one or vice versa) has been well documented. As chips and memory cells get smaller and smaller, with fewer and fewer electrons determining whether a cell is in a "zero" or "one" state, it is becoming more possible for a subatomic particle from outer space to change the value of a memory cell.
This phenomenon has been linked to an inexplicable addition of 4096 votes to a candidate in a 2003 election in Belgium. (Note that 4096 is a power of two -- sort of a "smoking gun" when it comes to tracking down problems like this.)
Veritasium did an interesting video on this called "The Universe is Hostile to Computers" czcams.com/video/AaZ_RSt0KP8/video.html
must always remember the computers are only as smart as the people who program them and we know that people are not perfect. 😊
Yet we force them to pretend to be able to operate complex machinery.
Let’s get rid of pilots please
It's a little more subtle than that. Programs are only as competent as programmers make them. And no human being can anticipate all possible conditions a program will encounter.
Pretty crazy what a single random neutron can do in the right circumstances.
"Otherworldly" in this context makes me ponder if the realm of the digital - of code - is not a more "other" one than that of space.
the cause in this context is relevant only as a sidenote, the important thing is to handle it.
if we cannot do this by changing reality, the realm of code is thankfully malleable.
Whenever I make an error, I always blame it on a cosmic particle.
Its also what I call my hog.
Functional testing of computer software isimportant, but even more important is the processes of design review and code review where skilled people with various roles analyse the design and the resulting code respectively to ensure that nothing falls between the cracks. Obviously they don't bother to do that at Apple, but in eg. the medical realm, agencies such as FDA and EMA requires this to be done.
It was gremlins.
One scenario that the investigators didn’t look into unfortunately
Were they from the Kremlin.
The fact that they already had programmers go to the effort of creating a filter for bad data in place tells me that bad data was not impossible, which worries me far more than any ‘space particles’.
Very interesting, definitely shows the vulnerability with tech, but it’s still better than mechanical. What I don’t get is why the efs only used 1 adiru for its calculation. The computers are always cross checking each other, so if only one was showing the spike why would the aircraft react at all if the other two were in sync.
Single event upset. Well known in ICs. Requires error correction but there is still a finite potential for upset. We haven't seen , in modern times, a serious Carrington Event . Something like that may ground all modern aircraft.
science channel veritasium: The Universe is Hostile to Computers; has covered this before, so basically strong electro-magnetic force from space impact the airplane's cpu changing its bit code 1 & 0 (which is cause the spike on the data). On ground level this event could make BSOD / Blue Screen for a lot of computers in an area.
HELLO AIR CRASH DUDE
A software tester walks into a bar
Runs into a bar.
Crawls into a bar.
Dances into a bar.
Flies into a bar.
Jumps into a bar.
And orders:
a beer.
2 beers.
0 beers.
99999999 beers.
a lizard in a beer glass.
-1 beer.
"qwertyuiop" beers.
Testing complete.
A real customer walks into the bar and asks where the bathroom is.
The bar goes up in flames.
Interesting 🤔
We have been in a magnetic pole shift since 1859 which has intensified dramatically since 2000, weaker magnetosphere equals more radiation inbound, im thinking back to Air France 447, it seems a similar scenario
Cosmic rays are responsible for a bit flip that resulted in a never-before-seen glitch in a Mario 64 speed run!
The truth is out there!
space particles?
god damn it Space Godzilla!
This is precisely why you should always wear your seatbelt when seated on an aircraft.
Possibly a dry joint on one of the circuit boards?
A note;
In your description you have Qantas Flight 32, where as the actual flight incident proper is Qantas Flight 72 in the title.
Just a minor small confusing thing you could easily fix with an edit.
Cosmic Radiation can cause this.
What happens is called a "Bit Flip"
Since ram is broken down into "Cells" if a 1 in a cell gets hit by one of these Cosmic Radiation particles it can change that 1 into a 0.
Even with the odds of a specific playing being affected by Cosmic radiation being so low if you have enough planes flying around for a long enough time eventually it's going to happen. This flight might have just been the one that was in the wrong place at the wrong time
It was definitely the flux capacitor in my opinion. How did you miss the flux capacitor?
Well, to answer the question, "No... I don't think we need anything otherworldly to cause the problem."
Occam's Razor, "All things else being equal, the simplest explanation tends to be the most correct."
SO I'll go to my experience with vehicular electronics in general. 99 out of 100 times, you can trace dubious spikes and weird signals to a poor ground connection... AND that's really all you need, whether it's just a bad connector pin somewhere not making reliable contact so the processing and signals can be fed through consistently, or some tiny wire just has a crack in the insulation in just the right spot to be nearly undetectable on its own, but to randomly throw voltage where it shouldn't be going, un-tracked, un-traceable, un-checked... AND you'll get spikes and voltage jumps where they shouldn't be at varying intervales. This can go on undetected for days, weeks, even years... AND then there's that one instance where two such voltage spikes (aka "Power surges") occur in just the right manner at just the right times, and the CPU gets the idea that "random garbage" is good information... and all hell will break loose.
SO why do I keep refering to voltage spikes???
Because every sensor and signal generator uses voltage to represent information. It's how radio-waves work, and how your TV differentiates channels from each other. A system is given a specific range (tolerance) of voltage values, and the sensor sends information within that range to the CPU to recognize whatever value is being analyzed for calculation at the time...
Let's try an example for Pitch (nose up or down)... You only really need values from +90 to -90 (using degrees from nose straight up to nose straight down, with 0 being "dead level". Keep in mind, to simplify our code, we're only interested in the precise angle of the plane from nose to tail in relation to "level" with the horizon. We need NOT concern ourselves with whether the plane is right side up or upside down, since we can also use "Roll" attitude for that, including any correction... We can then choose our range in voltage, from a potential of 12VDC to ground... With some clever circuitry, and preferring AC signals, we can turn that into a signal range from +6V to -6V with 0V being "dead level" according to our CPU code, and the resulting math would assign a signal +6VDC to mean "Nose straight up" and -6VDC to mean "Nose straight down", neither of which is good for the plane, and probably very rare... Most likely, even a "soft limit" coded for +/-2VDC will keep the plane within 30 Degrees of level, which is about the standard for operating an airliner most of the time (99.99%) unless something is horribly wrong... +/-1VDC will manage probably 80% of actual flight characteristics, being within 15 degrees of "dead level" and with electronics sensitive to the "millivolt" (0.001 VDC) there's still PLENTY of range for very precisely calculating the plane's Pitch Attitude via that 1VDC per 15 degrees of Pitch proportion...
BUT when you get a very minor "short" in the system, it's still dubiously easy from a 12 VDC power source, to accidentally signal a LOT of erroneous information... The problem with that is that computers aren't creative. They don't know that you can't be more upright than upright, and any voltage past our "soft limit" is only going to net a blanket assumption, "nose too high" and pitch down as a response, whether 2 VDC or the full 12, unless someone codes in that "12 V is impossibly stupid", then the computer (ECU) CAN try to calculate and react... leading to disaster like this one in the vid. ;o)
Needle, ball, airspeed. Altimeter, vacuum "steam gages" 👍🇺🇸
You would think that they would use ERC memory to avoid bit flips
Quite reminiscent of the recent mcas problems.
No. Not at all.
@@brucebaxter6923 absolutely it is. Uncommanded nose down attitude absolutely
@@juanitamannn faulty sensor combined with criminally poor software vs high energy particle radiation flipping bits?
How are they the same?
Ps, mcas was commanded nose down, it took minutes of watching that big bright painted wheel spin and feeling the stick load up before anything happened.
The pilot only had to flex his thumb and trim the load off the stick, just like every aircraft ever built, or flex his knee and jam the wheel that was spinning, or reach next to his seat and flip it off to solve the mcas problem
I have long wondered about this issue and am concerned that care has not been taken to isolate these crucial devices from EMP or other electronic interferences. For instance proper grounding in a guitar control cavity prevents it, including basically creating a Faraday-type box within.
Now I see a very long grounding-wire from a plane all the way to the ground when flying at kms high... ;-)
Wow
11 years later, MCAS was born
3:08: Good for that pilot --- bailing out of that flight by diverting that ailing plane to the nearest airport. For when something is wrong, you don't want to take ANY chances and instead you want to assume the worst that could happen --- including the death of you and all your passengers. Thank God that the sky has a few wise pilots who know when not to put up with fkn pig shit on their flights.
wait is there not like a restart button for the flight computer? like to turn all the systems off and on again, and use the full manual controls during that
Yes but pilots aren’t supposed to just turn computers off unless directed to by ECAM or QRH procedures. Also because if the redundancy turning off 1 computer won’t necessarily stop it. To force the aircraft into ALTN law you’d have to turn off 2x of the ADR’s
Interesting.
Not hardened enough.
Now we need to shield everything against gamma rays!