What SHUT DOWN Norfolk Southern?
Vložit
- čas přidán 2. 09. 2023
- UPDATE: Norfolk Southern suffered another system outage on September 29, 2023. The railroad said the outage impacted train movements, its dispatching system and the functionality of its terminal operating system. Furthermore, Norfolk Southern said it had "found no indication that this was related to an unauthorized cybersecurity incident, or the outage of August 28." The issues were resolved in the early morning hours of September 30th.
Norfolk Southern says it was hit by a system outage on Monday, August 28th, 2023. This caused a variety of issues on its network and trains were ultimately ordered to stop. But what caused all this?
Follow me on social media!
Instagram: v12producti...
Twitter: v12productions?la...
Flickr: www.flickr.com/photos/v12prod...
#trains #railway #news - Auta a dopravní prostředky
UPDATE: Norfolk Southern suffered another system outage on September 29, 2023. The railroad said the outage impacted train movements, its dispatching system and the functionality of its terminal operating system. Furthermore, Norfolk Southern said it had "found no indication that this was related to an unauthorized cybersecurity incident, or the outage of August 28." The issues were resolved in the early morning hours of September 30th.
I didn't know Norfolk Southern used Xfinity... That explains a lot.
It was prominent on Norfork Southern but UP had a small PTC outage and then was having problems with it loading up the route route according to what I heard over the radio.
UP quickly got it working like it should wasn't as big of a situation as it was on the NS
That kind of suggests there's some deeper corporate issues within NS right now
As an IT guy, it sounds like they were testing a "failover" solution that was not configured properly in the first place. To me, this does not at all sound like hacking or some sort of cybercrime at all. I come to that conclusion based upon this part of the PR release from Friday: "a defect in their software created an error which caused both our primary and recovery data storage systems to become unresponsive". It appears they had redundant data storage systems but a "defect" in their software caused both systems to become unresponsive.
What? It sounds like their IT department is run by idiots. Shameful.
I literally made a presentation for Norfolk Southern as what to do in the even all systems where to shut down. It would keep rail traffic moving and operations as normal. But I was told it was useless because i quote "the stystems will never shut down" that was said by the Director of Dispatcher Training. This would make 4 systems shut downs since 2020.
Arrogance.
To keep things running smoothly in the event of an outage would require more people and telephone lines. And we all know the operating ratio won't tolerate that.
Those higher up Norfolks kicked themselves in the Southern by not listening to you. Hmmm.. 🤔 I wonder if they are wondering where you are yet.
@@stevenslouber4947
Probably in the "best" spirit of "Ingsoc" and "1984" the NS leadership has already forgotten the whole meeting.
Ego isn't just one hell of a drug, it's also a lifestyle.
Cool story bro…
I better get models of Norfolk Southern engines while they last.
Why??? They will be made forever....and microscale as well as others make decals...
Umm, they’re not going anywhere any time soon
Yea they ain’t going anywhere lol
It's a former think 😂
Put your tinfoil hat back on. 😂
“Positive” train control is experienced engineers, conductors and dispatchers, with the ability to communicate clearly 100% of the time. “State of the art” technology can let you down catastrophically.
That is exactly correct.
Down with technology! Let's go back to using horses to move goods and people!
Yes, experienced people working on and around the trains represented by unions that are not in NS back pocket. Who the he** is the VENDOR? We need to wake up. Privatization got a strong foothold under President Ronnie Reagan. It was an excuse to get rid of unions. Who is watching this VENDOR? Does NS have any full time employees that know what the he** the VENDOR is doing? This really is a national security issue. Who the he** is this VENDOR that the NTSB won’t look into.
@@beeble2003That may be where we end up. Unfortunately this present generation wouldn’t know where to start. This present generation’s answer “Buy more ammo” Scary! And we sit back and watch while companies like NS lead us on to a dystopian future. NS are CRIMINALS. We are naive dupes.
Cool explanation... no one gives a damn
A video idea, how NS managed crews during the stoppage, getting them off trains, replacing them, etc.
As 21 years in railway and now 2 in trucking this happens more often than you think and as long as you keep relying on it, it will be more common....
We stll dont know. But as a conductor i can tell you whatever it was is still causing us issues. Ptc bugs and mtr issues.
PTC systems on the Engines require a stable signal from the Global Positioning System (GPS). Solar weather activity can disrupt GPS.
Pretty much everything that requires positioning info runs through gps. The whole “solar” thing doesn’t seem to matter that much these days.
@@colinwallace5286 well, that is something for you to pay attention to, then!
@@jamesalles139 We used to have large external antennas to run all that, and they have become either tiny or non-existent as the system improved. Those of us that live further north are used to sometimes continuing to use the larger antennas because they hold signal better, especially in a mobile application.
I was in altoona when this happened, the whole horseshoe curve was down and got no trains for 2 hours
V12 Productions , am always exited when your video comes out, they are so intressting and they teach me something, awsome !
Thanks!
We were putting RAIL CAB CAMERAS, GPS, HOTD, BATTERY SAVERS and AUTO-START on the Locomotives when I retired. We got most of our 312 Locomotives done, that were assigned to
NS CDS.
Thanks for your video
I worked for the worlds largest engineering and construction firm. They built their data center in a basement in Houston Texas and guess what? It flooded and the data center was fried. A pat on the back for avoiding cyber attack is really not an achievement when cost are cut to the point that the system is vulnerable to a single point failure.
I’ve seen this happen more than once on CSX over the years. The dispatchers had no choice but to issue an “all stop” to every train systemwide until they could reboot the entire computer system. Creates a mess that takes days to iron out in backed up trains and yards.
Very good information!
The day the trains stood still
I’m a retired IT guy. Underlying the NOC used for managing the rail network is their data center which has its own NOC. Based on the phrasing of the press releases you reread, I surmise the data center lost a critical piece of hardware. I’ve seen such outages cascade cascad through other systems until the computer network completely crashes. Critical computer systems should always have contingency plans for recovery as well as redundancy when they need to provide 24/7 services. It sounds to me like this was an unanticipated failure. Despite spending lots of money for generators massive battery backup and other redundant systems-incl including hot-swapable hardware, something occasionally is overlooked. It could have even been routine maintenance gone wrong-a fat-finger error by data center employee. Unless additional information is forthcoming, no one will ever know.
Its also Norfolk Southern. Probably some ancient POS system from the 90s.
Poor risk analysis.
I run intermodal out of Cincinnati
3 yards and 2 railroads. NS is 2 of those yards. I had a crappy 47 hr week last week. Cause of this.
Darth Vader said it best as I paraphrase. This technology you think so highly of isn't so great.
Amazing video v12
they just had a problem like this the other day here in FL on the wildwood sub. they would get a green signal and as soon as the train would move the signal would drop. so the jax dispatcher was busy
Lockups or deadlocks still happens but are very rare.
This is the situation when a single failure allows a control point to be completely jammed with traffic.
Norfolk Southern Railroad is in a Bad Tail Spin and knows it. Investors are Extremely Disappointed in their Conduct and NS will Find out the Hard Way. 🤪👎
UP or CSX would love to acquire them, no doubt
No more mergers.
@@cjones3710it will happen eventually
Mergers won’t happen if the STB says no. They’ve said they will say no to big companies like that
@@andrewreynolds4949 doesn't mean they won't hold up to it
1st rule of network security is don’t publicize network break ins. It just attracts more hackers.
From what you described, they were hacked. It is too hard to believe a major rail company would go live with a faulty software update without running it on a test bed matching their system for debugging.
So it boils down to hacking or stupidity.
This isnt the first time a company has had it systems go down that was not related to a hack
Heck even big companies like Microsoft have had several times were services went down like the most recent one being outlook going down for half a day. Usually happens from server/computer maintenance gone wrong or a server fails
Remember the Rogers outage last year? It was a bad router update that killed their entire BGP ASN. No traffic from their entire network out to the world
@@tux_the_astronautYeah. This isn’t exactly something that is always a cybersecurity issue. Just look at the Southwest meltdown for an example.
@@tux_the_astronautI believe NS is considered a “big company.”
@@berserker406 Rather, never underestimate Murphy's Law. Any stupidity is just failing to have a sufficiently robust backup system, whatever bug caused it took down both the primary system and the main backup
Awesomeness Video Homie 😎
Charlie, you're the best!
I live right near the NS line Binghamton to Sch. NY. No trains, it was weird. Now not so.
Computer system maiantenence often spells "software upgrade". Bugs in the software then do the job.... Not unusual that bank services go down that way.
Norfolk Southern whats your dysfunction?
I see what you did there.
Haha
Good stuff as usual!
Bro I just got off of the Acela and I raced so many freight trains I also saw one of the NS SD70-ACE heritage units
This is BS on the part of NS. Wall Street This company seems like it has gone down the tube. Derailments galore, and now this. I remember when growing up that the Southern Railroad was an example of what a railroad should be. Now, I think it is no more than a Wall Street suit comprised of financial wizards who don't know shit about how to run a railroad and are focused on keeping the stock price high. This is not railroading, it is shameful greed. Let's make railroading great again, NS, if you can, which I seriously doubt.
Won't happen ! When the merger took place they runoff all the people that knew what they was doing . They are so far removed from that it cant happen . it would be like waiting till all the 5 year olds reach military age and putting them on a battleship and say you guys make it work cause we don't know how . That's exactly what took place with the old southern RR.
Derailments are daily in my 21 years, the only reason why derailments are high on paper is the media is sticking their nose in it....I'm glad I'm out, but the public has some to do with this too....when a semi is on the tracks they call it a derailment as well even though the semi was at fault...
The only real problem we had at the Louisville terminal was crewcall which just tells us what job we were on
I was an engineer for a Class 1 railroad.
PTC is an absolute detriment!!
"Disaster" is also an acceptable term.
I hope they can get back on their feet. Norfolk Southern is one of my favorite railroads!
If Southwest Airlines can “get back on its feet,” after a glitch of some sort shut it down during the busiest time of year for almost 2 days, pretty sure NS will be just fine.
unfortunately NS will be fine idk people can favorite railroads who destroy the economy and market and completely work against why i love trains so much but fairs enough@@Prolificposter
I can tell you operations are continuing we are not shut down and we continue to move freight. It may be a little behind but trust me we’re still working.
@@samantharokosh551 Oh that’s good
They're not on my good side. East Palestine is still toxic
Lies we all know it was Garfield getting in to the control center again.
Nice video
They need to bring back conrail, southern, and Norfolk and western back
Ain't ever gonna happen
Nah, just a change of management
@@andrewreynolds4949Yeah
I heard about this on my scanner yesterday now I understand what was up
This is the problem with off the shelf computer software having the same problem at my company
I saw the southern heritage 8099 on the left here: 1:38
Love your videos
Thanks!
@@v12productions enjoy your day until the next one cya!
One day if or when power grid failures occurs the rrs will be in a major crunch being they don't seem to have much of a backup plan people have been trained on. It'll be total chaos.
Someone kept pressing the Restart Later button on the Windows Update reminder until Windows said "Fine, I'll do it myself"
Let me guess, a lead dispatcher was playing Grand Theft Auto on his board when he shoulda been watching train movements! 😵💫
It happens. Lol
With freight volume currently being down significantly, this system failure may be a blessing as it will be a learning experience that will help them in the future when it is critical to maintain operations during peak volume periods if this should occur again.
Omg somebody actually calls Twitter “X”
I am told it is supposed to be called ex-twitter.
"[We] experienced a hardware-related technology outage that impacted train operations."
Or, in English, "Our computer broke so we couldn't run trains." Why do people feel the need for so much verbiage?
Lawyer talk
Exactly. They rely wayyy to much on computers
@@Jabba.Da.Hutt_ How else are they going to run their business? Before computers came along, they employed thousands of people with typewriters and teletypes.
@@beeble2003 well I mean you basically said the solution. Back in the day railroads operated just fine without so much technology. What companies need to do nowadays, especially the railroad is to “not” rely so heavily on computers/technology to the point that they cannot operate or function if something happens to it. It needs to be a 50/50 balance of computers and tech and good old fashion paper and pen. So if the computers fail it doesn’t 💯 cripple the railroad company like it did with NS were they couldn’t even build trains in the damn yard. Let alone having to tell all locomotives on the mainline and yard to immediately stop because they couldn’t do anything without the computers being up and working. Even now NS is “still” struggling with these computer bug issues that employees/ train crews are speaking out about it. It’s like these new cars, there is so much garbage tech/computers stuffed into these things that if “one” little thing or computer bug happens… the entire car is basically useless and can’t function correctly until the problem gets fixed. Compared to older cars, that if something happens it doesn’t completely take the vehicle out of action. To where you can still use it and it will function even if something else on the car has broke.
@@Jabba.Da.Hutt_ Let's get this in proportion. We're talking about a half-day outage. That outage affected let's say a quarter of the US rail network (which is an over-estimate). When did you last hear about an outage like this? I've not heard of any at all, so let's say it's the only one in the last ten years. That means that computers have provided reliable railroad dispatch in the US for 99.997% of the last decade. And you're proposing to throw half of them out and go back to people pushing paper? That's the most ridiculous over-reaction.
Jesus that ain’t good. Can’t believe I didn’t know sooner. I’ve never heard of any system outages on RRs before, and I sure hope no casualties were caused by it.
Sounds like everyone was just given a stop order. Probably the only casualty would be fresh produce...
Fantastic video. We can't live without computer systems. And it is quite hard to project future failures. This failure will be fixed to be avoided in the future. Nothing is as easy as it looks.
Now why didn't this make the major news feeds?
It made a couple of the ones I saw
Larry, the news was being by the train.
Thankfully on the C line in west Georgia we don’t rely on many signals at all, its 95 percent 171 territory
Out of curiosity ....could the dispatcher issue a track authority without the computer? I thought they'd need that just to issue block protection? Only asking because most of our territory it 251/261
@@bobwarblejr.2987 yes, we tone up dispatch to receive a track authority and we copy everything he says on our TA form and repeat it back after he gives us our limits. Everything is verbal
CAD/CADx (Computer Aided Dispatching) probably went down. You can still operate at 49 mph if PTC drops out. Never heard of a CTC/ABS failing, that tech has been out for decades.
So at first they said it was hardware failure. Then it was software failure by one of the hired outside technicians. Yep, they got hacked.
It sounds like there was a bug in the other company’s software that bricked a server or something. I doubt it was an intentional hack
@@andrewreynolds4949 Norfolk Southern would not admit to it being a hack job. They would not want other possible hackers to know that there systems is vulnerable. As they may try to get in to the system. That should make ones ears stand up on end. The big clue is that first they said it was hardware issues. Then they said it was software issues. I heard the Union Pacific had some issues as well. The other day I also heard on my scanner that CSX was having issues with GPS in the locomotives. Trains had to come to a full stop to restart the computers. It could have been a solar storm, but I also know that Russia is like beyond mad at us right now. To top of the cake here, Russia has been playing with Poland's railroads. Causing them to shut down by giving conflicting train order messages over the radios. Granted that was not as high of a tech level as hacking computers and all, but it sure did mess things up a bit.
@@SleeTheSloth If a software patch bricks your server, that definitely looks like a hardware problem until you figure out what caused it. I would prefer to go by actual evidence rather than speculation, but even if it was some sort of Russian-sponsored "inside job" it wouldn't have been NS that was hacked. The problem seems to have come from the software provider rather than the railroads.
What I think is actually notable here is how much trouble NS has had trying to recover from this compared to the other Class 1s, likely symptomatic of deeper corporate issues.
Edit: several software engineers here in the comments have said it doesn't sound like a hacking job, more like a software error and Murphy's Law gone really bad
One 'Key Stroke' dropped them to their knees !
@wes. Well, Wes, perhaps you're an expect at dropping to your knees.
When I first saw this video, I thought it was the end of NS
Happened again this past Monday
So did this effect our Marc trains here in MD on Monday last week ?? My local train that leaves @5am made it to DC via CSX rails. Everything went down sometime after that and didn't come back on til mid morning. I had to use commuter coaches to reach the BWI area. Amtrak was delayed and when i reached union station the board was clear showing no departures for 2-3 hours. Everything came up after 3pm.
Great deception and non bias report.Well Done!
Nice to know.
The Wi-Fi was disabled while
I was in signalman training class that day
NTSB doesn't get involved unless there is a body count and/or a significant monetary amount in damage. FRA may ask questions but ultimately everything "failed" safe and nothing was moving. If enough harm was done to customers not getting their deliveries then the STB could open an inquiry but it looks as though service was restored in a timely enough manner.
Our new pay system at Norfolk Southern is completely broken in Sheffield. So there’s computer related problems everywhere, it seems.
And today United Airlines had a ground stop due to a technology slowdown caused by an update -- or so they think. Keep your eye on other transportation companies to see if they also have "issues."
PTC was rushed into use. It is much needed function but railroad executives really were hoping that a Republican administration would kill the FRA’s mandate. But because of all the derailments and grade crossings incidents that then most in Congress felt the pressure to support it as a stop gap to slow those things down.
Not sure it was really that rushed. There have been lots of iterations improving on it since the original implementation, it's relatively well-developed by now.
Not so sure it was rushed at all....I'm no railway expert but I was out of the railway in 2009....I think they started installing it in 2011 and only have about 40 percent complete....
Look to Seimens for the PTC control.
They're one of the manufacturers of it.
any connections between the latest 3 train accident and software issue?
This is helpful
The Clark Kent of the rail community!! Keep up the amazing work man!
Thanks! Will do!
All the eggs in one basket, I think that’s the best way right?
What is Norfolk Southerns function these days
Destruction
Unhappy customers
Greed
Clayton County Marta project was shutdown by Norfolk Southern
So much for green light on the southern
What Shut Down NS Thats Crazy!
that title got me scared for a minute
All railroads need to be able to operate without PTC at anytime. When SHTF, PTC won't be there.
Could make a good short movie. Pretty silly how reliant Amtrak and other operators are the freight lines. Make an infrastructure attack piss easy for anyone intentionally trying to cause a disruption. Hit the big 3 all at once and suddenly no trains are able to almost everywhere in the continental US.
Big 3? There’s 4-6 “big ones” depending on how you count
Big 3??? There is about 5-6 Class 1’s in the US alone.
2:29 “Poorly”
Indeed.
This is some switch towers should be used or at least left alone and operational at times like this.
Surely they tried turning it off and then turning it back on again. . . ?
Wait what the? How did the railroad shut down?
They pushed the off switch.
If PTC goes out at CSX we run 49 miles per hour or if a haz key train 40 mph we don't stop. We ran without it years before this ever came out. People rely too much on technology but a real railroader knows how to run a train without it called running with mileposts and signals and knowing with those signals mean for your speeds and what you need to have your train at when you pass them other than a clear signal. Just amazes me how All of a sudden we forget how to run trains without technology. I know I know my territories I don't need that PTC I can run a train perfectly fine without that and trip optimizer.
Well now the CEO's and the officers and the stockholders thinks you can't .
Sounds like a mighty big failure of their computer system; as well as that of their software provider.
NS headquarters is like the avengers tower 😂
Well those bugs are running wild 😁👍
norfolk southern was my faverite railway
Glad to hear you got over it.
"Too big to Fail".
now faces the concept of
"Too big to Operate effectively".......
Counting on machines to operate the Machinery of a Rail System,
appears to demonstrate faults in very dramatic ways.
The East Palestine crash would have been prevented by a Caboose crew on a train.
The lack of Towermen reduces the number of eyeballs to detect dragging equipment,
or a defective bearing.....
A Leading Global Vender? Who might that be?
Oh
NS has not had a good year😂
So what they are saying is AWS messed up
PTC, but it's similar to the British AWS system in some ways
NOC is pronounced Knock.
@Anon... Indeed that is the pronunciation. As in "Knock knock. Who's there? NOC".
They shut down?!
As much freight as the N&W moved during World War two, with no computers, bring humans back.
Stop with the cpu reliance. That is not smart. Need hard wired always back up.. Do as it was in 1979. Its safer.
The rail network today is substantially safer than in 1979, and FRA statistics strongly support that
Now that's a line of bullshit you know absolutely nothing about RR operations . you just proved it to me.
Good Ole technology. It'll fail you every time.
"We were not hacked". They were hacked.
They were struck by Murphy's Law and caught unprepared
@@andrewreynolds4949 Did you trust them when they said "burning off the chemicals is safe"?
@@BlackMan614 Safer than letting it go off like a bomb, spewing the unburnt product everywhere, yeah
@@andrewreynolds4949 Says you. Residents of East Palestine disagree. NS is the modern Penn Central and should be liquidated for its broken assets - assuming it has anything of value.
@@BlackMan614 So you'd rather have a chemical bomb go off in the middle of your town?
3:15 - Who wants a massage?
Hey guys I got doom eternal to run on a Norfolk Southern onboard display... in unrelated news...
Centralised systems and their single point of faillure... more to come