CDK Global Outage "Post Mortem" Dealerships Down
Vložit
- čas přidán 19. 06. 2024
- Okay, in no way is this a post mortem, but the point stands that there is a lot of warning signs and red flags here long before digging into anything further. We know tons about the vendor and the customers before we even know exactly what happened. And will we ever know the truth?
www.bleepingcomputer.com/news...
#cyberattack #ransomware #computersecurity
I've had years of videos warning about these things, but it's all warned about in my book, too!
Linux Administration Best Practices by Me:
www.amazon.com/Linux-Administ...
buymeacoffee.com/scottalanmiller - Věda a technologie
It's a major attack, I work at a dealership and have 30 years in the industry, 18,000 dealerships can't do the basics, and the amount of personal information that is stored is huge
I think in the context he meant "well coordinated, professional, carefully orchestrated attempts, high-tech..." Like, nope, some dumbass just got clicky on a "I know what you did last summer" or "see my newdz" file.
I'm affected by this as well, not because we use that system, but because I own a performance auto shop, and we often use and install OEM parts. We've had to do a little more shopping around to make sure we're able to keep our inventory up.
Our IT guy is throwing an "I Told You So" party.
He called it a few years back when we changed systems. He was complaining about it for weeks after CDK was set up.
Yet Reynolds was a joke aswell 🙄
I bet he is! LOL This was SO predictable. This is exactly what was expected. This system was "designed as an attack vector" by any definition.
Can you please do a follow up video on this. I’d love to hear some more in depth details on the security flaws here.
I remember when we were learning the “new” Cdk service application. We had a Cdk rep there and I asked him why this wasn’t a web based program. He replied “I don’t think they will ever get away from DOS, it’s better anyway”
Thinking dealers know ANYTHING about IT and Cyber Security is an absolute joke. We are car guys, not tech guys. This is 100% on CDK not dealerships. I agree with 95% of what you are saying about CDK. This is a DMS that is so outdated that something like this should be a MASSIVE wake-up call.
LOL, aw yes, DOS Is better if the goal is to have the data stolen.
i work at a dealer effected. this is brutal, cdk services have always seem half baked
I do too. I started the company close to when the company brought in CDK. It all has seemed like a mind F to work the system. It wasn't made for an ag business, many issues we have and have to fight through daily. The employee who cheerleadered this system is no longer with us and we are stuck with this shit. It is like they over sold the product. Lots of promises and they never worked (warranty claim filing).
As an IT pro in the car dealer space, it is very common for dealerships to not see tech as a profit center but rather a major expense. Most dealer IT departments I have worked with have a ridiculous time trying to get the simplest security best practices implemented. Other things to know about CDK... They were taken private by venture capital a few yrs ago and have been cutting cost everywhere. So much so, that their support has fallen off a lot.
I like the term "politics over profits." :)
@@samit8178 As they're spinning up test groups, they announced "large public group" is live on core DMS services. Definitely politics and AutoNation is the largest customer. Let's squeeze out the single points and independents.
The Cdk employees are literally Carnival Workers. Reynolds is a superb application but unfortunately is too expensive. Man I miss Reynolds.
There is absolutely no way you were ever an employee of any government agency. You make to much sense. 😊
LOL, outside consultant, not internal government staff ;)
CDK Parts guy here. With over 15,000 dealers using this software, you don't think there's any chance that some percentage of them had no idea it was completely insecure? I mean, I'm not an IT guy and I very naively believed that CDK was pretty good. At least, as far as how the systems function on our end.
How the system functions are our end is completely garbage. I missed automate the first 1 hour cdk was here for the transition
Hey @DonutGuard - so honestly, no. I think it required total abject "burying their heads in the sand" or, more likely, IT fearing that management would retaliate if they exposed what a bad decision had been made. Once techs are told to install something so blatantly insecure and against all professional standards, they don't know what to do. Do they just do as they are told, do they expose the incompetence above them?
There's basically two paths that are reasonably taken. One has decisions with exposure where the pros are tasked with looking into products and in that case, there's no plausible way that all three, let alone one of the three, items didn't pop up. Like... it's so obvious it's impossible to miss. There's just no way.
Then the other path is that people who know literally nothing and have no common sense choose the product without evaluating it in any reasonable way and tell IT to install it. In this case, in a healthy organization IT kicks it back to them and says "what are you thinking" and they move on to the next product (I've done this even with banking!) But FAR more commonly, management has conditioned IT that management makes IT decisions and IT is just there to follow orders in which case the IT department knows how bad this is, but feels unable to point it out. But management knows it is making reckless decisions and has made efforts to suppress the free flow of security information to hide it.
In all reasonable cases, the dealership knew. Even a high school intern should have caught this. There's nothing hard here. Nothing that requires technical training. Any, ANY technical knowledge, any common sense applied to evaluation, any good business process would have protected dealerships.
@@samit8178 wow thanks for the great response. I made a reddit thread about the CDK hack and in my research, I found out that CDK represents roughly 2.5% of US GDP, and one thing I noticed none of the news reports talk about was how CDK's DMS is used in 15,000 out of 18,000 dealerships. They talk about the 15,000 but without the context of what percentage that represents. This hack is a lot bigger than people realize, and the knock-on effects of this will be felt not only through the entire automotive industry, but the entire economy. Dealerships feel the initial impact, but after us are the warehouses we source parts from which get supplied by the manufacturers, then there are the cars themselves which are being sold, at best, a reduced rate.
Not to mention the impact this will have on quarterly GDP reports next month. I'm glad I'm not near retirement age because 401k's are gonna take a beating.
Great video . I wasnt aware that business vpns were so insecure. Does this also apply to personal vpns on devices being insecure that you dowload the app to the device ?
Not necessarily. It's two different things. Personal VPNs carry risks, but not the same. A business VPN implies you are using the VPN to bypass security and two entities get to be exposed to each other without extra security. A personal VPN generally (but not necessarily) only drops you to the open Internet and you assume it is just public the same as before.
It seems like this happened right after the single sign-on was rolled out.
What a disaster. CDK will be paying Millions for this.
I would love to see a video where you sit down with MSP CEOs etc that sell those systems you say no "professional" would ever use and discuss how they can sell that stuff?
A true MSP doesn't sell software. If they are selling this stuff, they are actually a reseller using the term "MSP" as misdirection. No one working in IT can LEGALLY sell this, it violates IT practices to deploy this AND it violates IT as a career to sell software. By definition, someone selling products can't be IT. So an MSP can't be a reseller. Anyone reselling this is just a store and it's "caveat emptor" because unless they lied about selling it and did it through a third party but got a secret kickback (which would constitute a crime in this scenario) anyone who bought from a reseller knew it was a salesperson and it's the business' requirement to provide their own IT oversight to verify that the salesperson is providing something of value.
Cox automotive.....they are allegedly doing the same in auto hauling. crash, consolidate, control
May I ask the noob question, what/why do you refer to VPNs as insecure?...Are you referring to overall their deployment/implementation?
I have several videos that touch on this in different areas. I'll make one soon that hits it head on. But the basics are... a VPN used as NORMALLY expected implies 1) that LAN based security is used instead of proper security and 2) VPNs are a tool primarily used (and in this case exactly used) for defeating security. Which is exactly what happened. VPNs are a means to bypass the firewall. They bypass assumed security, they expose one entity to another in ways assumed to never happen in this day and age and they create what we call an "open window" infection vector.
@@samit8178 understood, thank you for the clarification and look forward to the video!
So, here's something you didn't talk about. CDK is ADP. ADP runs on the exact same security CDK does. If CDK is compromised so is ADP. ADP is a global accounting software way bigger than the 15,000 dealerships alone. Also When CDK was sold to us it was under a "proprietary network solution" which was a literal black box that they won't tell you what is going inside it.
The site is shitty on a whole, i can see how it was attacked and im sure it was easy
Yeah, not only was everything about this clearly designed to be wide open, it also advertises itself as such. It literally INVITES attack. But "attack" is unnecessary. This wasn't likely focused. It was far more likely just an email attachment.
Interesting video, glad to be out of that virtual realm..........
I wonder if the *powersports dealerships are affected also? We had Lightspeed which is part of **CDK Global.
We sold our dealership back in 2020 and were paying a small fortune for that system.
**CDK Global Lightspeed is proud to be a part of the Brunswick Dealer Advantage Program, which provides exclusive pricing for LightspeedEVO. As the leading Marine DMS provider, CDK Lightspeed has helped hundreds of dealerships, marinas and boat yards gain control of their business.
*Lightspeed modules are tailored to help you solve common Powersports dealership challenges. It’s the one stop shop for all of your needs.
Hi
Great video. I'm buying a book.
Can you do a follow-up with GLB FTC safeguards impacts?
Thanks so much! ANd yes, follow ups to come.
CDK is a big Cisco shop, so naturally they install them in every dealer. I used to work for them. They laid off or fired every American I worked with. Their network team wasn’t staffed for weeks leading to this attack.
Well, mistake where made......... This is not incompetence, This look like sabotage!
Cisco is one of those "flag" products that you can use to visually see if a company is being tricked by sales people. As an IT pro, walk into a business... if you see Cisco devices, you know that you've got an opportunity because you can guess at all the insecure stuff that they overpaid for. It also tells hackers you are an easy target because you aren't evaluating your IT needs and likely some investor is making the calls and hates his IT staff and doesn't trust them.
@@Elvisgratton3x It's true, there's no possible way to excuse many of the decisions. The admin privileges, the VPN... those aren't plausibly honest mistakes. Those have to have been intentional setups for bad actors.
This isn’t true at all
I work for Ford dealerships that are affected what I can tell you in this case in my end is that the owner knows nothing about IT and he trust IT professionals to figure this out for him and clearly they were not up to par
That's common, the problem is that as the owner, he's got to start by hiring someone at the top to be trusted. Individual dealerships aren't really big enough for that. He should have an IT firm helping, IMHO. At least at the CIO level. One that isn't a reseller, one that actually does IT. Same rules as for hiring in any other department. As the business owner, it's his primary job to hire good staff. Sucks if he doesn't know how to do that, but that's where he needs to focus his efforts.
NoVA: Its still down...they say it will be back up tomorrow july 1st though..
If Cloud computing is just someone else computer. VPN is just someone else network.
It's slightly different. It's more like "VPN is just EVERYONE else's network." Because CDK didn't just expose themselves to the dealerships. They exposed all the dealerships to each other, all of them to CDK, and all of the integration vendors to all the dealerships!
Some manufacturers require certain software vendors of their dealerships.
Possible, if so, that could pass on criminal issues to those vendors if that is the case here. An outage of this level should constitute a FTC concern as this significantly interrupts American commerce by means that should never be a concern to an American business.
I thought it was a DDOS with a ransom and not a ransomware attack?
No, it's DEFINITELY not a DDoS. They were hacked. And still down.
Cash, papers, and pencils,
what's up Mother@#$%&Hackers !!!
I bet this hack was perpetrated by "big pencil". jaja
A lot of hot takes in this video, but here is mine. Spent my entire IT career working with aerospace, automotive, and financial industries and I see a lot of businesses that do not dedicate enough IT resources to properly manage a secure system. Most still lack a dedicated cybersecurity role. (CDK has a Sales Engineer as their head cyber) Even in the financial sector I see business owners who refuse to or cannot afford to spend the money required secure their systems unless their cyber insurance requires it or their customers require it and audit them to those requirements. Also, the cost of technology is rapidly rising and the number of tools required to secure that technology keeps growing. I bet a lot of these dealers have very limited IT resources with enough knowledge to even provide these basic best practices you are harping on.
This goes much deeper. Their system, from day one, has no component of security anywhere. Every aspect of it violated basic IT and software engineering practices. It would have literally been cheaper to do it right. But that wasn't a priority, obviously.
I agree, with CDK it goes deeper and showcases just how naive many executive leaders really are. In some cases all the IT leaders can do is say "I told ya so." But for c-suites, why be concerned when the worst consequence is a golden parachute? With CDK, they had the money to do it right, but profit today mattered more than business tomorrow.
You should watch CDK's State of Cybersecurity if you haven't already. It aged about like a dog turd from the 80's. czcams.com/video/4NWBegkCzTI/video.html
Chime in with my take..I am the IT director for a decent sized auto group. Much as I hate to say it, this is probably the best thing that could have happened. Yes it sucks for everyone impacted, but it is finally opening the eyes of owners to the threat that myself and many others in my position have been begging and pleading with them about for years. CDK will sink their teeth into every part of a dealership they can, and by doing so you are basically forced to use them. A simple unhook becomes something that will takes months of planning and coordinating. All the data, all the integrations with 3rd parties, the integrations with the auto makers, etc. Still very doable, but use to be impossible to get management to sign off on it. Majority of the time, the CDK sales reps will totally bypass us and go straight to upper management and feed them the latest buzzwords and get them to sign up for even more. They can’t stand when IT gets in the way and asks to many questions or tries to shut them down. They want full control of everything. Phones, Network, Security, all the way down to basic IT support.
Sounds complicated
Not saying the VPN staments are wrong but VPNs are not completely a horrible thing to have deployed and they are just like the network running already and its connecting to.
Filter/firewall the communications to what is needed and nothing more. Internal or not!
However allowing a vendor to manage a network and trusting them to secure it for you... you should then test, review and verify it it.
I can confirm CDK was horrible on the network management/security side ... Then add all the ownership changes CDK has had made it that much worse.
VPNs aren't exactly the issue, just their usage. THe problem is, if you need a VPN that can only happen for bad reasons. Technically there is a VPN anytime you use HTTPS, for example. And that's good. THe VPN itself isn't the issue. It's it being used as people use the term to do what people assume it is for that is bad.
Some tech got screwed out of way too many FRU’s too many times
Advisors kept tacking on free shit for csi scores 😂
I dont agree with most of what this guy is saying...
Okay, can you explain what you don't agree with? What aspect of criminal negligence and abject incompetence do you agree with?
I know right
23 years in the business and never thought it would happen. Complete mess.
It happens constantly in industry after industry. This is SO common, just people don't realize because it's isolated by industry. People outside automotive wont be aware of this one either.
In what world can a business develop their own erp? You're insane dude
As someone who does exactly this for his own business, I suggest you work with professionals, any car dealership makes plenty to fix this problem, let alone an industry.
@@samit8178it goes without saying you don't run a dealership, in fact your business is absolutely miniscule compared to a dealership. Many with deep pockets have tried valiantly and failed or run out of money in the effort, even after a LOT of money was poured into the effort. So it's either insanity or complete ignorance to the topic you're trying to preach on (but failing).
No hackers...it's the LORD AT WORK! Yeah!...Babylon has fallen! "And the merchants of the earth shall weep and mourn over her; for no man buyeth their merchandise any more" ~ Revelation 18:11.
Say what now?
@@samit8178~ just read the scripture that was cited....and your answer to your "say what" rhetorical question will be revealed to you.
This guy is very ignorant about the car deership industry. There are tons of reasons for a dealer group to not build their own system. Back on the day dealerships, and even manufacturers, had their own DMS. Those systems were never successful. It would be extremely expensive just getting basic functionality. But there are so many workshops beyond basic functionality. The very large groups are always in the hiring mode and training is also very expensive. Whe n they can hire someone who already k own the system is a huge cost saving. As for the phones -- this guy remains equally ignorant. The phones allow dealers to know who is calling and the customer's record automatically display think service RO with the latest status). It also allows dealers to send text messages notifying customers of the status of their transaction. This guy needs to learn the industry before producing an ignorant opinion piece.
If you knew CDK and other DMS/CRM Security was such an obvious issue why didn’t you expose it prior to the breach?
Monday morning QB!
How do you know that I didn't? I have meetings with dealerships about these guys every few weeks. EVERYONE in IT knew about this, I've been warning about this for nearly a decade. I've exposed CDK to their customers, and I've documented their practices as clear, unquestionably unprofessional that no one could use. So the question is, why didn't you listen?
czcams.com/video/TP7XhhyDB3c/video.html
Here's a warning from six years ago. If you didn't get warnings about this vendor, you need to ask yourself how that is possible. Literally no one that can call themselves even casually interested in IT can have not warned you. Ask yourself HOW you could possibly have ignored every IT pro on the planet screaming about this (no one has to name CDK specifically, although I've warned SO many about them for most of a decade) because it's industry best practices that have "zero exceptions". And lots of them. And lots more that have "only the rarest exception." So the real issue is... given the insane level of industry warning on this, to the point that no one can plausible claim to have not have known, and even anyone that hasn't heard but has a brain can use common sense to determine, what made you allow them in the door?
You realize I published a BOOK that you can get on Amazon that warns about much of this, too! LOL I'm literally the farthest thing from a Monday Morning QB on this that exists on the planet. Every. Single. Item. in this I have published posts, articles, videos, and a book on SO many times. I've been going nuts warning businesses about this all for decades. My company also provides professional consulting about this, for 25 years, all of them warning about this. And we specifically warn customers about these products specifically, just in case they have no IT, but literally no one needs to know CDK specifically, that's a panic response.
So why didn't YOU warn anyone?
I never heard of you or you’re Chicken Little attempt to get CDK customers to understand the vulnerability of them having CDK Websites, DMS and CRM, not to mention phones.
Did you contact Holman, Penske at other large groups like Group 1 ?
You have ABSOLUTELY NO IDEA WHAT YOU ARE TALKING ABOUT! I watched you speak about this industry, this company & their clients in such an “I know it all” Yet, every single point you made was terribly flawed with assumption, error & speculation! You need to learn & understand what this company offers & provides for their customers! You are totally speaking out of you xss! Please DO NOT FORM AN OPINION by someone who knows nothing about the industry or the product!!! I can’t even listen anymore … point by point … you are wrong!
So did you have a point or just want to rant? Did you feel exposed by industry standard knowledge? Nothing here is like form me, this is just basic knowledge, lol. Are you saying the entire concept of IT due diligence is wrong?
So for my next president:
My wishlist
1- help ease inflation costs
2- finally start drilling and help make the U.S more energy independent
3- work on security in the country
China, north Korea, Russia
This great country of ours has been on cruise control for so long
The world is beautiful with lots of evil folks who want to hurt us
Omg the hackers did this so easily
64 million bucks for random? My God.😮
Please contact me. I represent a group looking at litigation and we would love to talk with/possibly ask you to be a paid consultant for us. Thank you.