How to Burn Money in the Cloud // Avoid AWS, GCP, Azure Cost Disasters
Vložit
- čas přidán 4. 07. 2024
- Out-of-control cloud computing becomes very expensive very quickly 💸. Learn how a startup racked up a $72K bill in just 2 hours on Google Cloud. Tips to control costs on AWS, GCP, and Azure services that are priced to scale.
Thank you to announce.today for sharing your story!
#cloud #dev #learntocode
References
How we Burnt 72K blog.tomilkieway.com/72k-1/
GCP Budget Pub/Sub cloud.google.com/billing/docs...
AWS Budget Alerts aws.amazon.com/about-aws/what...
Courses
Security fireship.io/courses/firebase-...
Data Modeling fireship.io/courses/firestore...
Upgrade to Fireship PRO at fireship.io/pro
Use code lORhwXd2 for 25% off your first payment.
00:00 Hello World
00:49 $72k in 2 Hours
02:47 My AWS Story
03:35 Budget Kill Switch
04:32 Cloud Overflow
05:17 Scale Down
06:32 Bad Algorithms
08:02 Security - Věda a technologie
Have you ever burned money in the cloud? Please, do tell 👇
Yep
Nope.
I'm getting 100$ a month worth of AWS credits for my Alexa skills, which means that I *print* money in the cloud on a regular basis, in a way. :P
Yup, I've spent about 600$ on AWS by just starting and forgetting about cloud service. Thankfully, customer service sent back to me all the money.
Just applied for my credit card thanks for warning..........
I immediately checked my cloud dashboard after looking at that bill
Lol, me too, I have a free tier in AWS
Oh my God me too every single time I watch this video.
I too looked at my Azure cost management
lol Yeah i always check it when i log in and have an alarm set
Ive heard a few horror stories but they usually give you credit
do ✍🏼 not ✍🏼 create ✍🏼 infinite ✍🏼 loops ✍🏼 in ✍🏼 the ✍🏼 cloud
XD
This is gold!!!
I agree, but on the other hand, forcing user to use credit card is a dick move and must be stopped.
@@vogel2499 This. You should have the option for a pre-paid service aka "once your account runs out of money, you're informed that your services have stopped and will continue to run after you've added more money to your balance." Especially in countries where credit card aren't a thing. I e.g. live in Germany and here we have debit cards (aka you cannot go lower than 0€ in your bank account which is much more preferable in my opinion) and really only a handful of people have credit cards. And ordering one just so I'll be able to use cloud services is just ridicolous.
@@Black-Dawg-Jesus I see the option to add debit cards on AWS, Azure, and GCP
Fun fact : Cloud Overflow = Bank Account Underflow
If that means that I have $9,223,372,036,854,775,807 on my bank account, then hell yeah
I wonder if you can f*** up so bad an official from your bank shows up in the middle of the night...
Or, rather, underflow 😂
Mb
Cash Heap Underflow...?
😁
One of my major fears while learning cloud. There should be a limit for how much you spend on the services, but I think no provider has this as a configuration.
They do actually, but it's not as simple as it should be. I talk about it in the video :)
I've heard many tales about people forgetting to turn a demo project off but all of them just contacted their cloud providers (GCP & AWS, not sure about Azure) and all them replied "np" and cancelled their billing - Still I really want to watch this video asap. Great content as usual man!
@@Fireship which you have not published yet!
If you forgot something you can call your provider and request a refund. Yes, it works!
AWS provide it as well
Sorry to keep you waiting, please grab a free sticker in the meantime. EDIT sold out!
:D
Ok :p
@@harsh9558 Afaik we don't have teleportation technology as of now so you will have to wait a bit for the sticker to arrive, lol.
Is EU supported 🤔?
@ Yes, worldwide!
I'm so glad we're on premise. We recently had a problem in production because we loaded thousands of records into memory by mistake, spinning up CPU and RAM usage. This made some customers unhappy because they couldn't use our service, but imagine this happening in the cloud. Unhappy customers AND a bill for useless work done!
You still paid for the hardware wear and tear, you still paid for the electric, cooling, any networking costs, etc. It just feels good because the bill isn't lumped into "here all your compute, storage, etc usages now pay us".
@@rickytorres9089 But it's also a natural barrier since we could only use up the resources we physically have. While cloud providers have cost and scaling caps, it's easy to configure them wrong. One misclick and you spin up hundreds of instances.
@@rickytorres9089it feels good because that mistake probably cost them a few cents
@@rickytorres9089welp, all method has pros and cons
@@rickytorres9089there's basically no CPU/RAM wear, so there's at least that
Maybe it happened to google today, they were down for an hour!
I think they were fixing a hack or databreech but I still need to get some details.
@@asandax6 They should clean out all those Asian girl bots in comments as well, they are everywhere.
@@lardosian And those bots with sexy lingerie
My first experience with cloud computing was with a simple GCP windows machine that had set rules (I explicitly set it to NOT grow, to avoid cost). It fit well into the free trial, so I just let it be. I deleted the machine after 6 months (out of 12 for free trial) and never really used GCP again.
Then I got a bill for 50$ for a machine I didn't even have access to (because it got deleted), and only thing their support said is to tell my bank to block it.
Nice
It's not 24 hours a day for 12 months, Compute Instances have a limited free trial time by hours. And just by reading 6 months, I am pretty sure you exhausted that free time limit of Compute Engine.
😊 😊
Yes, I did burn money in the google coud, 5600$ bill in 4 hours - was attached by a scraping bot and thanks google cloud they understood and "reset" the bill
Using Cloud Overflow now!
This is the reason why I am using good old VPS with monthly fixed pricing.
So true
You can sleep without thinking of your cloud bill as well
6:00 It's interesting 69k/71k of the budget was related to Firestore reads pricing, whereas if we compare it to Oracle NoSQL database a 116B reads would cost around 287$ (99.5$ cheaper than Firestore reads pricing). Which means if Oracle is able to commercially sell these reads with such a price and gets a profit from it the actual cost is even much less, meaning when GCP takes this billing it loses almost nothing.
Incroyable.
Awesome. As a frugal freelance dev, cost is one things that I gotta watch out for. Thanks @Fireship
Somehow you always know exactly what I’m researching and come out with a perfectly explained video on it. Thank you sir
made me go double check my aws billing dashboard... just in case
lol! I just did the same on firebase hahaha
I had a hacker rack up around 32k on my behalf on AWS a couple years ago. Not fun. Gladly Amazon believed me and credited my account.
this explains why their pricing in general is so much above classic hosting providers like hetzner etc - I don't think those cheap provider would be that customer friendly if I would forget to delete an expensive server for a long time.
There should be a feature like simulating the cloud run functions or some other services to catch errors before deploying to production and in this Artificial intelligence might help to catch up errors early than an average person would.
write test units!
@@alii4334 Incroyable.
but you need to run the code to see the errors, and the code takes google cloud levels of computing to run... you can't simulate it without running it, that's the actual halting problem...
@@unflexian I agree.
This has actually terrified me into never playing with anything on the cloud.
Or you could just get a cloud server instance and connect to that for a fixed monthly cost that's independent of use. It might not be the most optimal for large scale applications or certain workflows, but if you're learning, testing, or just hosting small scale projects, it works perfectly fine. I have a handful of small apps that I deploy through Docker on an EC2 instance.
I've been binging your videos, thank you so much.
Never ever ever hardcode your API key values in your code, even for one second. You will probably forget and check the code in. Use environment variables or cloud secret managers.
i just gitignore everything
@@SUPABROS what exactly do you have on the git than?
@@feritperliare2890 maybe only .gitignore 😂
@@_imawesome nah the gitignore is on his personal computer doesn’t go up maybe he put a read me accidentally
Incroyable.
best web dev video of the year
Keep up these top tier videos mate! Cheers
"cloud overflow" is a useful coinage, thanks for that
The reason I am always almost on my wits end when on AWS.😅 most basic thing is I always makes notes of service I started and access so that I won't forget it later on so that I don't need to see that uncanny bill.
Good idea. You still need to be careful. Behind the scene so many provisioning happens for which you may be billed. For example, if you create a managed instance group as a backend service for a load-balancer, then a network endpoint group gets created automatically behind the scene. You may delete the backend service later, but the NEG doesn't get deleted automatically.
Something I just cannot wrap my head around is WHY you would use a cloud service in the first place if you don't need the instant scalability. Heck, you might use your MBP or a Raspberry Pi at first and when your project starts out and you need to deliver a page quickly, just use a couple of geo-located VPS instead.
However I think I might be trapped in an old mindset here - I'd be really happy to hear why you would use the cloud in what I think is 95% of the use-cases. It just seems utterly expensive for what it is.
I used the cloud in my minecraft game cus my dad was paranoid about port forwarding
It might be because it is easier, and also can be started free. I suppose no one would want to run the database on their laptop and let it run everyday, or not anyone has a PC that can run it everyday, or not anyone had/would bought a spare device for it.
Sorry for my bad English.
This should be interesting, can't wait!
Thank you for making technical subjects interesting.
Congrats on 500k 🎉 🎊
great stuff, thanks for the heads up
This is a valuable info !
Just an addition to the topic of leaked IAM Keys: GitHub nowadays has a "scanner" which scrans every commit for known Keys (e.g. IAM) and immediately disables this key by telling AWS (this literally happens in less than a minutes) and sending you a mail that you f***ed up. I only know this for AWS but I guess that they also have this mechanism for e.g. GCP and Azure :)
Azure for sure as GitHub and azure are both from Microsoft.
I was doing a project on aws and I am happy I found this video before starting to use the cloud lol
I'm so scared after watching this
Thanks to Fireship
I've built a decent enough startup which can work on its own
Appreciate the free schooling dude.
That's why dedicated servers are better for messing around :)
We had a similar issue with AWS where a hacker got hold of one of an old emplyees api keys and setup a load of lambda functions. By the time it was noticed our bill was $15,000 for just a few weeks. Thankfully in the end Amazon refunded the whole lot, but it wasn't easy and was very scary.
Protect your API keys like your life depends on it and regularly switch them out and clear out unused ones.
my firebase bill this month was a whipping $0.03 hahaha
wow super cool!
I read an article about this specific event yesterday
consistently impressed by the quality of your videos! one of the best programming channels on yt
Man, I'm laughing all the way through! Awesome video! :)
Didnt know about Budget Actions, thx!
I appreciate the The Office clips you put in there 😂
Dear Fireship, THANK YOU for this PSA, from the bottom of my paranoid heart.
I made one in firestore. A document needed to be updated based on values in an update. I hadn't taken into account an old client, which would result in a calculation turning out to be NAN. One and a half hours later, with a SINGLE old client causing this, $40 was used. This could have escalated like a nut, if I hadn't seen it by random luck.
I made a mistake when using AWS gamelift during my bachelor assignment two years ago: i forgot a fleet running a game instance. I got $150 bill, but after contacting them it was deleted. Very good service, I can highly recommend AWS. 👌
At a convention, I was talking to a Microsoft employee, he was apparently a higher-up. He accidentally racked up a $5,000 bill just playing around in Azure. He talked to his boys and got the bill removed. All kinda scary!
Thanks for reminding me about the billing :D I am on AWS free tier and using it over one month but just realized that I have 0.62$ charge.
That time I accidentally deleted a whole kubernetes cluster instead of a single node pool on Google Cloud. It was scary how easy it was to delete it automatically. Thankfully it was a test cluster and it took only 2 days to restore it completely from scratch. But my face when I realized that I did it and that there was no turning back :D
5:00 I think a better term is a recursive call.
Technically it is an infinite loop (since there's always a main event loop somewhere), but an infinite loop is more conventionally a while or for loop that never ends.
No it's not. A recursive call is when a function is calling itself without a termination condition, and infinite loop is when a data structure doesn't have a condition that fulfills to terminate the cycle.
love this REAL story
Thank you. This video has caused me to paranoid check my AWS, Azure, and Firebase billings...
thank you for the tips
That’s why I use my own dedicated servers...
Good to hear how recursive function to not only hard to understand but can also cause disasters like this
Thanks for reminding me to check my billing accounts 🙏
I love the office memes in video.
I like the word cloud overflow xD
You talk about disasters in cloud while in the same day pretty much every google service was down for 1 hour
Exactly the way I planned it!
@@Fireship 👀👀👀
The best videos! 🔥
In fear of cloud overflow, I've been using compute engine mostly. It looks like its a flat rate, and using it is almost like using a raspberry pi.
Thanks!
Priceless Content
I love burning money
Yea especially *investors*
burn it on me
Interesting. I read this story on hacker news a few days ago.
google and amazon really needs to make a tutorial on this
I recently read an history about this topic. Creepy things can happen in the cloud 💀
And your bank account too 💀
"which is a brand new word I just coined" 😂
God I learn so much from your channel, thanks for creating these videos
I accidentally did this, but to a much lower level. It was still surprising to see my bill go from the usual $5 to $500 though
Set up alarms and stuff!
This is an old video but its noteworthy to add that I was in a similar situation like the startup thankfully they were reasonable enough to refund over 90% of the bill. I am now afraid of touching the cloud again. I still have depts to pay on AWS too
Waiting ❤️
30 years ago I saw a $140k bill for an isdn service. A data connection was calling a wrong number every 2 seconds, with a flagfall tariff.
Thank you for this. As someone interested in cloud computing you made me realise it can get costly very quickly and unintentionally
I just started a video serie on udemy and a free subscription on azure ... this video is freaking me out :-).
Good job 👍
Salesforce has limits on recursions, so if there are infinite loops, they just roll back the entire transaction and throw an internal error.
every application should have it. poor written apps don't. So hiring cheap developers lead to expensive bad solutions.
Now, I know why time complexity and space complexity is important.
cloud overflow.... this will make history
I burned $6000 on some azure lab machines. We had just achieved gold partner status and I reviewed the estimated cost and it said $0. But I was just in the wrong place looking...
Wise by this lesson a few years later I told a colleague to be careful with his Azure subscription. He got himself an enterprise sql Analysis instance. $7000 spent there...
I ran a BigQuery search on a publicly hosted dataset just to explore the data and get a feel for it. I kinda gave up after 20-30 mins because the latency/speed was really bad and I'm used to Pandas rather than SQL, so I wasn't really sure of what I was doing.
Got a $500 USD bill for search results that never showed up :)
The comedy is killing me 😂. I love this guy
When I was a newbie I put minimum instances as 1 for 30-40 cloud functions with 1 GB memory 😂 I was raking in $300-400 per day in bill 💸
That y in embedded system standard(misra c) doesn't allow recursive functions
I left a public IP up for a GCP server I had deleted once and ended up with a large bill. Called them and they refunded the whole thing.
Hi jeff , can you gather all "firebase" videos into one playlist please?
Awesome Video too funny.
The reason why we should choose either DigitalCloud or Linode
I had to check Firestore after watching this
Just wanted to write that azure supports spending limits if needed
I got so scared that I just took down a test project that I had deployed on vercel 2 days ago for my portfolio after watching this because i remembered that it had my fire store(free tier) config exposed in front-end. I dont know if what I just did was plain stupid or what but now I'll rather do my research on it first than blindly deploying it.
Jeff: I'm gonna go ahead an wrap things up there.
My brain: Nooooo!
It's never enough, your content gold, you wonderful person, you!
I understand your suggestion to not make infinite loops, but what if someone is just endlessly computing (with no scaling / explosion of computational complexity as described above) rather than responding to user events?
To take an artificial example, let's say I'm just getting a list of prime numbers. So I do while(true) { i++; if(isprime(i)) { log(i); }}. Is there an alternative? I guess I could do batches of numbers, with human intervention in between (e.g. every million iterations, store i, wait until human says proceed), but this doesn't seem to really solve any problem.
So... is there a problem with this? And if so, what's the alternative?
Michael Scott was so appropriate 😁
Recursive function is scary, it's not as obvious as regular loops... That's why database orm is good, they help you avoid a lot of edge cases
There is massive risk in using cloud so I cannot comprehend the rush to get there. If you are a startup it may be the quickest way. For established companies, you can reduce risk by having your own platforms.
AWS Cost Explorer is about the Services that you forgot to turn off after use..
I shutdown my RDS instance only to realise that it starts back again after 7 days.
OMG AWS indeed is sooo expensive. It's almost criminal.
Go serverless, we pay peanuts for running Lambda's over EC2's
F***ing Bezos *breaths in
Go for lightsail VPS, cheapest VPS in the earth after vultr. AWS is indeed the cheapest unless you configure something wrong and expose everything outside
wrong
@@mr_wormhole Nah hetzner.cloud or scale away are the cheapest hetzner gives you 2gb ram, 20gb ssd, 1 vcpu for 3 € no way to beat that
This has been highly helpful and eye-opening. On not running infinite loops, does it also apply to a while(true) loop that just polls messages, but it does not call another cloud function? Thanks!