Day-2 | Kubernetes Troubleshooting | CrashLoopBackOff with 3 real time scenarios including OOMKilled
Vložit
- čas přidán 17. 04. 2024
- Grab the GeeksForGeeks Three90 Challenge - bit.ly/3UgkXZQ
Support my work
www.buymeacoffee.com/abhishekprd
GitHub Repo
github.com/iam-veeramalla/kub...
- CrashLoopBackOff
When you see "CrashLoopBackOff," it means that kubelet is trying to run the container, but it keeps failing and crashing. After crashing, Kubernetes tries to restart the container automatically, but if the container keeps failing repeatedly, you end up in a loop of crashes and restarts, thus the term "CrashLoopBackOff." This situation indicates that something is wrong with the application or the configuration that needs to be fixed.
- Common Situations of CrashLoopBackOff
The CrashLoopBackOff error in Kubernetes indicates that a container is repeatedly crashing and restarting. Here are explanations of how the CrashLoopBackOff error can occur due to the specific reasons you listed:
- Misconfigurations
Misconfigurations can encompass a wide range of issues, from incorrect environment variables to improper setup of service ports or volumes. These misconfigurations can prevent the application from starting correctly, leading to crashes. For example, if an application expects a certain environment variable to connect to a database and that variable is not set or is incorrect, the application might crash as it cannot establish a database connection.
- Errors in the Liveness Probes
Liveness probes in Kubernetes are used to check the health of a container. If a liveness probe is incorrectly configured, it might falsely report that the container is unhealthy, causing Kubernetes to kill and restart the container repeatedly. For example, if the liveness probe checks a URL or port that the application does not expose or checks too soon before the application is ready, the container will be repeatedly terminated and restarted.
- The Memory Limits Are Too Low
If the memory limits set for a container are too low, the application might exceed this limit, especially under load, leading to the container being killed by Kubernetes. This can happen repeatedly if the workload does not decrease, causing a cycle of crashing and restarting. Kubernetes uses these limits to ensure that containers do not consume all available resources on a node, which can affect other containers.
- Wrong Command Line Arguments
Containers might be configured to start with specific command-line arguments. If these arguments are wrong or lead to the application exiting (for example, passing an invalid option to a command), the container will exit immediately. Kubernetes will then attempt to restart it, leading to the CrashLoopBackOff status. An example would be passing a configuration file path that does not exist or is inaccessible.
- Bugs & Exceptions
Bugs in the application code, such as unhandled exceptions or segmentation faults, can cause the application to crash. For instance, if the application tries to access a null pointer or fails to catch and handle an exception correctly, it might terminate unexpectedly. Kubernetes, detecting the crash, will restart the container, but if the bug is triggered each time the application runs, this leads to a repetitive crash loop.
Here are some more very useful Zero to Hero playlists on the channel.
- Free DevOps Playlist: • DEVOPS ZERO TO HERO CO...
- AWS Zero to Hero Playlist: • AWS Zero to Hero - AWS...
- Terraform Zero to Hero Playlist: • Terraform Zero to Hero
- Python for DevOps Playlist: • Python for DevOps
Have any questions while learning ? Don't worry, Join our Doubts Clearing Group.
Join our Doubts Clearing Group
/ @abhishekveeramalla
About me:
========
Instagram: / abhishekveeramalla_off...
Telegram Channel : t.me/abhishekveeramalla
LinkedIn: / abhishek-veeramalla
GitHub: github.com/iam-veeramalla
Medium: / abhishekveeramalla-av .
.
Disclaimer: Unauthorized copying, reproduction, or distribution of this video content, in whole or in part, is strictly prohibited. Any attempt to upload, share, or use this content for commercial or non-commercial purposes without explicit permission from the owner will be subject to legal action. All rights reserved.
Buy any GFG course before 30th of April and complete 90 percent of the course to avail the 90% discount on the course
Use the below link to grab
bit.ly/3UgkXZQ
you are the pillar for devops engineers. You are my mentor
Thanks
Whatever I learned in the context of Kubernetes because of your inspiration . Being a non Technical background profile its always a challenging one to grasp new things . But your hunger to share things in such a simplified format is really commendable . Last but not the least is practice with your video makes amazing . Kudos and always thankful to you for what you taught us Abhishek bhai .😊❤👌👍
❤️❤️❤️🙏
Abhishek is a genius and legend...
Excellent Explanation. Its a very common issue and the way you explained deserves kudos !!!
Awesome explanation
Please continue the series
Thanks Abhishek for your great efforts to make such an awesome video especially Kubernetes troubleshooting. It’s really very important for devops engineers to their daily activities.
Probably the only person with this level of patience, humbleness and passion in sharing knowledge and for FREE!!!! just wondering in a year or two how crazy your channel is going to be and all the private guys will be behind you with huge quotes!!!!! Loads of love brother Abhishek from all your student community.
❤️❤️❤️
Informative and great explanation , Thanks for the hardwork
One of the best video. Thanks a lot Abhishek.
Most welcome!
Hello Abhishek, Your videos are really helpful, I have an interview scheduled on coming Tuesday and referring to your content is very helpful.
Best of luck!
Thanks Abhishek for starting this series now waiting for Ansible
Thanks 🙏
Thank you @Abbhishek 👏
Welcome 🙏
Awesome content. you're the best mentor in the industry .
❤️❤️❤️
@@AbhishekVeeramalla Hi Abhishek, currently i am searching job and i am still in the completion stage of AWS and Kubernetes projects and practical hands-on, do i have to prepare the troubleshooting as well as i feel so overwhelmed now with aws, Kubernetes altogether and i have less time for bagging a job, i have set a target to myself by end of may i have to land a job so what would be your recommendation? should i just focus on the aws and kuberentes playlist now and later troubleshooting as this video did help me understand better with wrong image issues and how to debug as it was correlated. But now in short span of time do i have to include this also? kindly help me here with your suggestion please
Thank you Abhishek 👍That was well explained
My pleasure 😊
Thank you Abhishek you are a leader, you are a light for now and next coming generations. No words for your contribution to the nation. This is also good service to the nation I felt.
Thanks alot
Great video!! Thank
You Abhishek
Welcome 🙏
Very useful. Thanks a lot Abhishek.
Most welcome!
Very useful. Thanks a lot Abhishek. You are really a nice person who shares the knowledge without money !!😊
My pleasure 😊
Hats off Abhishek..
Thanks 🙏
Really Appritiated Sir
Thank you Abhishek .
❤️
Thank you very much on your efforts , i really suffered from that kind of errors , i know understood the reasons behind it
I went through this is very interesting thanks a lot no way we can't have the offer later with this new interesting series. Please waiting for the next session 😊
💚💚💚
appreciate effortsss sir
Thanks a lot bro, forever grateful to you🙏
My pleasure!
You are awesome and I admire you a lot!!!!!
Thanks Navya
Hello Abhishek, I must say this "I Got Job Offer" 4 days back I had interview and your knowledge transfer helped me to crack interview. Company hired me with good package , joining bonus and equity. Thank you might be the small word I can gratify now. More powers to You🙌. Your contribution to the society is something beyond imagination and enormous in this generation. Everybody is running behind the money instead you're doing it for free which is helping peoples a lot. I wish you more and more powers with good health to You and Your family. My inspiration🙌🙏
This is amazing news Navya. Extremely happy for you ❤️ can you reach out to me on Linkedin, so that we can talk more about the job interview process and if possible a podcast ?
@@AbhishekVeeramalla Sure Abhishek it'll be my pleasure to talk to you 🥰
Day 2 understood successfully
🤩🤩🤩
Thankyou Abhishek :)
💙
Very useful bhai...
Great content Abhishek. In case of crash ,kubelet restart container not the pod, pod status or age will not change. This is obvious incase pod has multi containers
Yes I mean to say container, just lost with the flow 🤠, Thanks Santosh
Hello Abhishek,
Thank you so much for all your efforts. I have followed almost all your videos.
There is one request, if possible can you please make Maven Zero to the Hero series also.
Eagerly waiting
Thank You Bro ❤
41:40 Hat's of to u r dedication man.... but I feel like u recorded this video at early in the morning with fresh mood thanks a lot btw one doubt how to fix livenessProde crashLoopBackOff error
You should inform the probe failure to developer
I can see that the way you learn things is different way that so precise ! Can you give any suggestion or do any viedo on how to learn any new concept
Thanks alot
Great videos 🎉
Thanks! 😃
Hi Sir.. Thanks for Videos. Your Videos are so informative and easy to understand.
Please make video on Bitbucket and Jenkins Webhook integration so build will get trigger in Jenkins after commit happened in Bitbucket Repo.
2)Once build start then how to get Chat card messages in Google chat for the status of builds. Which are running in Jenkins
sir tussi great ho. thank you for such informative video
Welcome 🙏
Eagerly Waiting for Day3
Thank you sir
Thank sir, I used to have a hard time with this error
❤️
Thank you ^^
Noted all troubleshooting steps
❤️
Nice video
Thanks for the great content Abhishek ❤
Request you to please post more troubleshooting vidoes on regular basis, It would be great if you run describe / event commands and explain how to troubleshoot
Sure 👍
Hi bro , Many thanaks for the sharing valuble info .. Could you please add more scenarios to increase the CPU, memory, and storage of the nodes? SSL/TLS-related errors
Thanks for the video. Could you please also explain how to set up alerting to teams/slack for the condition of deadlock pods.🙏
bro, please do one complete video for Livness, Readiness and startup probes.
Sure
Thanks for making videos Abhishek, could you please also make video for ingress and ingress controller issues
Welcome
Tq bro❤
Welcome
Crashloopbakoff status- if your pod crashes in loop.
Pod has container, container has application running as process.
Deployment manifest applied.
If pod, container and process are running, it will show status running. After a while maybe pod didn't get enough resources. Pod moves to crashed state, Kubernetes will again try to restart pod due to its default restart policy.
Increase restart time in incremental way till span of 5 minute, delay is called backoff delay.
Crashloopbackoff reason, miscommunication, wrong env variable, persistent volume that doesn't exist, liveness probe/redines probe fails, memory limits not enough, wrong command line arguments, bugs and exceptions in application code.
Supreme leader button >> you're doing a good job abhishek,
Please share your daily schedule as how you make time to make videos and doing job ?
❤️❤️❤️
Waiting for the new video please keep posting
we are waiting for Dy 3
Please add more troubleshooting videos for all the components of kubernet elements and kubernets cluster
Will try my best
I am new to DevOps and i watched many of your videos about kubernetes and jenkins etc. and guess what i am facing this exact problem CashLoopBackOff these days . i am trying to deploy my simple crud application which uses mysql database in the cluster. i haev a special request to you , can you please make a video about deploying a simple crud java spring-boot application which uses mysql database ? thanks. and second thing is all your videos are very helpful. thanks for all the tutorials.
Thanks and will try
Hi, very nice session. Thanks a lot for that. But in the first scenario you already knew that there is a mistake in dockerfile and that is causing crashloopbackoff. In actual environment, how do you trace back to docker file and identify that the culprit is in dockerfile?
Hi abisheck in the jenkins if master node reaches the thresholds in that scenario in ream time what does they do?
Hey Abhishek. I have recently started following you. You made amazing knowledge videos. Can you help me with freelance work on aws? Really want hands on experience
Please watch the previous video with Cloud champ where I explained how to get started with freelancing
Can you please create video about pod affinity and cluster auto-scaler
Sure
Hi Abhishek, can you show how to handle the env that using inside applications that deployed in eks.
Hi Abhishek, Good content as always, Thanks ! Can you consider creating a course on RedHat OCP (RHOCP) cluster basics/bring-up/architecture details/debugging etc ?
It will be the same k8s troubleshooting
@@AbhishekVeeramalla - Thanks. Sorry, I could've asked more precise question. I meant, can you consider creating a separate playlist like you did for public cloud (aws/azure) zero to hero, for private / on-prem cloud especially RHOCP ?
Hi Abishek when is the next video releasing in this series? Eagerly waiting
Soon
@abhishekVeeramalla I m not having laptop hw to study these devops any online compiler is there
You have some playgrounds like Katacoda
Is there anyway that we can see the path details where persistent volume of kubernetes is mapped to physical volume path (Could be worker node storage / cloud storage / local storage ?
Hi, Please share a video about SRE in modern AI technology.
Hi bro one query here you have created pod now and run the command kubectl get pods -w so then you came to know it is Oom killed but if suppose pod went in crashback one or 2 days back how I can know oom error is causing it
Hi @AbhishekVeeramalla
In organization, how they would maintain dabases(MySQL or mongo) for 3 tier application, would they maintain in kubernates or directly they use in Cloud and which approach is recommended way for maintaining database tier either in Cloud or kubernates..?
In k8s cluster, you can see my e-commerce app demo
Sir, what does the mean container status =exited, when I run simple docker image i.e print hello world used in docker zero to hero Playlist. When I run docker ps - a it shows container status =exited
❤❤❤
:)
Hi Abhishek do devops jobs require high availability
Hi Abhishek, thank you for the video is very helpful, but how can we check all these errors in the log to know exactly what is wrong with our pods.
Welcome. You can use kubectl logs command
Hi Sir
I'm currently in my 3rd year (6th sem) of engineering in CSE in Hyderabad and I'm considering learning about DevOps or AWS. Given your expertise and experience, I was wondering if you think DevOps would be a good option for someone in my position?Do you think it's a good fit for someone at my stage? Your insights would be incredibly helpful.
Should I choose DevOps or AWS
Any insights or advice you could share would be greatly appreciated.
Thank you so much for your time.
Yes you can pick DevOps
Hi Abhishek, can you please let us know how to troubleshoot due to which scenario crashloopbackoff scenario is coming?
If u do kubectl describe on the pod you will see the errors which I am talking about
Hi bro can u suggest front-end Developer videos like ur content
How to write/build a Docker File? Is there any video please send me the link? I am in preparation of Interview.
Check the docker zero to hero playlist
Hi abhishek, what if kubectl logs and kubectl describe command doesn't gives you enough information to troubleshoot further, next where I can go and check for further debugging, kindly assist on it..!
sir what should be propr roadmap for devops on your channels for very begineers like which playlists should watch in sequence
Hey, u can check the roadmap video
@@AbhishekVeeramalla sir I have not even done programming language yet so it's showing
Devops 0- hero
AWS
Teraform
@@AbhishekVeeramalla sir is it necessary to do web dev before devops
When next episode coming for kubernet troubleshooting
Anna ee CNCF, KCNA , DEVOPS ALL 3 SMS DOMAIN AHH, or CNCF ,devops seperate tracks ahh please explain this anna
Ardam kale bro ne question
Please help me understand one thing, may be i m missing something, it may be silly but, why are we pushing Docker image to docker hub ?
Hi 👋, because the image is on your laptop not on the k8s cluster. You should watch docker videos from devops zero to hero for learning docker. Its very useful ❤️
Got it sir, thanks a bunch ! I ll see them @@AbhishekVeeramalla
Please boss man I know you May ne occupied. But first all big tanks for everything only gif can paye you. You are doing a lot. But please if you have a space continue with this séries
sir we have kubernetes, docker, terraform, in devops then why we are using ansible? can please clear out my doubt?
hey, ansible is used for configuration management which other tools can't do.
If your pod crashes and this happens again and again in loop, it throws crashloopbackoff error
On kubectl get pods.
❤❤🎉
❤️❤️❤️
Hi Sir, How we can get the Redhat Openshift console IP? & is it possible to access the console with the IP instead of URL?
it depends on how is the console service configured. If you want to learn about Kubernetes Service Types, you can watch my Kubernetes Services video in the k8s zero to hero playlist.
@@AbhishekVeeramallaThanks for the response sir
exec /opt/java/openjdk/bin/java: exec format error
i got this error while deploying CICD Spring boot application using argocd on k8s cluster.. Im using MacBook M1 and i also installed java in my machine and setup environment variables also ..even though im getting the same error...the status of the pod is also "CrashLoopBackoff"... Waiting for your reply brother❤
Yes because you are using wrong jdk. Please use the jdk for arm
2024-04-19 11:20:43 [✖] getting Kubernetes version on EKS cluster: error running `kubectl version`: exit status 1 (check 'kubectl version')
when i am trying to do 2048 game project while cluster creation time i am facing this error,what is the problem abhishek.could please reply me
On your laptop you might have an old or unsupported version of kubectl
Sir actually i am doing this project on EC2 instance (t2.large),and I installed kubectl form documentation itself, pls help me sir.
@AbhishekVeeramalla I am getting crashloopbackoff error with exitcode 0 I watch your video for troubleshooting crashbackoff error but none of the case is there which you explain in video
Exit code 0 means error in the application itself. You can use kubectl logs command to understand the error in the app.
I want to Learn aws but your playlist is confusing....can you please tell me how to learn order wise am i starting....
follow the roadmap video on the channel.
watch the roadmap video on the channel. You will get a clear understanding
Hi Abhishek I need devops full course in Telugu version bcz I can't understand completely but want to learn from you plz iam eagerly waiting for your response
Ipudu motham telugu cheyadam chala kastam andi .. End to End DevOps Course 100 + videos vunnay.
Hi Abhishek how many episodes are going to be in this series
These episodes take a lot of time. So I am thinking to do as per the interest of people.
@@AbhishekVeeramalla We are more interested in troubleshooting k8s issues. Please make more videos bro.
Please do full series @@AbhishekVeeramalla
Or else abhi what you can do is you can make a video on all the troubleshooting issues and give us just summary, some thing like a definition and we can learn on our own, just a suggestion
@@AbhishekVeeramalla please we are really interested so please do full séries
Please complete this series