Complete Python Pandas Data Science Tutorial! (Reading CSV/Excel files, Sorting, Filtering, Groupby)
Vložit
- čas přidán 15. 05. 2024
- Practice your Python Pandas data science skills with problems on StrataScratch!
stratascratch.com/?via=keith
Data & code used in this Tutorial: github.com/KeithGalli/pandas
Python Pandas Documentation: pandas.pydata.org/pandas-docs/...
Let me know if you have any questions!
In this video we walk through many of the fundamental concepts to use the Python Pandas Data Science Library. We start off by installing pandas and loading in an example csv. We then look at different ways to read the data. Read a column, rows, specific cell, etc. Also ways to read data based on conditioning. We then move into some more advanced ways to sort & filter data. We look at making conditional changes to our data. We also start doing aggregate stats using the groupby function. We finished the video talking about how you would work with a very large dataset (many gigabytes)
I realized as I upload this video there are some additional things I want to talk about in a later video. The first thing that comes to mind immediately is using the apply() function on a dataframe to alter the data using a custom or lambda function. If you have questions on this or anything else before I get around to making a part 2, feel free to write me a note in the comments.
If you enjoyed this video, be sure to throw it a like and make sure to subscribe to not miss any future videos!
Thanks for watching friends! Happy coding! :)
Join the Python Army to get access to perks!
CZcams - / @keithgalli
Patreon - / keithgalli
---------------------------------------------
Follow me on social media!
Instagram | / keithgalli
Twitter | / keithgalli
---------------------------------------------
Link to original source of data from Kaggle: www.kaggle.com/abcsds/pokemon
---------------------------------------------
Video Outline!
0:00 - Why Pandas?
1:46 - Installing Pandas
2:03 - Getting the data used in this video
3:50 - Loading the data into Pandas (CSVs, Excel, TXTs, etc.)
8:49 - Reading Data (Getting Rows, Columns, Cells, Headers, etc.)
13:10 - Iterate through each Row
14:11 - Getting rows based on a specific condition
15:47 - High Level description of your data (min, max, mean, std dev, etc.)
16:24 - Sorting Values (Alphabetically, Numerically)
18:19 - Making Changes to the DataFrame
18:56 - Adding a column
21:22 - Deleting a column
22:14 - Summing Multiple Columns to Create new Column.
24:14 - Rearranging columns
28:06 - Saving our Data (CSV, Excel, TXT, etc.)
31:47 - Filtering Data (based on multiple conditions)
35:40 - Reset Index
37:41 - Regex Filtering (filter based on textual patterns)
43:08 - Conditional Changes
47:57 - Aggregate Statistics using Groupby (Sum, Mean, Counting)
54:53 - Working with large amounts of data (setting chunksize)
-------------------------
If you are curious to learn how I make my tutorials, check out this video: • How to Make a High Qua...
*I use affiliate links on the products that I recommend. I may earn a purchase commission or a referral bonus from the usage of these links.
Hey ya'll! I created a second channel with more Python content (including additional Pandas tips & tricks).
Please consider subscribing 😊
czcams.com/users/techtrekbykeithgalli
You cleverly edited the code between 25:50 to 25:59 list(df.columns.values) to list(df.columns)😉👍
Error:Cannot mask with non-boolean array containing NA / NaN values - gives me error when usinf df.loc (on 40:49 in video)?
df.loc[df['Our Global Company'].str.contains('Smith', regex=True)]: this is code, I imported another .xlsx table when practising.
@@Vribejs go google it... you can't expect him to do it for you. He checked the documentation just to give us a good overview of pandas.... google out your error if not you will not learn.
i have been working on a excelworkbook having 8 worksheet and i m performing operations on data nd want to place dataframe in the 6 sheet in place of its data .but everytime i do all other sheets gets vanished nd a single gets get formed with the dataframe .plzz help me in appending df into an existing excel
Hey Keith , can can please help me to download the csv.file on an android tablet.
sorry for bad english.
I know this is 5 years old but I learned more about using Pandas from this one video than all the other videos ive watched on the topic combined! Just awesome! Thank you!
Glad that it is still helpful!!
This video was super helpful, thank you Keith!
In case anyone gets to the end of this video, around 48:00, Keith talks about the groupby operator and starts to go over the section "Aggregate Statistic using Groupby (Sum, Mean Counting)". You might run into errors due something that changed after Pandas version 2.0.0.
Instead of writing: df.groupby(["Type 1"]).mean()
Try writing: df.groupby(["Type 1"]).mean(numeric_only=True)
After version 2.0.0 the numeric_only value was changed to False versus True as it's default, causing errors such as "can not convert strings". Hope this is helpful, have a good one!
Thank you very much, I ran into the problem, this is really helpful! :)
Thank you
Thanks Man .
Thanks Dude
Was facing the same issue, thanks a lot.
This 1 hour video did more for me than entire semester of my Data Analysis course... Amazing
SAME DUDE omg
me too!
Same here bro
Were you paying attention?
am a self taught but this one saved me
Mannnn your one of the best Python go-tos PERIOD. Straight to the point and easy to understand. thanks for teaching us all!
One of the best tutorial that I've ever seen in CZcams! Thumbs UP!
Dude you deserved all the subs for this video alone. You explained everything so good. keep it up :)
I haven't started this yet, but based on your previous videos I know this is going to be great. Thanks Keith, you are a great teacher.
Wasted an hour watching a completely useless video on pandas, didn't understand a thing......
Then found this pure gold of a video, it really helped me a lot. Why didn't I click it earlier............
lol you had me in the first half 😂
glad it helped!
@@KeithGalli yeah, really nice job explaining it
Currently watching the other pandas video (real life problems)
Excellent!! I like the way you organize the videos on different topics and functions of working with data. Please make more videos on how to work data science in Python. E.g. Statistical analysis (descriptive statistics, t-test, linear regression) or data processing tutorial (like what we do in SQL).
this is an excellent tutorial, especially the filtering/conditional changes section. I have always loved how google sheets has built in queries, and I wanted to be able to do a lot of the same things using pandas. This essentially gave me all of the power I needed! thanks!
Keith
You are more than a teacher. Your level of simplicity in explaining Python in details is out of the moon. Keep up the good work. Your video is always my “go to” any time.
Again, thanks a lot for using your skills as a blessing to people around the world.
You break down all the details in a way that I can't believe this is for free. Very high quality stuff. I was up and running with this library in short order
Best Pandas tutorial on CZcams, especially 24:25
hahahahaa
LOL
jaajajajaaj
Huge hhhhh
29:19 is where he's getting texts from actual pandas
On point Keith. 5 hrs worth training covered in an hour. Made my day.
A strugling biologist here thanks you! We are mostly dealing with big data and it can get a little overwhelming, but you made it a lot easier!
Awesome
big data in a csv file? lol
Hey, you might enjoy SAS
they dont lie when they say data is everywhere and every field needs data scientists
Thank you, Keith, for making this super helpful tutorial. You're a great teacher!
I can't believe I watched this for free, thank you so much!
This was pretty good. I would also check udemy or r/learnpython for other free resources. Found a 30 hour FREE pandas course there the other day
www.udemy.com/course/the-ultimate-pandas-bootcamp-advanced-python-data-analysis/?couponCode=FF041817B54B4BC9EB6B
@@johnwiley1221 It's not free now, unfortunately :(
ki
The documentation is also free.
This is an amazing tutorial! Please keep publishing like this. very well explained!
I would love to see about matplotlib, numpy and if you can get inside machine learning
Great video to get people up and running. It took me two hours to watch, take notes, and test out some examples. I feel like this was time very well spent. Thank you for this.
Hi Keith - not sure you will read this but wanted to sincerely thank you for this tutorial. 3 years ago this was the first python video I ever watched after graduating from unrelated subject. Today I'm typing this from a business class lounge at JFK, on my way to London where I just got a job as a quant developer at a hedge fund, building pricing models and infra for trading. Worked hard for this but if not for your videos I could be at a very different place. Thank you from the bottom of my heart, your work means a lot to many people. Cheers!
Bro hire me
dude this is an amazing introduction to pandas. Really helpful, thanks a lot
Excellent Tutorial Keith. Very clear, at the right speed and interesting to learn from. This material is very suitable for a self learner. Keep it up.
I like the way he interacts with his viewers
Saved my day! I started learning Pandas, but when I missed several months during circumstances and this video about basics helped me quick comeback. Thank you!
This video helped my massively! Been learning through online python courses with people trying to act and saying unnatural jokes, but your video felt super natural and easy to watch. Many thanks!
thanks for useful video
If anybody have a problem with calculating the mean of Type 1 grouped data, use this:
df= pd.read_csv('modified.csv')
df.groupby(['Type 1']).mean(numeric_only=True)
instand of this:
df= pd.read_csv('modified.csv')
df.groupby(['Type 1']).mean()
That way, it won't include string-type data in the mean and sum functions.
thanks it helped a lot...can't understand the error while all the values are numreic already
so is it got updated now, since you can only perform the method on int or float columns ...
There is something to the way Keith teaches that keeps me coming back.
Besides being a good teacher and utilizing techniques which help people grasp the material quickly and remember for long time, he sends forth a wave of positivism. He is such a positive, energetic person.
Thanks for sharing your knowledge. May it grow and enable you to bless more people with it.
This is such high effort content. I was amazed that you actually went back and sped up the video where you said you would. 11/10 great tutorial. Thanks!
Been looking for a complete tutorial on Pandas. This is amazing! Thank you.
Comprehensive, perfectly paced.... Lovely tutorial!
I watched the entire video in 30 minutes and learned more than I did with hours of video content. Amazing work.
From the bottom of my heart, Thank you very much. May you never lack. May the elements, forces, and the entire Creation align itself for your own good.
Very simple yet comprehensive tutorial on Pandas. You had my attention throughout. I do use Pandas for data analytics along with numpy. That said I learnt quite a few tips and tricks.
Thank you for sharing your knowledge. Way to go Keith!
Liked and subscribed.
Awesome tutorial! One advice I'd have for any python developers is to get in practice of working within virtual environments. Really helps to avoid conflicts when you're working on a project which may require some older versions of a library but your other projects may require latest ones, stuff like that.
Thank you so much for your time and effort. This is the best python tutorial I have watched. Straight forward and well organized. I appreciate the time stamps.
Between jobs for the first time in decades I wanted to learn data science using software other than just Excel and Access. Your video was well explained and frankly better than anything else I have seen so far involving Python and Pandas. Thank you for a job well done.
WOW! This was just what I have been looking for! Fantastic tutorial! You explained everything very well and clear from start to finish. Best Pandas tutorial on youtube for sure! Thanks man :)
Like, seriously, The best of the best Pandas Course I have done so far! Starts off easily and basically and the explodes!
This 1 hour course is all I need for my data analysis course. This is the best video I found on CZcams. Thanks ❤️❤️❤️
Excellent tutorial; exactly what I was looking for. Liked and subbed. Thank you for sharing your expertise.
Great video! One of the best pandas tutorials I've seen.
I have one comment though. When you run (at 40:00)
df.loc[df['Name'].str.contains('Mega')])
You are actually including Meganium in this filter, even though it is not a Mega pokemon. So, one needs to include a space after Mega, such as:
df.loc[df['Name'].str.contains('Mega ')])
One can see that this makes a difference because when you run
len(df.loc[df['Name'].str.contains('Mega')])) and len(df.loc[df['Name'].str.contains('Mega ')])), to know the number of rows, there are two distinct outputs (respectively 49 and 48)
Omgeeeeee!!!! Thank you so much!!! I've searched sooooo many videos trying to help with the delimiter problem I've had (i didn't know that was the problem) and you're the ONLY one I've found that even mentions it!!! 🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾
wow! this is amazing, Thanks a lot! I love how things don't go as planned and you just find a way, it adds so much to the video. Amazing tutorial!
Ok I've been learning Pandas for a while now, over many different sources, and this one video has shown me much more helpful little hints and tips than all of the other material I've looked at previously!!! Thannnnnk you! Please do more Pandas stuff as this has been so awesome =]
Coming from the R environment, I must say this is an excellent tutorial to learn about Pandas. I'm very happy to learn that the tools I use in R for data management can be implemented in a similar way in Python. Thanks for taking the time to put this together! Great job.
Same here
I agree - coming to Python from RStudio and after looking at videos all day this is definitely the most helpful and intuitive video!
sometimes the syntax may be getting confused for python and r right? if you use both
can someone tell me why is r so encouraged in the data science/analysis circle when python can do everything and more and it is so intuitive
@@manan-543 I think Python is far more general and overall can do a lot more, but in my field, packages associated with statistical models are far more abundant in R than in Python. For example, I'm not sure Python comes even close to R for the implementation of Bayesian hierarchical models, GLMMs, GAMMs, etc. Also, methods papers often publish packages in R, so it seems to remain the default for statistics. Until the statisticians start switching in large numbers I'm not sure this is gonna change anytime soon; and when it does, it probably will be Julia, not Python.
Your iterative approach is very effective to expand concepts without overloading. Great job. On to the next one.
I have my final exam in my data structures course soon, this was a great way to study and work through things my class covered months ago, thanks so much!
The best pandas tutorial, best mentor having strong grip on subject
Makes me want to play the old Emerald games again, wonderful tutorial, keep them coming
The best tutorial I have seen so far on data analytics. I now see how python/pandas helps in data analytics. Thank you very much for making and sharing this video.
Keith, you are the best Python instructor! Very easy to follow. Thank you!
I have bought multiple Udemy courses on pandas and this one blows them all out of the water, and it’s free! I’m deff subbing!
Great tutorial. I'm just starting with Python, and this is a great video for picking up a lot of knowledge fast. You asked for suggestions about other videos. I'd definitely like to see more tutorials that are about cleaning data. That's the hardest part, I think. The most laborious and time consuming. I'm learning the re module (regular expressions) for Python, but it's going SLOWLY.
Keith, I stumbled across your video from random search on deeper understanding of pandas . I felt like you did a great job presenting the material. Well done man, it was easy to follow and understand. I did appreciate the part at the end with the chunk size and group by - would like further explanation of those concepts. Thanks.
Started my PhD in hydrogeology and learning Python from the scratch. I love your work, keep it up!
Dude... you should make more videos... you are a natural born teacher!!
Wow, thanks for this tutorial. I'm starting on python and took a course of udemy, but it was confusing, with your explanations many doubts are cleared up. Thanks Keith:)
I swear this is the most useful python channel on CZcams. Top stuff.
You are the GOAT. Your explanations using Pokemon makes so much sense.
Awesome video Keith! I'm a beginner programmer but your explanation is super clear! Thanks for the videos:)
Thank you Keith for this video, absolutely amazing and valuable for many! THANK YOU!
Glad you found it helpful! :)
What an excellent video. I have watched so so many tutorials, but this is definitely one of the very best.
I have been learning python and using pandas for about 3 months now and done innumerable searches on the internet with questions regarding use of specific statements and coding. I wished I had come across your video earlier! You are a born teacher and know how to layout and explain complex terms and concepts. How can someone that looks so young have such a strong grasp on presentation and user needs? The concepts you explain are the same things I have sought information on for 3 months but all in one place and succinctly explained. Thank you for all your work.
I just went through your numpy tutorial. And that's the reason I come here. Thumb up!
Appreciate it!!
I came for the tutorial, stayed for the cutesy pokemon stuff, really warmed my heart
This was such a good walk through for Pandas covering so much information. Thank you so much, I hope you continue to do more videos.
No BS, To the point!
Man, probably the best tutorial out there. 🔥🔥
You have a sub right here. Thank you so very much for explaining this like no one ever has!
Awesome tutorial Keith, I learnt a lot by following your hour long tutorial.
Created a new notebook instead of using the GIT version as it doesn't show what happens before you commented the code.
I've been looking for a good pandas and python video for quite sometime now. I have to say that this is really amazing. You've explained it so well that a beginner like me could easily understand. Great job and thank you. Can't wait for more videos. (if possible, matplotlib)
Thank you Keith for an amazing crash course! I've filled in the gaps I've had with Pandas and now ready to apply this library in my work project! Many thanks!
this is my second python tutorial follow along, thanks man u explained well, its appreciated
53:30 you can use .size() to get the count of each Pokemon type instead of adding a new column.
It would look like this:
df.groupby(['Type 1']).size()
Great tutorial!!
This tutorial helped me alot. Thank you so much!
you've been really helpful mate, been struggling with pandas/numpy and finally im getting somewhere! more of this please!!!
Love the way you say 'axsending'.
Great tutorial!
Wow man! Holy smokes that was such an amazing breakdown. I came into this knowing nothing about Pandas and now I want to get back to work with my personal data! Thank you so so so so much. I’m off to find the documentation!
Glad you enjoyed! Your comment made my day :)
This was such a great introduction to pandas and on DataFrame. This is exactly what I was looking for.
Since I hadn't previously downloaded pandas onto my mac, and didn't feel like installing anaconda either, I was running into some troubles installing pandas with just "pip install pandas" so I thought I would include the instructions as to how I did it.
simply do:
pip install pandas --user
If nose and tornado aren’t downloaded do:
pip install nose --user then pip install tornado --user (nose needs to be installed first)
then terminal also suggested I add it to my path, so I did:
sudo nano /etc/paths
add the path at the end of the file
do ^X and then Y then hit enter
Day 1 on my journey to learn data analysis with python, this vid and kaggle's free pandas course is just what i needed to give me more motivation to keep learning.
This helped me so much Keith thank you. Work does not want me to use excel anymore for any data frame so I have to learn python. I have spent nearly a month trying to learn, even something as simple as adding a data set to python I was getting wrong. Now I am so confident with the work I have produced.
Thanks for posting! As an MIT student taking a data analysis class, this video was very helpful, more useful than the other tutorials online!!
Found it very useful too!
"As an MIT student"
Weird flex but ok
Video Outline!
0:45 - Why Pandas?
1:46 - Installing Pandas
2:03 - Getting the data used in this video
3:50 - Loading the data into Pandas (CSVs, Excel, TXTs, etc.)
8:49 - Reading Data (Getting Rows, Columns, Cells, Headers, etc.)
13:10 - Iterate through each Row
14:11 - Getting rows based on a specific condition
15:47 - High Level description of your data (min, max, mean, std dev, etc.)
16:24 - Sorting Values (Alphabetically, Numerically)
18:19 - Making Changes to the DataFrame
18:56 - Adding a column
21:22 - Deleting a column
22:14 - Summing Multiple Columns to Create new Column.
24:14 - Rearranging columns
28:06 - Saving our Data (CSV, Excel, TXT, etc.)
31:47 - Filtering Data (based on multiple conditions)
35:40 - Reset Index
37:41 - Regex Filtering (filter based on textual patterns)
43:08 - Conditional Changes
47:57 - Aggregate Statistics using Groupby (Sum, Mean, Counting)
54:53 - Working with large amounts of data (setting chunksize)
Thanks for watching friends! :)
Let me know if you have any questions
YES!!! THANK YOU!
Thank you so much for posting this! I have a test in Python soon, so I've been watching this for a review. You explain everything so well and make it easy to follow. I also like how the data was from Pokémon - it makes it more relatable.
great tutorial
A reference notes to help you while you watch the video.
docs.google.com/document/d/16qcfjwLp1vV-5VnIOGuDC2vxkHQ534_RzQd2Gihk7x8/edit?usp=sharing
Hey dude, love this video by the way but I have a question, can this data be used for machine learning? I have my exams coming up where I have to find a dataset to make predictions and stuff. Are these pokemon cards, do they have label and features if you understand what i'm talking about? Any help would be greatly appreaciated. Thanks in advance.
Bro I started a data science internship in the beggining of the Year, we use a lot of pandas and you are saving my life from day 1.
Thanks again, you are a god send! Subbed on both channels, cheers!
This tutorial is built so well, I keep going back to this. thanks
27:15 It seems that the dataframe got scrambled up a bit there, most likely from having the cell running multiple times. Even when there was an error message, it appears that either the Total or the Legendary column was moved to the left of HP. Upon running the cell again (with the corrected version?) it calculated a new Total adding the previous values and generating corrupted results.
This is an extremely usefull tutorial. You explain so good bro. Thank you very much. Like and subscribed. Hugs.
i watched more than 10 different videos about pandas, this is the most easy and understandable one. Worth your time!
literally one the most useful videos on pandas ever
Great job dude.
finally.. a new video... I was waiting for a Long Time😍😋
Thanks for this content. I really apriciate when people pass their knowledge ahead. I'm starting with Data Analytics, after getting to know the basics of Python. Your video helped me a ton! Hope we get to see other videos with panda a cases of use.
Thanks Keith for this video! I'm new on data science I'm using your videos to practice and learn a lot more. Really thanks!
When I start making money with these knowledge, I'll give you some share!
SQL person w/ limited exposure to Python here. This was useful as hell.
The is one of the best python videos that I've seen online. Thank you.
Thanks so much, you are a life saver. I have been stuck at running just pandas since Monday.
I learned so much, thank you. Then at the end...that music tho. I lost it! LOL! Did not see it coming.
In the chunksize section, you pick a well-documented bad practice, namely calling pd.concat inside a for loop. As the loop runs repeatedly, this operation becomes more and more expensive (because new_df gets longer and longer). Per the pandas documentation, the better approach is to append each df to a list and then pd.concat the list elements just once, after the for loop.
Hello, can you please provide with a tutorial for that? Quite new and clueless here.
dataHere = []
for chunk in pd.read_csv('modified.csv', chunksize=5):
dataHere.append(chunk)
newnew = pd.concat(dataHere)
This looks right?
This is so great. I appreciate you taking your time to do this. It takes the mystery out of Pandas and got me started. Thanks
This the best Pandas videos I have ever watched. Thank you so much for this.