Daniel Chen: Cleaning and Tidying Data in Pandas | PyData DC 2018
Vložit
- čas přidán 25. 07. 2024
- PyData DC 2018
Most of your time is going to involve processing/cleaning/munging data. How do you know your data is clean? Sometimes you know what you need beforehand, but other times you don't. We'll cover the basics of looking at your data and getting started with the Pandas Python library, and then focus on how to "tidy" and reshape data. We'll finish with applying customized processing functions on our data.
===
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.
PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.
0:00 Introduction
0:18 Setup: Github Repo, Jupyter Setup
5:35 Loading Datasets - panda.read_csv()
7:43 Dataset / Dataframe At A Glance
7:53 Get First Rows: df.head()
8:58 Get Columns: df.columns
9:15 Get Index: df.index
9:37 Get Body: df.values
10:46 Get Shape: df.shape
12:04 Get Summarizing Statistics: df.info()
13:12 Filtering, Slicing a Dataset / Dataframe
13:25 Extract a Single Column: df['col_name']
14:12 Dataframe vs Series
14:41 Extract N Columns: df[['col1_name', 'col2_name']]
15:51 Panda's Version: df.version
16:26 Extract Rows: df.iloc
17:30 Extract Rows: df.loc vs df.iloc vs df.idx
18:45 Extract Rows: df.iloc
19:37 Extract Rows: df.ix - Deprecated
20:38 Extract Multiple Rows and Columns
22:00 Extract Rows using Boolean Subsetting
23:24 Extract Rows using Multiple Boolean Subsetting
24:55 Cleaning a Dataset / Dataframe
25:38 General Issues according to a "Tidy Data" Research Paper
29:45 Issue 1: Column Headers are Values and not Variables Names
30:19 Load Pew Dataset
32:55 Transform Columns into Rows: pd.melt()
36:59 Load Billboard Dataset
37:05 Transform Columns into Rows: pd.melt()
42:00 Issue 2: Multiple Variables are Stored in 1 Column
43:06 Load Ebola Dataset
46:22 Transform Columns into Rows: pd.melt()
47:14 Split Column using String Manipulation through Accessors
51:19 Extract Column / Series from Accessor Split: accessor.get()
53:13 Add Column to Dataframe
54:13 Contracted Form for pd.melt() and Accessor String Manipulation: pd.merge()
56:10 Issue 3: Variables Stored in Rows And Columns
56:25 Load Weather Dataset
58:30 Transform Columns into Rows: pd.melt()
1:1:00 Transform Rows into Columns
1:2:00 Transform Rows into Columns: pd.pivot() vs pd.pivot_table()
1:4:30 Transform Rows into Columns: pd.pivot_table()
1:6:19 Flatten nested / hierarchical table: pd.reset_index()
1:7:42 Issue 4: Multiple Types of Observational Unit in Same Table (i.e De-nomalized Table)
1:9:43 Extract Type Observational Unit in new Dataframe, Drop Duplicates
1:11:30 Create "key" for extracted observational unit dataframe
1:12:11 Save new dataframe: pd.to_csv()
1:13:22 Merge / Join dataframe on common columns
1:16:25 Randomly Sample a dataframe
1:17:15 Note on Memory Consumption between all 3 dataframes
01:18:25 Summary from "Tidy Data" Research Paper
01:20:06 Q&A
01:21:21 Q&A 1: Simulating R's Chaining in Python
01:24:49 Q&A 2: Best Practices on Braquet Notation vs Chaining
Huge s/o to github.com/KMurphs for the video timestamps!
Want to help add timestamps to our CZcams videos to help with discoverability? Find out more here: github.com/numfocus/CZcamsVi... - Věda a technologie
Time stamps :
32: 57 : Melt function for Transposing columns and Rows.
42:17 : Handling multiple variables
48 :11 : Split Function
56: 20 : Variables stored in both Rows-Columns.
1:01:10 : Pandas Pivot table
1:06:11 : Reset_index() function
1:10:23 : Drop duplicates function
the video is timestamped
This is the single most helpful python pandas tutorial that I've ever seen. Thank you Daniel Chen. Great job!
you could drop the word “single”
Excellent lecture! Thanks for letting it public!
Great video, that gets to the point and explains complex concepts so elegantly.
One of the best data cleanih and tidying tutorial on CZcams
Well explained and easy to understand . I appreciate your help so much. Many thanks
This is such a wonderful tutorial.
Simple and easy to follow. Completely new Python student and this was such a great introduction, eager to continue learning! Thanks a lot Daniel
Those timestamps were well made and very helpful - thank you!
I'm just 10 mins in and I already love you. Thank you for this presentation
Great presentation! Thank you very much for this helpfull and understandable-easy to follow Tutorial.
Great tutorial for beginners like me. Thank you
A great tutorial. Thanks!
Very excellent and explanatory in detail
Thanks for the video. I started with pandas and got confused with the brackets, it is clear now.
Awesome explanation and super content @Daniel !
This is a terrific video, Daniel. Thank you very much!!
Great video. Underrated, isn't recommended by youtube. Btw thanks for the video.
So useful, thanks.
it was very useful, thank you so much
Excellent tutorial
Excellent Tutorial about pandas and Data Cleansing and Data preparation
simply wow !!
Great tutorial
Very nice talk, I finally got it. Seems like a nice dude also
Excellent really learnt a lot 👍👍👍
great tutorial
Amazing tutorial thanks man
Awesome 👍
Thank you. Very Very helpful
Good skills I've learnt here
Thank you^^
جميل جدا ورائع جدا ورائع جدا
Thank you
Thanks you 😊
Ver good Aman
A bit lengthy but worth it. Lots of good stuff in it.
where is the URL to download the data? I missed it somehow. Thank you.
Newly starting Python. I realize that Data Manipulation is Far easier in R (Use tidyvarse)!!
But how do you melt all the other columns in the pew dataset? You just demonstrated the
hey guys where can I find the relevant datasets from the tutorial.
somebody please let me know
(TIA)
Hello, please how do we get the datasets you used in the video? Could you share a link we could download from? Thanks.
@KKC, Sir, right at the beginning, 0:04:00 to 0:40:00, datasets to be found at repository github, the link is there.
Thanks for the timestamps but
@ 16:26 it's df.loc not df.iloc
I feel like Daniel and Leonard's characters in The Big Bang Theory have a lot of similarities. Did anyone else notice that?
This is the first time I've understood why melt is called melt. 32:36
so somehow, problems 1 to 3 contradicts problem 4 and 5?
looks like the url for the data has been removed. I also can't find it in github
from where i can get the dataset
could someone post the link to github repo?
Hadley Wickham'ın askerleriyiz.
5:24 to skip all the pointless nerd BABBLING.
That's why you're still an underling.
Excellent tutorial