Python Tutorial: Real World Example - Parsing Names From a CSV to an HTML List
Vložit
- čas přidán 7. 09. 2024
- In this Python Programming Tutorial, we'll be learning how to parse a CSV file and output the data to an HTML unordered list. This is a real world problem I ran into with my website. The list was becoming too large for me to maintain manually, so writing a Python script to automate this process saved me a lot of time in the future. Let's get started.
The code from this video can be found at:
github.com/Cor...
✅ Support My Channel Through Patreon:
/ coreyms
✅ Become a Channel Member:
/ @coreyms
✅ One-Time Contribution Through PayPal:
goo.gl/649HFY
✅ Cryptocurrency Donations:
Bitcoin Wallet - 3MPH8oY2EAgbLVy7RBMinwcBntggi7qeG3
Ethereum Wallet - 0x151649418616068fB46C3598083817101d3bCD33
Litecoin Wallet - MPvEBY5fxGkmPQgocfJbxP6EmTo5UUXMot
✅ Corey's Public Amazon Wishlist
a.co/inIyro1
✅ Equipment I Use and Books I Recommend:
www.amazon.com...
▶️ You Can Find Me On:
My Website - coreyms.com/
My Second Channel - / coreymschafer
Facebook - / coreymschafer
Twitter - / coreymschafer
Instagram - / coreymschafer
#Python
Dude you are the STAR. Please make more of those real world examples. Why don't you rather create a new category in your tutorial series "Real World Example" etc? You are just amazing Corey!
Glad you found it useful. And yes, perhaps I'll create a separate playlist of real-world examples once I get more videos like this created. Thanks!
@@coreyms will be waiting for that....
@@SanataniAryavrat Still waiting...
😆@@envy_coppr2146
@@envy_coppr2146 Waiting :p
I'm gonna complete all of your Python tutorials, and then write in my resume, "Comfortable with Python - taught by Corey Schafer".
I think if in the code, Corey uses continue instead of break, it would have been better...
html_output = ''
names = []
with open('patrons.csv', 'r') as data_file:
csv_data = csv.DictReader(data_file)
# We don't want first line of bad data
next(csv_data)
for line in csv_data:
if line['FirstName'] == 'No Reward':
continue
names.append(f"{line['FirstName']} {line['LastName']}")
html_output += f'There are currently {len(names)} public contributors. Thank You!'
html_output += '
'
for name in names:
html_output += f'
\t{name}'
html_output += '
'
print(html_output)
I don't even like programming because I never understood the the core basics well. It's so cool now after your tutorials that I also try to practice. I can be the last man to indulge in programming, however, when I start watching your videos I couldn't stop watching it. Thank you so much for all these stuff
I was almost dropping out of CS class back in 2015 when i was in second Year, then i bumped into this guy...You damn good man, you made me a Python Developer...
I love this channel! I learn more from you than from my college professor.
You have done a great service to so many people with these videos, thank you
see mine too. The playlist provides most of the fundamentals for Python. And soruce files too.
Corey, these videos are excellent, so clear and I appreciate your thought processes as you go through the problem. Many thanks
Thanks a lot for these videos Corey 😃 I just successfully automated the process of copying and arranging photos and videos in specific date folders from my dslr to my PC 😃
this was so cool! Using just two modules, CSV and Random, we can actually take in data online and generate our very own huge chunks of dump data sets for various testing and research purposes or even just for fun, to play with it :p
Thanks for this, really amazing!
Very nice explanation with no time-loss in between, thank you for that!
One small thing for improving your efficiency is to use str.join() method instead of += (concatenation) because it's much more slower and uses more memory.
That is because strings are immutable, hence everytime a new string needs to be created by parsing all the characters from the previous one.
Would be very interesting to see some OOD tutorials in the future since you have such an amazing and clear way of explaining stuff!
Best python teach ever. Thank you corey
Very useful video for me who does a lot of manual stuff using CSV files. Keep up the good work Corey
what video do I watch that takes the html_output and insert it into the actual html page?
I seriously can not thank you enough for the amount of help you are providing. This tutorial was really helpful. One thing i'm having trouble with is practicing OOP. I've watched your tutorials on OOP and they are pretty good but i don't know where to use them and how to implement them into my daily routine of practicing python as most of the code i practice is usually in functions or sometimes not in functions at all. I hope you understood what i asked as my english is not that good. Great stuff !
Hey there. Your English seems great to me. I wish I could speak a second language as fluent as that. And yes, applying Object-Oriented concepts to everyday problems is a common hurdle that many people face. I would say that the most important part of initially understanding Object-Oriented code is so that you can look at libraries and frameworks written by other people and understand what it is they are doing. Once you start to see how others are using object oriented code in their projects, then it will slowly start to give you ideas for how you can do the same. I would love to put together some videos on practical real-world examples of object-oriented code in the future. I'll see what I can come up with. Thanks!
Thank you for the great advice ! :)
Corey Schafer
Hey that'd be great thanks
Every time, thank you very much. Best python channel across youtube
Hi Corey, you have such a fluency in python and your teaching style is really good. I appreciate it and will contribute to your efforts once i am financially capable.
loved all videos and subscribed, your explanation was best than others who say "we can learn it later, what that function does!"
Yes, we enjoy these tutorials. Thank you very much.
It's funny you posted this. I had to use this Python library recently with some data I gathered from an experiment of mine. I ended up using pandas, though. It's a nice option for data analysis/visualization and it offers functions to deal with csv files, converting them to data frames.
Great video Mr.Corey!! , Very useful in daily tasks.Greetings from Mexico.
sir when i use f''{line[0]} {line[1]}" and after doing all according to you i get error of EOL while scanning string literal i search on net all time but i didn't found any answer please help me in this problem,thanks
First of all, your teaching style is good, and anyone grabs ur teachings easily, and thanks for teaching us. you are the best tutor I have ever come up with in python language. can u suggest any python language book, which is best to go through other than your channel? Moreover, I am a beginner to python language I thoroughly enjoyed ur videos, if u can suggest me best python book to ur sensibility and I guess it suits me and it helps me to learn further (I like physical like books). In case if u written any books related to this language can u suggest one (or) your favorite book, please......................................
awesome tutorial.
thanks a lot! this makes the concept clear.
and yes, do make a video on real time use of oops in python... that would be very helpful.
Is there a way to do somethine like "next(csv_data, occurrences=2)" , so we can skip multiple lines and keep the code a bit more clean? (min 7:02)
import csv
html_output=''
names=[]
data_file=open('TechCrunchcontinentalUSA.csv')
csv_data= csv.reader(data_file,delimiter=',')
print(csv_data)
after this...its showing this output Why? the file have 3000+ lines
[Finished in 0.2s]
I just discovered this channel and all I can say is that you do a great job at explaining different Python concept. Could you do a video on async/await (asnycio)? And explain what asynchronous programming really is, since so many people have problems with understanding the concept.
What if I wanted to print the names after "No Rewards"?
I am new to Python, your videos are awesome
I like to thank you,+Coreyms for your awesome videos. Your videos are not only informative but also interesting. Can you make some videos on python GUI relate?
I want to be a data scientist and am following your videos to make my base strong.Falling in love with you.....I wish I could get a mentor like you in real life....Love from India...
Another absolutely awesome video. Thank you, Corey.
Your videos are excellent. Please consider using a larger font - it's difficult to read your code. Thank you very much.
Hi Corey, truly fantastic video as always.
Have you thought about making a series about some mini-midsize projects that a freshman as myself and the other viewers can take as a step towards making a portfolio for a future job interviews? To transfer your experience about project life cycle, process some examples... There are some examples around, but with your gift for lecturing, it would be one of the best I'm sure.
That's a great idea. I would love to do that. I do have some larger project tutorials on the back-burner right now. I just need to find the time to get them written and recorded. I try to keep a continuous flow of videos coming out, and when I get stuck on big projects for too long (like my OOP series or Intro to Python series), I start to hear from a lot of people who want me to push out videos faster. So I juggle between working on the large-scale projects and these shorter videos to keep them coming. But yes, I will work on something like that in the future. May take some time though.
That's understandable, you must be overloaded with wishlists :). BTW, OOP and PY intro series are a major success so just keep up the good work and we'll be waiting.
This is great, next step, how can you get this on a HTML page?
Thanks sir 😀 , most helpful 🙂 , heart ❤️ for you
_No Reward_ Smith contributed 1000$ !
Haha. I'll just have to hope no one chooses that as their first name. If so, the script will need to be altered a bit.
lol
Erellei - It could be Google or Anaconda contributing.
Wish you would get that app that shows your keystrokes would love to learn the hot keys you use
If someone put a comma in the name, for example, and we don't want to split on that, how does the csv module handle it?
Corey thanks so much for this real life problem example.
Could you please do real life problems with pandas?
Thanks
Excellent tutorials. Admire your style of coding and clean and neat explanations. +1 for more of these real world examples.
Instead of using DictReader isn't it easier to use reader and make the code also readable with a codeline like fname, lname = line[0], line[1] ?
Hey, corey , I only discovered your channel a week ago and I watched 3 videos of which I can't get enough of, as I am new to programming your vids were so helpful and full of knowledge not shown in any beginners courses, so thanks. One thing in this vid I tried but is not working well is for loop inside the with statement is running forever please take a look and tell me what is wrong:
import csv
html_output = ""
names = []
with open("prayertimes.csv","r",encoding="utf-8") as data :
csv_data = csv.reader(data)
for line in csv_data :
print(line)
Hey there... I'm not sure why this loop wouldn't end. Just by glancing at it, it looks fine to me. How many lines are there in your csv file?
hey I found the problem it's not in the script it's in another script that I made the csv file out of it's basically a web scraping script I fixed it, but thanks for the help.
Thank you Corey! Your videos have been an amazing help for me. Is it possible to take each line of a csv and turn it into a separate instance of a class and assign the instance variables? My goal is to create methods and functions with the data from the csv file. Thanks again and keep up the good work!
I'm not gonna lie...it was rough going, but I had chatGPT on standby to help me. LOL! The issue was weird and it may be more VS Code than anything else as when I was messing with stuff per chatGPT's suggestings, my terminal finally showed data instead of just [ ]. After that things were fine...until the f-string line where I used single quotes when I needed to use both. :)
As I'm not currently on Python 3.6 yet, how would you append to names without using the f-string?
I've tried pretty much anything from the "String Formatting - Advanced Operations for Dicts, Lists, Numbers, and Dates" video, but none of it gives a satisfactory result. What am I doing wrong? Python version is 3.5.2.
The format method should work for you. I'm not sure why the examples in that video wouldn't work. I opened my python interpreter and created an empty list and appended a name to it twice with these two different methods:
>>> names = []
>>> line = {'FirstName': 'John', 'LastName': 'Smith'}
>>> names.append('{} {}'.format(line['FirstName'], line['LastName']))
>>> # OR
>>> names.append('{0[FirstName]} {0[LastName]}'.format(line))
>>> names
['John Smith', 'John Smith']
>>> # You could also do this
>>> names.append('{FirstName} {LastName}'.format(**line))
So you have several options... I hope one of these work for you. If you'd like explanations of any of those then I would try watching the string formatting video through one more time.
In my last version I had the format(line) *outside* of the names.append method, and I had "line" instead of 0.
I will rewatch the video to try and understand why the 0 works and 'line' does not...
I need to also get my head around this unpacking business... 8-)
Thank you very much for attempting with me.
I didn’t understand the term of html_output += "......."??
Thankyou Corey for all of your contribution, they help me alot on learning python. on this vid, downloading offline file and upload to your website, i'm thinking is there a more fancy way such as getting the data from Patreon API if it have?
5:41 you are able to print the csv_data when parsing it through list(), when I do this i get an "I/O operation on a closed file error". python v 3.6 - what am I doing wrong? Using anaconda(IDE)
Are you sure your indentation is correct? It sounds like you may not have the line indented correctly and you are operating on the file object outside of the context manager
It even happened when I copied and pasted your source code in! must be something wrong with my builds?? - huge props for responding btw I know how busy you must be as your channel is so popular.
I tried it in visual studio and it worked - must be something wrong with my Anaconda install - I will reinstall it.
That is a hard CSV file to parse. Have not seen the whole video. I am looking forward to see how you parse it.
you can add this code:
data = pd.read_csv('file1.csv', error_bad_lines=False, warn_bad_lines=False)
Old comment but instead of break you should use continue at 10:23 so the "No Reward" line should skip and get the other names after maggie, iam sure you ment to change it but it just passed. iam learing a lot from your videos hopefull ill watch all of them. God bless you!
Really great and intuitive video. I just have a doubt.. what if we want the headers to be there but don't want that second line (i.e., "reward +1.....), without using the DictReader? How do I do that?
do you have a video on how to automate that data to automatically update an Html page based off a time frame or schedule. That would be extemely helpful content for most everything I would want to do, if included with Json data for micro controller data logging. Thanks for the great videos, I really love your approach and thinking/teaching style. Thanks Corey for taking the time
Corey, thank you so much. This is great. I do have a question if you could prepare something about this. I have 2 csv files. The first column is device serial number say from 1 to 2000. Then, the rest of the columns are data like, one of the columns is jitter, one is frequency ppm, one is current etc. I want to write a script that takes file1 and file2 (same file structure, but serial number column in file2 is a subset of serial number column of file1). The script should plot the difference between elements of specified measurement column. I want to make sure these these differences will correspond to the same corresponding serial numbers of file1 and file2. Thank you.
Great video, just what I was looking for.
As usual, great work, Corey!
- Jacob
This video was helpful, I wanted to ask if you could make a video for making a report using python, like extraction of the the data and making report .
Thank you Corey!
Great video. But what about the names after line 35 or after 'No Reward', those are not being added to the list ?
Think you meant to use "continue" and not "break" there
remember those people didn't want to be mentioned thats why corey used break and not continue
using keyword 'continue' instead of the 'break', this can be a better understanding of 'continue'.
Could you make a tutorial on Tkinter? (maybe you have made one already but i cant find it)
OMG you are the best seriously
to test the code in the same time with youw need patrons.csv
Thank you
From Nairobi-Kenya-Africa I have come to love your work Corey. Thanks alot. Could you please do some Python GUI stuff ...
love your videos corey!! but you talk sooo fast, haha as a non native speaker and programming beginner i always watch you at 0.75 speed.
Cool video as usual. What about a csv file that is not ordered or structured with hearder but instead a continuous string that separates data with a ; delimiter. Iterating over the data returns one really long string line. Objective is to parse the string with carriage return or new line after each ;. example ( ST*834*0001*005010X220A1;BGN*00*1*20120106*010510****4;REF*38*170175;) Must regular expressions be used in combination with a csv reader or csv writer dictreader or dictwriter?
Thanks Corey!🙏
kudos mate
Perfect example ❤ thank you Corey!!!!
Great lesson thanks Corey!
Will you pls tell me how to do validation on accessed emails from csv file ??
please do the video on parsing reading and writing csv files also.
Amazing and very informative videos....
How do we extract this exact csv file you used? Is there any link?
I personally just use the pandas module for any IO python related tasks these days.
extraordinary bro
it is not allowing me to use 'break' after if statement.
html_output = ''
names = []
with open('patrons.csv', 'r') as data_file:
csv_data = csv.DictReader(data_file)
# We don't want first line of bad data
next(csv_data)
for line in csv_data:
if line['FirstName'] == 'No Reward':
continue
names.append(f"{line['FirstName']} {line['LastName']}")
html_output += f'There are currently {len(names)} public contributors. Thank You!'
html_output += '
'
for name in names:
html_output += f'
\t{name}'
html_output += '
'
print(html_output)
you need to make videos on python mini and advanced projects
Great video, thank you!
10:32 Why cut out the rests of the names after Maggie Jefferson? Aren't they important? 🤔🤔
This is the right code, for line in csv_data:
if line[0] != 'No Reward':
names.append(f'{line[0]} {line[1]}')
i get error in this lines. names.append(f"{line[0]} {line[1]}")
video time at 9:01 to 9:14
Error:list index out of range.
i am using python version 3.6.6
In your code also i get error in same line.
thanks for this tutorial.i really appreciate your work!!
Hi Parth, if you can share your code on Repl it or Paste Bin, for example, I can try to help you... =D
Double check the csv file and see if there is a blank line at the end. There should NOT be one. That's what was wrong with mine when I did this.
did u manage to solve ? I have the same error
Hey Corey ... little help what if someone just wants to skip row 2 and row 35 only ?
Hmm, that's a pretty unique use case. In that case I would probably just initiate a counter outside of the loop and then increment by 1 each time through the loop. Put in an "if" statement that skips the loops using "continue" if it's the lines you want to skip.
yeah i tried to do same but it didn't help me when there's some random text line at row 9 text was 'amount...', at 23 it's 'shopping...' and similarly, at some other row number it's 'generate...', i was looking for an algorithm that can identify the rows and differentiate such rows which do not make sense like in your case it's first name, last name and email and rest of the data is irrelevant, I can't read the data or would not give the effort to identify such rows because in my case there are over 1 million rows......
Saadeep, All you have to do is to replace break with continue in your code. for the data file used in this case. However, for the case your are trying to solve I will suggest to try with regular expression.
sir please tell pattern based programs
Thanks Corey, That great little tutorial. What will it take to write the output_html to a file?
It would be pretty simple. You would have to include whatever additional HTML you want for the entire file, and then after everything is appended to the output the way you like, you would just need to do something like:
with open('webpage.html', 'w') as html_file:
html_file.write(html_output)
Thanks. Work like a charm!
Thank you 🙏🙏🙏
this was great, how to show the result on your website automatically , do you have to upload python file to your server or how it work ?
No, I just have this script on my personal machine and occasionally run it from there. I have a login to my website where I can drop in that section of HTML
thanks Coreny , can you make video on that if you have time .
Rather than using next, is there a reason not to just slice the list from the data you're reading? Like, "for line in csv_data[2:]" ?
You're making me think pythonic ;)
Need python 3.10/3.9
Decompile parse
Great tutorial !
Great tutorial
CHARLES DARWIN IS A CONTRIBUTOR!?
Ah yes... Ole Chuck. We've been mutual fans of each others work for some time ;)
Awesome video!
Very useful✌✌✌✌
C - Corey
S - Schafer
V - is Vegetarian
couldn't find a word for V 😜
¯\_(ツ)_/¯
Whole thing in 7 lines. Let me know if it can be done shorter/cleaner.
data = pd.read_csv('patreons.csv')
stop = data.index[data['FirstName']=='No Reward']
names = data.iloc[1:int(*stop),[0,1]].values
html_output = f' There are currently {len(names)} contributors!'
html_output += '
'
html_output += ''.join([f'
\t{name[0]} {name[1]}' for name in names])
html_output += '
'
Wish to make a video using those commands ?
Are you asking permission to? Sure.
thanks bro
You should do some HTML tutorials
Thanks you so much