YouTube Data Analysis | END TO END DATA ENGINEERING PROJECT

Sdílet
Vložit
  • čas přidán 18. 06. 2024
  • Join My Data Engineering Courses - datavidhya.com/courses
    In this video, you will execute the END TO END DATA ENGINEERING PROJECT using Kaggle CZcams Trending Dataset.
    If you are someone who wants to learn Data Engineering by doing hands-on projects then this video is for you!
    👉🏻Part 2 of the video - • CZcams Data Analysis ...
    ✨Visit ProjectPro for more projects - bit.ly/3iIGfM5
    Useful links mentioned in the video -
    1. Create Your AWS Account - aws.amazon.com/premiumsupport...
    2. Download AWS CLI - aws.amazon.com/cli/
    3. Download Data From Here - www.kaggle.com/datasnaek/yout...
    Find Commands and Code Used in the video here - github.com/darshilparmar/data...
    👉🏻 Learn Basics (If you don't know) 👈🏻
    1. Learn Python From Here - bit.ly/3t9pFLx
    2. Learn SQL here - bit.ly/3KNmUW8
    3. Learn Spark and Big Data - bit.ly/3telBcW
    4. Learn AWS Lambda - aws.amazon.com/lambda/
    Join Data With Darshil Discord Server: / discord
    Timestamps
    0:00 Project Introduction (Very Important)
    2:22 What You Will Learn
    4:57 Understand Dataset That We Are Going To Use
    6:16 On-Premise vs Cloud Data Processing
    8:50 Create Your AWS Account
    9:19 Best Practices To Follow On AWS
    9:59 Create IAM Account For Admin
    13:15 Install AWS CLI
    16:05 Create S3 Bucket and Upload Our Data
    22:05 What is Data Lake?
    23:40 Understand Data That Got Uploaded
    24:45 Build Glue Crawler and Catalog
    30:14 Use Athena and SQL to Query Data
    32:20 Solving Error and Preprocessing Data
    34:57 Writing ETL Job In Lambda and Cleaning Data
    42:33 What is AWS Lambda and Layers?
    47:51 Querying Our Cleaned Data on Athena
    👦🏻 My Linkedin - / darshil-parmar
    📷 Instagram - / darshilparmarr
    🎯Twitter - / parmardarshil07
    🌟 Please leave a LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
    3 Books You Should Read
    📈Principles: Life and Work: amzn.to/3HQJDyP
    👀Deep Work: amzn.to/3IParkk
    💼Rework: amzn.to/3HW981O
    Tech I use every day
    💻MacBook Pro M1: amzn.to/3CiFVwC
    📺LG 22 Inch Monitor: amzn.to/3zk0Dts
    🎥Sony ZV1: amzn.to/3hRpSMJ
    🎙Maono AU-A04: amzn.to/3Bnu53n
    ⽴Tripod Stand: amzn.to/3tA7hu7
    🔅Osaka Ring Light and Stand: amzn.to/3MtLAEG
    🎧Sony WH-1000XM4 Headphone: amzn.to/3sM4sXS
    🖱Zebronics Zeb-War Keyboard and Mouse: amzn.to/3zeF1yq
    💺CELLBELL C104 Office Chair: amzn.to/3IRpiL2
    👉Data Engineering Complete Roadmap: • Data Engineer Complete...
    👉Data Engineering Project Series: • Data Engineering Proje...
    👉Become Full-Time Freelancer: • Best Freelancer Series...
    👉Data With Darshil Podcast: • Podcast Series - Data ...
    ✨ Tags ✨
    data engineering projects, big data project, data engineering project hands-on, hands-on data engineering projects, learn data engineering, data engineering roadmap, how to become data engineer, data engineering free projects, big data engineering, big data
    ✨ Hashtags ✨
    #dataengineer #project #darshil

Komentáře • 673

  • @kpyoutuber4671
    @kpyoutuber4671 Před rokem +52

    Thank you to Darshil Parmar!.
    Please note that you deployed the lambda function at 39:00 minutes of the video. It is not mentioned specifically in your explanation.
    If not deployed it will only run the default code which will anyway run successfully with hello-world print.

    • @DarshilParmar
      @DarshilParmar  Před rokem +13

      Yes, I have made mistake while editing the video, lot of people faced this error

    • @haziq7885
      @haziq7885 Před rokem +1

      thanks for the solution! i was stuck here for a long time 😅
      Darshil Parmar thanks so much for the video! hopefully this solution can be pinned for others to refer to :)

    • @fahadbakshi5449
      @fahadbakshi5449 Před rokem +1

      @@DarshilParmar can you plz provide me a solution of error at 39:00 . i am getting error as
      "{
      "statusCode": 200,
      "body": "\"Hello from Lambda!\""
      }"

    • @DarshilParmar
      @DarshilParmar  Před rokem +4

      @@fahadbakshi5449 It's not an error, You need to click on Deploy button

    • @theniyota
      @theniyota Před 11 měsíci

      This post needs to be pinned. I wasted todays trying to figure out how to make the lambda function work until I decided to go through the comments.

  • @DarshilParmar
    @DarshilParmar  Před 2 lety +325

    It takes a lot of effort and energy to execute the entire project and record it! I hope you find this useful and make sure you Like this video :)

    • @aritra1414
      @aritra1414 Před 2 lety +1

      What according to you will be the best resource to understand lambda in depth? I need help on that. I am working on bigdata project, but this was not my domain, learning new things and I need to learn faster. Any leads will be helpful for me. Thanks in advance. Also, please keep producing such awesome contents. Thanks a lot!!

    • @DarshilParmar
      @DarshilParmar  Před 2 lety +3

      @@aritra1414 check out AWS reinvent videos on CZcams on lambda and read white paper on lambda to understand more

    • @vivekpuurkayastha1580
      @vivekpuurkayastha1580 Před 2 lety

      Hi Darsheel .. great video as always and yes i did click on like 😀 ... Can you please make a video on how to create a project in Dev environment and then switch to production environment in AWS ..... Basically how to manage the code Lifecycle in AWS from Dev to Production.... or may be you can point to a resource ... Thanks

    • @shashibhushansingh1628
      @shashibhushansingh1628 Před 2 lety

      By following these steps is im able to build this project in azure

    • @tanmayshinde7853
      @tanmayshinde7853 Před 2 lety +1

      I respect your efforts but to be honest I didn't understand anything. please me it in simpler way or breakdown it into smaller chunk if you can

  • @shrirajpawar7817
    @shrirajpawar7817 Před 8 měsíci +38

    For people watching this tutorial now,
    AWS DataWrangler has been changed to AWS SDK for Pandas. Name has been changed but core functionality remains same

    • @vishnuvardhan9082
      @vishnuvardhan9082 Před 4 měsíci +1

      thank you so much man, was going nuts on this! how did you know about this?

    • @TheAINoobxoxo
      @TheAINoobxoxo Před 3 měsíci +1

      thanks man much appreciated

    • @nikitha_sirka
      @nikitha_sirka Před 2 měsíci

      @@vishnuvardhan9082 hii,
      When I changed the layer to AWS SDKPandas and modified the code I found the same error
      Error :
      {
      "errorMessage": "Unable to import module 'lambda_function': No module named 'AWSSDKPandas'",
      "errorType": "Runtime.ImportModuleError",
      "stackTrace": []
      }

    • @user-wi6fk8hu6x
      @user-wi6fk8hu6x Před 2 měsíci +1

      Thank you very much

    • @iamayuv
      @iamayuv Před měsícem

      thanks bhai

  • @snehakadam16
    @snehakadam16 Před 8 měsíci +38

    Thank you to Darshil.
    This is for those who facing issues -
    1) Replace awswrangler with awssdkpandas in the code. The rest code remains the same.
    2) Add Layer : AWSDataWrangler-Python3.8 replaced it with AWSSDKPandas-Python3.8 version 10
    3) Create db_youtube_cleaned db using Glue or Athena before running the code.
    4) For Task timed out issue - increasing the memory along with time, for eg. time = 5 min, memory = 512 MB
    Hope this helps :)
    Tip: Guys, please go through the comments, if you are stuck. You will be able to find a solution for sure.

    • @DarshilParmar
      @DarshilParmar  Před 8 měsíci

      Thanks for putting this in one comment

    • @snehakadam16
      @snehakadam16 Před 8 měsíci

      Thank you for the amazing tutorial and putting a lot of effort @@DarshilParmar. Looking forward to more projects :)

    • @vamsivenna
      @vamsivenna Před 8 měsíci

      @snehakadam16 @Darshilparmar facing issues like
      "errorMessage": "Unable to import module 'lambda_function': No module named 'awssdkpandas'",
      "errorType": "Runtime.ImportModuleError",
      "stackTrace": []
      help me out

    • @vamsivenna
      @vamsivenna Před 8 měsíci

      it worked i think the database name should be = ""db_youtube_cleansed"""
      even awswrangler with AWSSDKPandas-Python3.8 version 10 and memory 256 mb is working fine for me. Thank you.
      But as per the video the database should get created automatically

    • @prathmeshsinha4705
      @prathmeshsinha4705 Před 8 měsíci

      How did you solved this ?@@vamsivenna

  • @vitoriagarcia9876
    @vitoriagarcia9876 Před rokem +2

    These tutorials are so helpful for me! And also, they show how much effort on production you put into them. Thank you so much, Darshil!

  • @teja_surya
    @teja_surya Před 2 lety +1

    This project and you explaining it in a simple and elaborate way was awesome. Keep them coming!

  • @fabriciomiriani
    @fabriciomiriani Před rokem +2

    Amazing job - I'm just starting to use AWS because I would like to become a Cloud Engineer and this just incredible. Thank you a lot for your effort !!

  • @jerichocruz29
    @jerichocruz29 Před rokem +2

    Darshil, this is an amazingly executed project and it was easy to follow. Thanks for taking the time to put this together. Great channel.

  • @ajitagalawe8028
    @ajitagalawe8028 Před rokem

    Best video I have seen so far for the end to end project in big data. Thanks!

  • @imsdengineer
    @imsdengineer Před rokem

    Great Stuff, You Rock boey! The entire video was very much intuitive and I must say that without a shadow of doubt that all the nitty gritties of Data is discussed in this, heading for the second part now. Worth a ⌚

  • @mananyadav6401
    @mananyadav6401 Před rokem +7

    Amazing @darshil ....It is clearly visible how much effort u have put in for ppt , video reording , storyboarding and including small small nuances and error that could be potentially faced.
    It can't express in words how valuable it is and how much information you are providing for the community. Really inspiring and motivating.
    Someone in other comment rightly mentioned It is a pure gem on CZcams

  • @janwienke6479
    @janwienke6479 Před rokem +4

    Great Project Documentation to try for yourself. One little thing to add would be a rough aws cost estimate. Definitely a thing I would be looking for if I was starting.

  • @bfkgod
    @bfkgod Před 2 lety +22

    Darshil, you have made one of the most valuable DE learning channels on youtube. Keep up the amazing work! Thank you.

    • @DarshilParmar
      @DarshilParmar  Před 2 lety

      Thanks, will do!

    • @Punithan-rj5ng
      @Punithan-rj5ng Před 3 měsíci

      @@DarshilParmar Function Logs
      START RequestId: 2b020a60-532e-4b33-9933-7cc87b5406cc Version: $LATEST
      An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found.
      Error getting object youtube/raw_statistics_reference_data/CA_category_id.json from bucket de-on-youtube-raw-useast1dev. Make sure they exist and your bucket is in the same region as this function.
      LAMBDA_WARNING: Unhandled exception. The most likely cause is an issue in the function code. However, in rare cases, a Lambda runtime update can cause unexpected function behavior. For functions using managed runtimes, runtime updates can be triggered by a function change, or can be applied automatically. To determine if the runtime has been updated, check the runtime version in the INIT_START log entry. If this error correlates with a change in the runtime version, you may be able to mitigate this error by temporarily rolling back to the previous runtime version. For more information, see docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html
      [ERROR] EntityNotFoundException: An error occurred (EntityNotFoundException) when calling the CreateTable operation: Database db_youtube_cleaned not found.

  • @ShubhamKumar-fs9wi
    @ShubhamKumar-fs9wi Před 7 měsíci

    Thank you Darshil for this amazing video, it was very helpful. Just completed this whole project plus did some extra work of moving data to redshift using glue job as well while creation connection and enabling vpc endpoint.:)

  • @ArnavMondal14
    @ArnavMondal14 Před 11 měsíci +1

    Great video. Loved it and helped me build my resume. Would love to do what you do today and freelance

  • @shreeyajoshi9771
    @shreeyajoshi9771 Před rokem

    Thanks a loads for this video Darshil! Very much appreciated! 👏👏👏👏

  • @rohitsaha08
    @rohitsaha08 Před 10 měsíci

    this is a great project with your excellent guidance Darshil. Thank you!😀

  • @ajtam05
    @ajtam05 Před rokem +5

    Just wanted to say thank you to Darshil Parmar for these projects. It's hard to find anything online that helps to this extent from end-to-end. This is great stuff! Cheers! :)

    • @mcaddit6802
      @mcaddit6802 Před rokem

      I am unable to proceed further after clicking on test getting err0r:"errorMessage": "'s3_cleansed_layer'",
      "errorType": "KeyError", can anyone pls tell what's the problem?

    • @vinaydhande9926
      @vinaydhande9926 Před 7 měsíci

      Are You Solve the error@@mcaddit6802

  • @revathil8986
    @revathil8986 Před 2 měsíci

    Excellent explanation. Each and every step is easy to follow and understandable.

  • @castlemonohunter3019
    @castlemonohunter3019 Před 2 lety

    Thank you so much for doing this.... The only one on youtube with curated data related content ❤💫

  • @dntking40
    @dntking40 Před rokem

    Excellent work. Thank you for this amazing workshop.

  • @lloydwang8108
    @lloydwang8108 Před rokem +1

    hey Darshil, I rarely comment but just wanted to say a big thank u. This helped me out a lot! Looking forward to more of such content in the future :D

  • @penninahgathu7956
    @penninahgathu7956 Před rokem

    Thank you so much for teaching us such valuable content! Be blessed

  • @ritikkeshari1463
    @ritikkeshari1463 Před 2 lety

    Thanks alot brother..needed such kind of lecture ..really helped in enhancing my skills..please make more such videos

  • @ataurrehman3664
    @ataurrehman3664 Před 2 lety +1

    Thank you so much for making this video!!! This would be 6-7th video of yours which I've added to my playlist. I request you to post more such project videos in different domains.

    • @DarshilParmar
      @DarshilParmar  Před 2 lety +2

      Thank you, yes I will try to post such videos

  • @iamdare
    @iamdare Před rokem

    Hi Darshill, good video and thanks very much. I learned a lot. Please in your subsequent videos, do try to zoom in more often so we can get to see what you’re doing on the screen. Thanks.

  • @HemantSharma-fw2gx
    @HemantSharma-fw2gx Před 2 lety

    Thank you so much for this darshil!...keep up the good work🙌

  • @mohammedjouhar6363
    @mohammedjouhar6363 Před 2 lety

    Thank you, man.. Keep up the good work!

  • @adityaanand835
    @adityaanand835 Před 2 lety

    Really appreciate your hardwork you bring to tthe table!!!

    • @DarshilParmar
      @DarshilParmar  Před 2 lety

      Thank you making my hardwork pay off by watching video

  • @ashutoshprakash9468
    @ashutoshprakash9468 Před 2 lety +1

    Great Work, Darshil!!! Next level Data Engineering knowledge provided by you in this content. ✌️ Industry level project.

  • @trytrybutdontcry1-zf9yu
    @trytrybutdontcry1-zf9yu Před měsícem +1

    Great Video, Thank you so much!!

  • @idhwanibhatt
    @idhwanibhatt Před 2 lety +7

    Thank you so much Darshil for this video. We need more such project based learning in data engineering instead of just cliche theory. 😆

    • @DarshilParmar
      @DarshilParmar  Před 2 lety +2

      Yes, more videos like this is coming

    • @mcaddit6802
      @mcaddit6802 Před rokem

      I am unable to proceed further after clicking on test getting err0r:"errorMessage": "'s3_cleansed_layer'",
      "errorType": "KeyError", can anyone pls tell what's the problem?

  • @ahmedmohiuddin1866
    @ahmedmohiuddin1866 Před 2 lety

    WOWW. This is amazingggg. Thanks Darshil. I have just started watching the video and looking at the content got me excited.

  • @takisally
    @takisally Před rokem

    Thank you for this particular video

  • @gomes8335
    @gomes8335 Před 2 lety +1

    Just love your content man. Keep it up !!!

  • @prikshitbatta
    @prikshitbatta Před rokem +3

    Hi, Darshil thanks for this project. Faced a lot of errors but took two days to complete the project. In the end, it is satisfying.😀

    • @vanadin8009
      @vanadin8009 Před 8 měsíci

      can you give the estimated cost of aws services used in this project it will be of so much help and thank you

    • @vasudevreddy3527
      @vasudevreddy3527 Před 8 měsíci

      @@vanadin8009 we can do basically for free with free tier AWS account

  • @mackshonayi943
    @mackshonayi943 Před rokem

    Thank you Darshil, I completed this part successfully. Your content is invaluable may God bless you

    • @fahadbakshi5449
      @fahadbakshi5449 Před rokem

      bro i got stuck at 30:00 minute can u plz help me

    • @ishan358
      @ishan358 Před 10 měsíci

      How did you solve runtime error

  • @youraverageguide
    @youraverageguide Před 2 lety

    Thanks for all your efforts! This is perfect.

  • @trishasingh8832
    @trishasingh8832 Před 2 lety

    Great work Darshil! 🔥🔥

  • @SankarJankoti
    @SankarJankoti Před 2 lety +2

    Your content on data is pure! No match.

  • @intrepidm8753
    @intrepidm8753 Před 2 lety

    its a great one, very useful n resourceful for aspirants like me👍🏼

  • @raghuboyapati7311
    @raghuboyapati7311 Před 2 lety

    Great video man.
    AWS has updated the Emphemeral storage of Lambda to 10 GB

  • @ijaj.datanerd
    @ijaj.datanerd Před 2 lety

    Thanks, man.....appreciate your hard work and effort

  • @Bijuthtt
    @Bijuthtt Před rokem

    Awesome tutorial project. I could complete this session.

  • @paytmoffers7794
    @paytmoffers7794 Před rokem +2

    I hope you have more such projects for us in your pipeline 😍 Please do it

  • @huzaifa_2590
    @huzaifa_2590 Před 9 měsíci

    It Was An Amzaing Project. I Learned Alot From This Video. Thank You So Much, Appreciated The Work.

  • @arpansatpathi9645
    @arpansatpathi9645 Před 2 lety +11

    Thank you Darshil for this wonderful video. One thing I would like to point out that might help others following this tutorial is whenever you update your lambda function, click on deploy first to actually test your changes. In my case I wasn't getting any errors and later realized that the default hello world code was still running.

    • @DarshilParmar
      @DarshilParmar  Před 2 lety +4

      Yes, I might have made mistake while editing the video, I did click on deploy and lot of people missed it.
      I will keep this in my mind
      Thank you for the feedback

    • @arpansatpathi9645
      @arpansatpathi9645 Před 2 lety +2

      @@DarshilParmar while we're at it, could you please give a solution for the EntityNotFoundException that somebody else also pointed out. I'm also getting the same error and haven't been able to resolve it. Tried creating the cleansed database in glue manually but still it is not working. Hope to get a reply.
      Thanks in advance :)

    • @angelnadar1209
      @angelnadar1209 Před rokem

      Thanks Ashutosh ,same thing faced by me .thanks for posting this comment ,its helpful.

    • @angelnadar1209
      @angelnadar1209 Před rokem

      ​@@arpansatpathi9645 was it resolved ?if yes could you post the solution?​

    • @thesevenacoustics
      @thesevenacoustics Před rokem

      @@arpansatpathi9645 rename 'db_youtube_cleaned' to 'de_youtube_cleaned' in env varialble

  • @zendr0
    @zendr0 Před 2 lety +1

    Absolute gem! Thank you for making this video. Learned a lot today.
    And if possible, Although I know you have your job, please try to make more of such content in future.
    Lots of love💛💛

    • @DarshilParmar
      @DarshilParmar  Před 2 lety +1

      I will try my best to provide as much as I can

    • @mcaddit6802
      @mcaddit6802 Před rokem

      I am unable to proceed further after clicking on test getting err0r:"errorMessage": "'s3_cleansed_layer'",
      "errorType": "KeyError", can anyone pls tell what's the problem?

    • @ishan358
      @ishan358 Před 10 měsíci

      How do you solve runtime error

    • @drishtihingar2160
      @drishtihingar2160 Před 7 měsíci

      @@mcaddit6802 yeah I am also getting same error, how did you solved it. Can you help me out

  • @pranjalipandey4233
    @pranjalipandey4233 Před 2 lety

    Thanks Darshil for such a valuable learning...

  • @shyam96105
    @shyam96105 Před 2 lety

    Hi Darshil, please make video on how do you deliver the project to your clients after completing it.

  • @rocky6517
    @rocky6517 Před 2 lety

    Thanks Darshil for Creating this Video

  • @youraverageguide
    @youraverageguide Před 2 lety +1

    Part1 done. It was really informative. Waiting for part 2!

    • @DarshilParmar
      @DarshilParmar  Před 2 lety

      Check link in the description for that

    • @harshalshende69
      @harshalshende69 Před rokem

      Bro can u plzz help me for this actually I stuck in part 1 during catalog data from1week so I can move forward if u tell my mistakes over their🙏it will big help for me

    • @ishan358
      @ishan358 Před 10 měsíci

      ​@@harshalshende69do you find solution

  • @yichiz9389
    @yichiz9389 Před 5 měsíci

    Thank you so much!

  • @ririraman7
    @ririraman7 Před 2 lety

    Thank you very much brother.

  • @sudhanshuupadhyay458
    @sudhanshuupadhyay458 Před rokem

    Great and helpful tutorial.

  • @shubhamdeshmukh6339
    @shubhamdeshmukh6339 Před rokem

    Hi Darshil! You r doing great job
    Please make more video on end to end project

  • @ueeabhishekkrsahu
    @ueeabhishekkrsahu Před 11 měsíci +1

    After consistently working for 2 days, finally done with the project.

    • @sembrueldorinvil4167
      @sembrueldorinvil4167 Před 10 měsíci

      How did you solve the runtime problem?

    • @ishan358
      @ishan358 Před 10 měsíci

      ​@@sembrueldorinvil4167same question bro

    • @ishan358
      @ishan358 Před 10 měsíci

      How did you solve runtime lamda error ?

    • @sembrueldorinvil4167
      @sembrueldorinvil4167 Před 10 měsíci +1

      Found the answer. Assign memory to your function. I think 500 MB should work.

    • @ishan358
      @ishan358 Před 10 měsíci

      @@sembrueldorinvil4167 then also same error so much frustrating it is

  • @nadianizam6101
    @nadianizam6101 Před rokem

    Excellent.plz upload more project

  • @abhisheknakate9347
    @abhisheknakate9347 Před 2 lety

    thank you darshil very informative content........please upload 2nd part of video

    • @DarshilParmar
      @DarshilParmar  Před 2 lety

      It is uploaded, check link in the description

  • @mrutyunjay7877
    @mrutyunjay7877 Před 2 lety

    Thank you so much Darshil !

  • @nguyentiensu4088
    @nguyentiensu4088 Před rokem

    Thank you very much!

  • @dimlight1172
    @dimlight1172 Před 2 lety +3

    This is the first video I'm watching from your channel sir. Even though I didn't understand much, I watched till the end. Your explanation of the stuff just kept me going. I'm a 2nd year engineering student. I have done web development earlier and have basic knowledge of SQL. This data engineering field seems to be very interesting field to dive in. Can you please guide me through sir? So that I can learn the topics and build some good projects using necessary and relevant technology.

    • @DarshilParmar
      @DarshilParmar  Před 2 lety

      I have complete roadmap on my channel for Data Engineering and study plan, you can check that and start working

  • @yassaryelurkar3631
    @yassaryelurkar3631 Před rokem

    Hey great video. I wanted to ask whether I will be charged for using AWS Athena coz it mentioned additional charges for using athena query when I opened it. Thanks for the video.

  • @rizbasamalah5326
    @rizbasamalah5326 Před rokem +3

    Thank you so much Darshil for the video! I am having an issue when trying to create a crawler, getting error : "The following crawler failed to create: "name of the crawler"
    Here is the most recent error message: Account 'Number of account' is denied access." Tried to check the IAM roles created, deleted recreated again, however still receiveing the same message. Would you have an idea what could be the issue?

  • @dipankrdey1850
    @dipankrdey1850 Před rokem

    Thank you for your effort. How much amount AWS billed to build this project ? Or all services are under free tier ?

  • @siddhideshmukh6424
    @siddhideshmukh6424 Před rokem

    Hey Darshil ,i rarely comment but just wanted to tell you tht you are awesome nd you content is just amazing ❤️

  • @pankajchandel1000
    @pankajchandel1000 Před 9 měsíci

    among this and covid project ..which one should i try building first as a beginner ?

  • @agnimitram340
    @agnimitram340 Před rokem

    Thanks Darshil big help.

    • @meetpatel1873
      @meetpatel1873 Před rokem

      Did you get charged while using AWS services under free tier?🤔

  • @Watson22j
    @Watson22j Před 8 měsíci +1

    Hey Darshil, if you were a beginner how would you write such long code in lemda? I am a beginner and wondering how will I be able to write such codes

  • @AnandSharma-kz2bs
    @AnandSharma-kz2bs Před 2 lety

    That is pro level content ❤️💥

    • @DarshilParmar
      @DarshilParmar  Před 2 lety +1

      Pro level viewer who watches entire 1 hour thing in the world of reels/shorts

  • @ankitchilkalwar8410
    @ankitchilkalwar8410 Před rokem

    Please create similar end to end project using GCP services there is no content available on CZcams.
    it will be very helpful,
    Thanks👍

  • @aman6646
    @aman6646 Před 2 lety

    Great Video 🔥🔥💯 Thanks

  • @ranjansrivastava9256
    @ranjansrivastava9256 Před 4 měsíci

    Dear Darshil, Could you please let us know which architecture have you used in the demo -- Lambda architecture or Kappa Architecture. Wanted to understand more on architecture prospective. Please share your thoughts.

  • @surrealsoupuniverse
    @surrealsoupuniverse Před rokem

    Hello what should i learn before doing this project? What are the prerequisites? Thanks

  • @rishav144
    @rishav144 Před rokem

    best project 🔥

  • @ZahraNoordoyejama
    @ZahraNoordoyejama Před 2 lety

    wow amazing man

  • @skateforlife3679
    @skateforlife3679 Před rokem +20

    (edited) Important note on missing libraires :
    - AWSDataWrangler-Python3.8 is not still available
    - I replaced it with AWSSDKPandas-Python3.8 version 1

    • @anupammathur918
      @anupammathur918 Před rokem +3

      I am having error for database db_youtube_cleaned not found can you please check once?

    • @soumyaranjandash3597
      @soumyaranjandash3597 Před rokem

      @@anupammathur918 same here

    • @anupammathur918
      @anupammathur918 Před rokem +2

      @@soumyaranjandash3597 becz that is not created go to athena nd create one db with that name

    • @mackshonayi943
      @mackshonayi943 Před rokem +1

      @@anupammathur918 Thanks this helped. I created the database in Glue and it worked

    • @sahityamamillapalli6735
      @sahityamamillapalli6735 Před rokem

      @@anupammathur918 can you please elaborate in anthena data sources are there

  • @aminearguig3114
    @aminearguig3114 Před měsícem +1

    I have a question and i hope you will answer,
    is these services free in aws free tier

  • @danala5963
    @danala5963 Před 2 lety +1

    This was a great help..one question though..when you executed this project using different AWS services S3, Athena, Glue etc.. what was the approx. cost you got after full project execution...Thanks

    • @DarshilParmar
      @DarshilParmar  Před 2 lety +1

      Most likely there won't be any charge if you are under free trial but even if they charge you it will be max 3-5$
      You can raise support ticket stating you were just trying to learn about service and they won't charge you

  • @shouryanagpal5813
    @shouryanagpal5813 Před 4 měsíci

    Hi Darshil, I always think of starting your project videos but I always got stuck whether aws cloud services willl be charged or it's free or is there any other alternatives

  • @paytmoffers7794
    @paytmoffers7794 Před rokem

    You are amazing 💯

  • @vivekpuurkayastha1580
    @vivekpuurkayastha1580 Před 2 lety

    Hi Darsheel .. great video as always and yes i did click on like 😀 ... Can you please make a video on how to create a project in Dev environment and then switch to production environment in AWS ..... Basically how to manage the code Lifecycle in AWS from Dev to Production.... or may be you can point to a resource ... Thanks

    • @DarshilParmar
      @DarshilParmar  Před 2 lety

      Hey Vivek,
      Thank you and converting from Dev To Prod is same but you need to follow best practices such as instead of creating lambda from UI you will have some code that can handle it, you will deploy that code on github and then it will run automatically, etc...
      There are no definitive guide, you just have to figure it out based on requirements but here is a blog on that
      blog.gruntwork.io/how-to-build-an-end-to-end-production-grade-architecture-on-aws-part-1-eae8eeb41fec

  • @yeggadisaisiddharth7877
    @yeggadisaisiddharth7877 Před 3 měsíci

    Superrr bro

  • @surajpatra8705
    @surajpatra8705 Před 2 lety

    Thank you bhaiya for this video

  • @akshatchatterjee
    @akshatchatterjee Před 27 dny

    best video

  • @sunny60035
    @sunny60035 Před 2 lety

    Hi Darshil I have followed your end to end Data Engineering project on covid data Analysis it helped me to learn about different services on Aws and what exactly a data engineer does. Can u please make a end to end Data Engineering project using MS AZURE and Databricks. Thanks again 👍

    • @DarshilParmar
      @DarshilParmar  Před 2 lety

      Maybe in future I will do it, if you want then check out ProjectPro, not saying because I am sponsored or anything but they really have good projects.

    • @charudattbelsare2903
      @charudattbelsare2903 Před 2 lety

      @@DarshilParmar Does they provide Azure or AWS platform access?

  • @rohitagarwal5319
    @rohitagarwal5319 Před 9 měsíci

    @DarshilParmar I tried using flatten transform in ETL job but it didn't work
    is it because json contains array?
    can you suggest me how to proceed with ETl in few words so that I can work on that

  • @shivarajhalageri2513
    @shivarajhalageri2513 Před 11 měsíci

    Great Darshil i sometime fear these errors ,, bcs there are lot of services each time a service may through a error I feel difficulties in debugging these errors could you please tell how better can I understand these errors nd resolve Thanks in advance🙌

  • @ram.FxTrading
    @ram.FxTrading Před rokem

    thanks

  • @mandardeshpande2246
    @mandardeshpande2246 Před 8 měsíci +1

    Hi Darshil while running the Athena job getting HIVE_CURSOR_ERROR: Row is not a valid JSON Object - JSONException: A JSONObject text must end with '}' at 2 [character 3 line 1]
    This query ran against the "de_database_raw" database, unless qualified by the query. error

  • @ericalbertobernal101
    @ericalbertobernal101 Před rokem

    good job !

  • @satishmajji481
    @satishmajji481 Před rokem +1

    @Darshil Parmar - In part-2 of this video, "region=us/" folder is not created for me; only ca and gb folders are created upon running the ETL job. PS: I added "predicate_pushdown = "region in ('ca','gb','us')" as well but folder is missing for "us" region. Can you please take a look at this?

  • @rohanchoudhary672
    @rohanchoudhary672 Před rokem

    Took me 5 days to complete this video with hands on.
    But these were all worth it.
    - Complete noob me

    • @ishan358
      @ishan358 Před 10 měsíci

      Bro help me to solve run time error @rohanchoudhary672

  • @mairios521
    @mairios521 Před 7 měsíci +1

    Hi Darshil! First of all, I would like to say "Thank you" for this tutorial.
    I need to mention something, I was following each steps but AWS is now different and some options are no longer available or they are so different.
    I can't believe that AWS platform changed so much in just one year.
    My question is: will you update this tutorial in the future?

    • @DarshilParmar
      @DarshilParmar  Před 7 měsíci +1

      Everything is same, you just have to find right options with new UI

    • @mairios521
      @mairios521 Před 7 měsíci

      @@DarshilParmar Thanks for your quick response!!

  • @shubhamgargade4045
    @shubhamgargade4045 Před 2 lety

    Hi Darshil, I'm confused to choose the career between backend developer and data engineer both finds interesting to me. I like more coding than SQL queries which will be better.

    • @MDARUN-ph1dw
      @MDARUN-ph1dw Před 2 lety +2

      backend then because u like coding heavy

    • @DarshilParmar
      @DarshilParmar  Před 2 lety +2

      As Arun said go with Backend because Data Engineering is more SQL heavy

  • @davidaliaga4708
    @davidaliaga4708 Před měsícem

    do you have a end to end but not with AWS? (like hadoop or spark?)

  • @fayssalelaazouziai1573

    Thank you so much Darshil for this video.
    I have a question , does the glue table created automaticaly ? because i get a timout , and i think the problem is whit that . can u please provide with more information like should i create new crawel or what to run the create the cleand_table automaticly

    • @imenbenhassine9710
      @imenbenhassine9710 Před 10 měsíci

      faced the same issue , so i found out that you need to create the cleaned catalog_db in Glue then the cleaned_table will be created automaticly ; for timeout try to increase the memory to along with the time . hope it helps

  • @ashishkumarg5
    @ashishkumarg5 Před 2 měsíci

    What is the similar service in GCP for AWS Glue Crawler ?

  • @AdityaRaj-dp1no
    @AdityaRaj-dp1no Před rokem

    Hey Darshil!
    I have followed the part 2 video till where we have to create an ETL job but I am finding it a bit difficult to create the job as the GUI has changed and the AWS glue studio is updated. Can you please tell me the steps to create the job in similar manner in glue studio GUI?
    PS: I tried few things but I had to delete the jobs as they were not doing the same function as your job in the video.

  • @angelnadar1209
    @angelnadar1209 Před rokem

    Thanks Darshil for this detailed video on project .just needed to know in free trial do we need to pay charges for using aws athena? kindly confirm as when I opened the athena console a msg popped regarding charges for usage.kindly confirm