Streaming ETL With AWS Glue | ETL | AWS Glue | Kinesis Data Stream | Glue Crawler | Glue ETL Job

===================================================================
1. SUBSCRIBE FOR MORE LEARNING :
/ @cloudquicklabs
===================================================================
2. CLOUD QUICK LABS - CHANNEL MEMBERSHIP FOR MORE BENEFITS :
/ @cloudquicklabs
===================================================================
3. BUY ME A COFFEE AS A TOKEN OF APPRECIATION :
www.buymeacoffee.com/cloudqui...
===================================================================
repo link : github.com/RekhuGopal/PythonH...
#etl #awsglue #streaming #crawler #etljob #aws #dataengineeringessentials #sitereliabilityengineering #devops

[@HoustonPillay]

Thank you so much. Perfectly reproducible. Awesome video.

[@cloudquicklabs]

Thank youfor watching my videos.
Glad that it helped you.

[@neelkanthbk]

Thank you Bro, your videos are very helpful. I was stuck in one issue, through your video I got the solution :)

[@cloudquicklabs]

Thank you for watching my videos.
Glad that it helped you.

[@kumaru5796]

thanq nicely explained.

[@cloudquicklabs]

Thank you for watching my videos.
Glad that it helped you.

[@noufalrijal9811]

What will be process if i need to write transformations on the data, by comparing the existing data (previously processed data).

[@cloudquicklabs]

Thank you for watching my videos.
You might need to have two branches one branch taking care of current data , while on another sourcs you are just crawling. And then you are comparing at a single Transform task like "custom transform". You can try many other approaches as well. I shall create video on your scenario if you can explain bit more.

[@noufalrijal9811]

Thanks for the quick response 🙂
My scenario is -
1. The source will be generating some ticketing information via kinesis stream
2. I am creating a report which is an aggregated table from almost 8 other tables
3. We are pushing data to an s3 data lake
4. So I need to to perform all the aggregated transformations related to the report on the flight within the stream

[@cloudquicklabs]

Again when you say aggregate from stream data + table stored data ( May be rds) , are you merging or joining data from stream with table an then storing s3 bucket data lake

[@noufalrijal9811]

Data in kinesis stream will be a CDC from RDS and the tables to join meanse we can say tables from data lake via data catalogues

[@cloudquicklabs]

Okay.. and where is target to store the merge of CDC RDS + Table from Datalake catalog?

[@user-uu8su3xl1f]

you told like will say the iam role cofiguration setting in final. but you did'nt

[@cloudquicklabs]

Thank you for watching my videos.
Apologies if I have not covered but let me tell you that it full admi access with required trust definition. You can watch other videos on ETL I have shown it.