Video

Very informative ! 🤝

Does anybody have any idea what this error means: DBFS file browser StorageContext com.databricks.backend.storage.StorageContextType$DbfsRoot$@5c512926 for workspace 3667672304132597 is not set in the CustomerStorageInfo. I am not sure what this means. For context, I had uploaded some files yesterday on DBFS for practice purposes. Those files are still available if I try to find them through notebooks, however, the DBFS tab can't show them and throws this error. Any help will be much appreciated!

Hey Manoj, great session on Data extraction via APIs. Few points I'd like to share from my experience working on this: 1) While dealing with huge volumes of data from source it's crucial to involve Pagination to iteratively collect all data. 2) Admins who manage these end_point URLs usually discourage having multiple API calls within a certain timeframe which would cause a dead-lock on your batch id. Like you suggested either have bulk data pulled all at once or optimize our framework.

Just completed this amazing project 😍 Can i add this in my portfolio?

I see this when trying to upload files using DBFS file browser Missing credentials to access AWS bucket

Thanks for the content, really beneficial.

Good efforts however please minimize the usage of words like "perfect" and "well & good" after every sentence

Great explanation thanks

subtitles are not fully visible

super manoj..txs for u time as you said its really tough usecase if possible do it with easy data set next time bcz every one can understand easliy keep going....👌

My Doubt is --> In spark we have cache and persist, used to save the RDD. As per my understanding cache and persist/MEMORY_AND_DISK both perform same action for DataFrames. If this is the case why should I prefer using cache at all, I can always use persist [with different parameters] and ignore cache. Could you please let me know, when to use cache, or if my understanding is wrong.

with tab1 as( SELECT order_id, max(quantity) as max_value, avg(quantity) as avg_quantity FROM orders GROUP BY order_id) SELECT order_id FROM tab1 WHERE max_value > ALL(SELECT avg_quantity FROM tab1 )

Thank you for your time and efforts. Its not a easy job 👏👏👏

important content is Questions asked 😅😅

Appreciate your great effort and share your knowledge brother!👍

please add the dataset

can you add the dataset link

Can you please help us with the dataset?

awesome sir

which pentab/device is used for video , can you please share ?

with cte as ( select order_id,AVG(quantity) as avg_quantity from t1 group by order_id ) select order_id from t1 where quantity > cte. avg_quantity and order_id=cte.order_id;

how come the project discussion is happening in the first round. Usually, they asked about Python, and SQL questions in the first round to check the basic foundation. Correct me if I am wrong here

Don't you think his intro was too long and the interviewer has to cut him in between to ask questions quickly?

Why would you setup a session in spark when you already get it ?

Replied in my previous question but it seems not visible so making again. Getting in first pipeline. AnalysisException: Failed to merge fields 'customer_id' and 'customer_id' Any suggestion would be appreciated. Thank you Please find the ap code I am trying to follow along here: github.com/maaz-ahmed-ansari/apple-product-analysis/tree/main

Getting in first pipeline. AnalysisException: Failed to merge fields 'customer_id' and 'customer_id' Any suggestion would be appreciated. Thank you

SELECT name, department_name, MAX(DATEDIFF(promotion_date, hire_date)) AS longest_time FROM employee JOIN department ON employee.dept_id = department.dept_id JOIN promotion ON employee.employee_id = promotion.employee_id GROUP BY department_name ORDER BY longest_time DESC;

This also increases the data size of the second dataset as we explode it, so is it still optimized as the data scan will be increased a lot and lot of shuffle will be involved?

A practical demonstration will be relaxed tomorrow. Kindly watch this video to understand the theory in depth.

Thank you , waiting

nums = [0,0,2,3,3,3,3,5,5] k = 2 new_nums = [] for i in nums: if nums.count(i) == k: if i not in new_nums: new_nums.append(i) print(new_nums)

@The Big Data Show Do you develop restful webservices or rest API as well as a data engineer?

Can you do even a small project using Kafka?

Great explanation Ankur !!

Great video Ankur. Being following your content and blogs via linked-In Congrats !!

Hello I am getting error to read delta table that is on default at 01:21:50 IllegalArgumentException: Path must be absolute: default.customer_delta_table_persist.Please help me through that

Please mention the experience of the interviewee either in caption or in thumbnail. It would be helpful

The best playlist who wants to learn Kafka in data engineering domain. Every video has a clear cur explanation. Hope it will be continued. It can be a one stop tutorial for those who wants to learn kafka.