It was nice content. I had an interview question on performance issue of using Merge into query in Databricks and what is the best way to load the data using Spark SQL or PySpark. Could you please make a video on this?
* for upserts use MERGE * MERGE INTO myTable USING myData * Loading via autoloader, you need to read the stream first * val df = spark.readStream.format("cloudFiles") .option("cloudFiles.format", "json") .load("/input/path") df.writeStream.trigger(Trigger.Once) .format("delta") .start("/output/path")
Great...thanks
It was nice content. I had an interview question on performance issue of using Merge into query in Databricks and what is the best way to load the data using Spark SQL or PySpark.
Could you please make a video on this?
* for upserts use MERGE *
MERGE INTO myTable USING myData
* Loading via autoloader, you need to read the stream first *
val df = spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "json")
.load("/input/path")
df.writeStream.trigger(Trigger.Once)
.format("delta")
.start("/output/path")
is there any future demand for Azure Synapse as Fabric is its successor? should we learn fabric instead as a Data engineer?
You can learn the entire Microsoft Fabric in 3 hrs and become a Fabric expert after watching Mr. K's video.