Complete Azure Data Factory CI/CD Process (DEV/UAT/PROD) with Azure Pipelines
Vložit
- čas přidán 5. 10. 2023
- This video goes over how to write code to package up and promote a DEV Azure Data Factory to a UAT Data Factory and PROD Data Factory.
Links:
-GitHub repo code: github.com/DataEngineeringWit...
-Data Factory automated publishing CI/CD documentation: learn.microsoft.com/en-us/azu...
-Npm Data Factory utilities package: www.npmjs.com/package/@micros... - Věda a technologie
I really appreciate the layout and completeness of .yaml files with Azure Pipeline definition. It gives an idea how real-live project could implement it (unlike some toy examples that are extremely popular in these kind of tutorials)
absolutely stellar! Thank you for the clear instructions and examples. This really helped me understand!!
Great efforts and explanation!!!
Great Content!
really great work
This is awesome, I would love one for Synapse as well as things are slightly different
How would you get this to work with Azure Databricks as a linked service. The connection details aren't parameterised so therefore can pass in values such as the databricks workspace URL
Yoooo thanks
Curious why there is no displayName in the .yml powershell sections?
How do I sett pipeline arguments different in dev, uat and prod?
can we use terraform script instead of using ARM templates?
did you created that yaml files manually or using ADF
Using ADF automatically. Then you just replace your connection/linked service/global parameters in the cicd/adf-cicd template parameters files for the uat and prod environments
Yes I am interested to see how we can use linked templates for deployment. Could you please share it.
+1
Thanks for the comment. I've seen a couple of comments for this. Currently working on a linked template deployment for Data Factory video :)
Could you please explain how to implement parameter replace like link service sql connection string?
Look at the cicd/adf-cicd folder, the uat and prod template parameters files is where you’d replace those values.
please suggest if I need to run 10 pipelines everyday at 2 pm, what is approach, should we go with schedule trigger or any other approach
Schedule trigger.
Hi, I had created this cicd pipeline long back running fine without a single bug since deployed.
Currently arm template size is more than 4mb and getting the error. Would you please do create video for that?
Yes, you'll need to use linked templates. I've seen a couple of comments about this and currently working on a video for it :)
I have tried this but it doesn't create the Global Params in UAT/PROD. Any idea what I could be doing wrong?
Did you by chance forget to click the include global parameters button in the Manage/Arm Template section of your Dev data factory instance?
Do u have discord channel?
How to configure the service connection for uat and prod?
In the cicd/adf-cicd folder, there’s 2 files called adf-uat-template-parameters.json and the prod one. Replace your connection strings for each environment there. See the repo link in the description to get to the files.
@@dataengineeringwithnick7532 should i add the service principal to data factory also as a member? Or just the connection strings in the files
@@dataengineeringwithnick7532 Should I do something in adf with the service principal like add it as a member? Or just the connection strings in the files. I am referring to 23:15 because you didnt show in details what we should do with the azure resource manager
@@mainuser98The Azure Resource Manager (via a service principal) would need to be an RBAC Contributor role (or enough permissions to deploy an ARM template to a resource group) at the resource group level. You don’t need to add the service principal info to any of the template parameter files.
@@dataengineeringwithnick7532 can u send me documentation how to configure this