Deploying Azure Data Factory With CI/CD Using Azure Pipeline
Vložit
- čas přidán 17. 06. 2024
- This video will focus on how to use Azure Data Factory CI/CD Pipelines and promote ETL Pipelines from Dev environment to UAT then production. How to create an Azure DevOps pipeline to deploy Azure Data Factory pipelines, link services, triggers, datasets, etc to several Azure Data Factories.
Learn a complete tutorial for creating Azure Data Factory CI/CD pipelines with Git enabled that use different branches for developing new features for ETL pipelines and other Azure Data Factory components.
This will cover, creating three Azure Data Factories, two Azure Storage accounts, Azure Release pipeline with Azure Artifacts based on Git repo and deploying Azure Data Factory ARM templates files to another environment.
It will show how to develop all the Azure Data Factory components in the feature branch. Then how to raise a pull request to merge those changes with the master branch. After that publish Azure Data Factory development which will create adf_publish branch which will include the ARM template that describes the current Azure Data Factory Environment. And by using this ARM template, we can deploy it to UAT and Production environments.
Commands:
Stop trigger
---------------
-armTemplate "$(System.DefaultWorkingDirectory)/[artifact source alias]/[SC folder]/ARMTemplateForFactory.json" -ResourceGroupName "[RG name]" -DataFactoryName "[Target ADF Env]" -predeployment $true -deleteDeployment $false
Start trigger
-------------
-armTemplate "$(System.DefaultWorkingDirectory)/[artifact source alias]/[SC folder]/ARMTemplateForFactory.json" -ResourceGroupName "[RG name]" -DataFactoryName "[Target ADF Env]" -predeployment $true -deleteDeployment $true
Stop trigger
---------------
-armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADF-Radwan-Dev/ARMTemplateForFactory.json" -ResourceGroupName "adf-rg" -DataFactoryName "ADF-Radwan-UAT" -predeployment $true -deleteDeployment $false
Start trigger
-------------
-armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADF-Radwan-Dev/ARMTemplateForFactory.json" -ResourceGroupName "adf-rg" -DataFactoryName "ADF-Radwan-UAT" -predeployment $true -deleteDeployment $true
-factoryName "[target ADF name]" -AzureBlobStorage1_connectionString "" -AzureBlobStorage2_connectionString ""
-factoryName [target ADF name] -AzureBlobStorage1_connectionString "" -AzureBlobStorage2_connectionString ""
-factoryName "ADF-Radwan-UAT" -AzureBlobStorage1_connectionString "DefaultEndpointsProtocol=https;AccountName=storageaccradwan1;AccountKey=xx;EndpointSuffix=core.windows.net" -AzureBlobStorage2_connectionString "DefaultEndpointsProtocol=https;AccountName=storageaccradwan2;AccountKey=xxx;EndpointSuffix=core.windows.net"
-armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADF-Radwan-Dev/ARMTemplateForFactory.json" -ResourceGroupName "adf-rg" -DataFactoryName "ADF-Radwan-UAT" -predeployment $true -deleteDeployment $false
About the Author
-----------------------------
Video:
• Mohamed Radwan is a Pr...
Blog:
mohamedradwan.com
Linked-in
/ mohamedahmedradwan
GitHub
github.com/MohamedRadwan-DevOps
Twitter:
/ mradwan06
DevOps #Azure #Azure_Data_Factory_tutorial #Azure_training #Azure_DevOps - Věda a technologie
Good to find you here Mo! This is a great and didactic video, thanks!
Glad you liked it!
Thanks for this detailed walk through. I got up and running with my Dev to UAT deployment really fast following your demo. Really helpful!
Glad it was helpful!
It's a great presentation and a nice walkthrough on creating all the stages. Thanks, Mohammed!
Glad it helped! and thanks for your comment!
There's a slight mistake in the values for Start and Stop Trigger. In the Video Mohamed sets the delete deployment flag to true for clean Resources and Start Trigger. But you also need to set the predeployment flag to false for it to start the triggers again as this is the flag to distinguish between pre and post deployment.
Fantastic! Thank you very much for this detailed video on ADF CICD. I was stuck at an initial point where my account and project were not visible in ADF, it helped. Also, now have a clear understanding how the flow from various branches takes place. This is a very big thing which every ADF developer must know. Thank you very much. Wish you lots of success!
You're very welcome!
Thank You
Mohamed Radwan. Excellent Video
Glad you liked it
Bravo Man !!! Thanks for making this helpful video.
Thanks! This is what I’m looking for.
Glad I could help!
Thanks so much for the video, really helpful!!
Great Video; Detailed explanation. It helped a lot.
Thanks Mari, I am happy you like it!
Thanks Radwan, It's a clear explanation. Looking forward from you to get more videos
Thanks!
Excellent Video! Thanks!
Glad you liked it!
Fantastic...Very good explanation...Thank you Mohamed..
Glad you liked it Thanks for your comment!
Excellent tutorial, thank you.
This is really helping for me...i got so many things which i was looking for..Thanks
Super helpful - Thank you so much Radwan !! #StayBlessednHappy
Excellen video and very thorough explanation..Thank you
Glad it was helpful! Thanks for your comment!
Thanks Mohammed, its a very good one
Glad you liked it! Thanks!
Fantastic walkthrough, helped a lot, thank you so much!! One thing I did need to do for the ci-cd.ps1 to run was under Use Latest PowerShell Version, was to choose Use PowerShell Core under the advanced option below. Otherwise it didn't like the format of the script.
thanks for the end to end demo :)
Very Well Explained
Thanks for the video :)
My pleasure!
This is really awesome
Thanks!
Thanks for the great video, Mohamed!
I was wondering how this would work if the ADF kept on growing and more and more pipelines along with more linked services would be added. Would that require to keep adding connection strings and alike to the "override template parameters" section of the release pipeline?
Nice one!
Thanks!
Excellent
Thank you so much 😀
Thanks for not speeding through this demo!
Hi, Radwan. Thank you very much for this! It saved me a lot of time. I have one little question: what if the data factorys are in two different resource groups? Thank you very much again.
Thanks a lot Mohamed for the great presentation, very informative and straight to the point!
I configured the CICD pipelines for my organization to auto-deploy ADF changes, however I ran into an issue in which the deployment messed up my scheduled triggers start times as it used the start times from the source code which is an old value.
I was wondering if there is a way to deploy all the changes without changing the triggers start times? if no, how we can keep the trigger's start time value from being overwritten?
Hi Nemairi,
Do u know is there any way to release pipeline with disabling adf triggers.. ??
Please, I am stuck 😔
Thanks mate
Happy to help
Great presentation. In a scenario where you have say two developers and each developer works on their own dev branch, would you have two ADF-DEV instances or would the two developers share the single ADF-DEV instance and work on their individual branches via the git integration? I am trying to get my head around what happens in Data Factory as you switch git branches, does all the associated artefacts “swap themselves out”? Does the Data Factory git integration really imply you just have one ADF-DEV and one git branch being developed at a time? Any thoughts you have on this would be most appreciated, thank you again for taking the time to put together such a useful video.
Thanks for the great video, Mohamed Radwan.
The pipelines which you are pushing to the destination(UAT), it reaches to UAT live mode.
Is it possible to move pipelines from DEV to UAT git repo mode instead of livemode ? (in which both DEV and UAT are connected to git.)
Thanks Mohamed, this is really great.
I have a question on stop and start trigger , if i dont include the script which you have added before and after the ARM template deployment, will it just copy the datasets and pipelines but will not trigger the pipeline to copy the blob content to source to destination correct? I have multiple Data Factories in my resource group and is it safe to use the stop and start trigger script, hope that it will not disturb the other Data Factories
Thank you for sharing this relavent information however I would like to ask about data flow and where to make the data transformations ? Thank you in advance :)
This information is very informative and has been incredibly helpful, especially due to its hands-on nature. Instead of creating a single resource group for all three environments (dev, UAT, and prod), I believe it is more practical to create separate resource groups for each environment. This approach better separates the environments. What do you think?
Thanks for the video, Mohamad Radwan , Can you please create a video or guide us how to create a CI-CD pipeline using Azure SQL table. how should we proceed for that.
Hi Mohammed, Thanks for the video. I am using managed identities to blob storage and database. How can i form the variables for "override template parameters"
could you pls help clarifying. thanks.
Very nice video @mohamed, Can you please elborate more on powersell script we are using it. For what purpose it is used?
Awesome work Mohamed, you explained it so well in your video. thank you so much. Mohammed is there anyway you can give access to this presentation(ppt) you prepared for this demo?
Thanks, I am happy it was helpful.
Sorry, I can't find the ppt.
@@MRadwanMSF no worries Mohammed. Please keep up this great work you do in teaching some of us so easily and with step by step guidance. I am looking forward for Azure DevOps with Databricks, Synpase, HDInsights would be great to learn.
So helpful. Thank you. My only problem is that my dev and prod are in different subscriptions and resource groups so I don't quite know how to tie this into your where your dev and prod are in the same subscription. for example. At 40.51 when setting up the ARM template. Should the resource group be your dev or Prod one if they are different?
Glad it was helpful!
This is just a small example but in reality you should parameterize everything and most probably we will use variable group
docs.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml
Thanks for that, But in different environments we have different DBs (connection strings), so how can we manage this dynamically?
Hi! I would like to know if there's a way to avoid clicking on Publish every time you perform a pull request to the master branch. Is that possible? Thank you!
Great presentation man, thank you! Where I can get the ci-cd.ps1 file content?
I am happy it helped. Here is the link where you can find the file
docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment
Excellent video Mohamed. Thank you!!. I have just some questions about deployment in different resource group (Dev, Uat, prod) and each time with different resources names. For example using dev-storeaccount in Dev resource group and uat-storeaccount in Uat resource group and so on ... My first question is : do we use the target resource groupe name in start/stop trigger script ? . Second question how we can handle the parametrization of resource name from an environement to an other ?
Thanks!
You should parameterize everything and most probably we will use variable group
docs.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml
@@MRadwanMSF Thanks !
Hi Radwan ,
Can we parameterize self hosted run time in Azure data factory while doing azure devops ?
great video, I wonder can we just get changed files in repository and only deploy these changed files without all files. I stuck in how to do that
You can use another way that only deploy the ADF pipelines that you want, see this link:
marketplace.visualstudio.com/items?itemName=liprec.vsts-publish-adf
Thanks Mohamed, it was a great presentation.
One question - Instead of Master can we add Powershell script files to adf_publish branch? Is there any pros or cons adding them in adf_publish?
Thanks!
Hi Mohamed. This video was great. I have a question for you regarding ADF. I am facing a problem in creating a single pipeline. Pls reply if you are interested to sort out my problem.
Hi, Radwan. Thanks for the detailed explanation!
I wonder if we can change the dataset source programmatically in release pipeline?
For example: in ADF-Dev I use Azure SQL Database as dataset source, but when publishing to ADF-UAT it automatically changed to SQL Server On-Premise Database (assuming both of data source has the same table structure).
Yes you can do that, you can use YAML and replace function. See the following.
devblogs.microsoft.com/scripting/use-powershell-to-replace-text-in-strings
@@MRadwanMSF Thanks!
Hi Mohamed, Thanks for a great video. Appriciate it. One of my linked service is conencted to Dataverse. In the template parameter i'm unbale to pass Service uri for the uppper environements. i can only able to pass Service connection and its secrect. is there a way to pass the Service URI as well in the parameters? Please advice.
Thank you
Great presentation Sir
I have one question - we have 3 different self hosted integration runtime with different names for each environment dev, uat and prod respectively.
how to parameterize it while deploying to uat and prod env?
Hi Bharath, did you parameterize self hosted IR now ?
what do we do if we have copy pipelines in dev and uat but we dont want them in prod? is there a way to specify which environments to pass copy pipelines into?
I see a task - publish data factory - is this the new form of deployment or not? It has checkboxes which says 'StartStopTriggers' as well
Thanks for this it's been very helpful! I got it working after a lot of trial and error.
One question and maybe I'm misunderstanding, but shouldn't the start trigger be -predeployment $false ?
Glad it helped!
Yes, for the start trigger and if I didn't do that, it's my mistake :-)
but how we would change the storage accounts for UAT and prod as they would be different for UAT and prod ?
What's the difference between adf and master branch? Is master a staging and adf actual master in each environment?
Can I use multiple ADF for developing one pipeline in one repository?
For the Data Factory Scheduled Trigger, in my Dev the Trigger status is Stop and I would like to ask if I deploy it to Test or Prod would the "Stop Trigger" and "Clean Resource Resource and Start " task would start the status of the Data Factory Scheduled Triggers?
The script should start all triggers
I just have a question . Where is the build pipeline ?. Don’t we have build pipeline like other services for adf?
Mohammed, having issues to deploy data factory with global parameters.. can you send any link or info regarding this..
Predeployment should be set to false for starting triggers , isnt it?
Unable to connect the AAD in Azure devops. Error is the current organization policy does not allow guest users to access the organization . Change the policy setting to external guest access
Hi can you help me in stop trigger task while selecting Azure subscription I'm not able to see the subscription list..what should I do if want to see list of subscriptions?
Hi Radwan,
This is a great video!
I have a question - in this video, you have demonstrated that all those changes made in Dev will be deployed on UAT and Production in the same CD pipeline. However, in the real time scenario, if I make 10 changes in Dev, all those 10 changes should be deployed on UAT. Imagine, out of those 10, we got an approval to deploy only 4 changes onto Production, and the rest of the 6 should still be in UAT for extended testing. I am curious to understand how can we deploy only those 4 changes onto production. I look forward to having answer, and that will be of great help.
Thanks in advance!
Did you get a solution for this?
So the UAT and Prod don't need to be associated with Azure DevOps Git?
Yes, only Dev assoicated with Git and the rest of envs are not
Hi
I'm unable to add custom parameters for Snowflake.
Could you please get back to me?!
Great video man! Thanks
We need the powershell script that you copied from microsoft website at the time of creating a file.On microsoft website it is not available. Please watch video at 34.26 time and guide me
هل ممكن حضرتك تعملنا فيديو بالعربي شرح ل azure data factory
Please share the yaml pipeline
Hi, can you send powershell script link which you have copied from microsoft website
Hi Radwan, I have one Databricks. How to deploy it using CI/CD
You need to use this tasks
marketplace.visualstudio.com/items?itemName=riserrad.azdo-databricks
Hi,
I have a scenario and need community advise to manage it in best possible way.
Due to compliance issues, I am not allowed to save our source code in any Geo location supported by Azure Devops Services. so under repo when we create a repo, by default its created somewhere in Geo location, ideally where azure devops service is hosted.But this approach is NOT-OK for us.
And I want to use our own server "hosted in MS-azure" as repo server.so whenever, we create a repo using create repo under the project repositories this should be stored in that particular server.. please advise how can i achieve this?
our own server is having Visual studio 2019 and connected to azure devops also.
you can use tfs server deploy local.
But now the arm approach is having limitations with parameter limit of 256. Secondly the concept of branches in this approach isn't ideal and acceptable from MS.
بدنا دورات بالعربي
احضر المحاضرة الجاية
czcams.com/video/dDZw1K7ow68/video.html
Hi Mohamed,
I have follwed the same setps which you explied in the video. I'm getting below error while running the pipeline. Please help me out in this.
Error:
2023-11-03T06:31:42.8563878Z ##[error]Could not find the modules: 'Az.Accounts' with Version: ''. If the module was recently installed, retry after restarting the Azure Pipelines task agent.
2023-11-03T06:31:42.9672510Z ##[error]PowerShell exited with code '1'.
2023-11-03T06:31:45.6177222Z ##[command]Disconnect-AzureRmAccount -Scope Process -ErrorAction Stop
2023-11-03T06:31:45.7413326Z ##[warning]Run Connect-AzureRmAccount to login.