The common practice in Azure Data Factory ( ADF ) CICD pipeline is to make the deployment from the ARM templates present in the adf_publish branch. The detailed explanation of the Continuous Integration and Continuous Delivery is explained in the Azure Documentation.
The following diagram explains the complete flow of the ADF CICD. The below image is taken from the Microsoft Azure Website. The left side of the image shows the code movement in the Git and the right side of the diagram shows the deployment of ADF in the various Azure environments.
The high level summary of the usual flow is given below.
- The developers develop the features in their feature branch
- After completing the development and verifying the feature, the developers raise pull request to the master branch.
- The approvers/reviewers review the changes and approve the merge request. The features will be merged to the master branch
- master branch is the collaboration branch. The final set of pipelines will be available in the master branch. The pipelines can be verified by selecting the master branch in the ADF.
- After the review, by clicking on Publish button on the Data Factory, the ARM templates for the complete ADF artifacts will be generated and published to a default branch with the name adf_publish
- The master branch will have the JSON definition of the pipelines, datasets, linked services etc. ARM templates are not present in the master branch.
- The adf_publish branch has the ARM templates
The usual deployment happens from the adf_publish branch. The ARM templates of the ADF will be deployed in target environment using the ARM Template Deployment Task in the Release pipeline.
In some cases people want to take the json objects from the master branch and use it for deployment. In this case, we will not be able to use the ARM Template Deployment Task. We will have to use a different approach to do this deployment. The detailed steps of this approach is explained in the below video.