Abp: CI/CD guidelines

[aklscv]

While this is not directly related to the internals of ABP, I would like to hear other people opinions about how they are setting up CI/CD with ABP.

I manage to setup the CI part (which is building, testing and publishing the project) but I am a bit confused with the CD part. How are you guys dealing with the deploy? Do you first publish the DbMigrator and execute a dotnet command line on the DLL and then deploy the web project itself?

This question came up to me while reading #3574. Maybe @Stirda can fill in.

[Stirda]

This is my typical Azure Pipelines YAML file for a Build Pipeline in a project based on an ABP ASP.NET Core MVC Template. For brevity I omitted pipeline/branch triggers and SonarQube tasks.

pool:
  vmImage: 'windows-latest'

variables:
  BuildConfiguration: release
  LocalPublishWebAppFolder: WebApp
  LocalPublishDbMigratorFolder: DbMigrator

steps:

- task: DotNetCoreCLI@2
  displayName: Restore NuGet packages
  inputs:
    command: restore
    verbosityRestore: minimal

- task: DotNetCoreCLI@2
  displayName: Build solution
  inputs:
    command: build
    arguments: --configuration $(BuildConfiguration) --no-restore
    configuration: $(BuildConfiguration)

- task: DotNetCoreCLI@2
  displayName: Run tests
  inputs:
    command: test
    projects: '**/*.Tests.csproj'
    publishTestResults: true
    arguments: --configuration $(BuildConfiguration) --collect "Code coverage"
    nobuild: true

- task: DotNetCoreCLI@2
  displayName: Gather web app binaries for publication
  inputs:
    command: publish
    publishWebProjects: false
    projects: '**/*.Web.csproj'
    modifyOutputPath: false
    configuration: $(BuildConfiguration)
    arguments: --configuration $(BuildConfiguration) --no-restore --no-build --output $(Build.ArtifactStagingDirectory)/$(LocalPublishWebAppFolder)
    nobuild: true
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))

- task: DotNetCoreCLI@2
  displayName: Gather migration app binaries for publication
  inputs:
    command: publish
    publishWebProjects: false
    projects: '**/*.DbMigrator.csproj'
    configuration: $(BuildConfiguration)
    arguments: --configuration $(BuildConfiguration) --no-restore --no-build --output $(Build.ArtifactStagingDirectory)/$(LocalPublishDbMigratorFolder)
    nobuild: true
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))

- task: PublishBuildArtifacts@1
  displayName: Upload web app publication files to artifact
  inputs:
    PathtoPublish: $(Build.ArtifactStagingDirectory)/$(LocalPublishWebAppFolder)
    ArtifactName: WebApp
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))

- task: PublishBuildArtifacts@1
  displayName: Upload migration app publication files to artifact
  inputs:
    PathtoPublish: $(Build.ArtifactStagingDirectory)/$(LocalPublishDbMigratorFolder)
    ArtifactName: DbMigrator
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))

And below is the conversion in YAML code for the associated Release Pipeline tasks. Here are some preconditions:

• In order to have the FileTransform@1 working, you have to set ConnectionStrings.Default in release variables or use project library and link.
• In order to have a DbMigrator binary file compatible with CD, you also need a ABP 2.6 template or newer, or apply these changes to your DbMigrator project. See why in https://github.com/abpframework/abp/issues/3574.
• Remember to replace the value of the MigratorDllFilename variable below with the name of your actual migrator binary file.

variables:
  UnzippedMigrationAppFolderPath: 'UnzippedMigrationApp'
  MigratorDllFilename: 'MyCompany.MyProject.DbMigrator.dll'

steps:

steps:
- task: AzureAppServiceManage@0
  displayName: Stop web app
  inputs:
    azureSubscription: '$(Parameters.ConnectedServiceName)'
    Action: 'Stop Azure App Service'
    WebAppName: '$(Parameters.WebAppName)'

- task: ExtractFiles@1
  displayName: Unzip migrator binaries
  inputs:
    archiveFilePatterns: '**/DbMigrator/*.zip'
    destinationFolder: '$(UnzippedMigrationAppFolderPath)'

- task: FileTransform@1
  displayName: Set migrator connection string
  inputs:
    folderPath: '$(UnzippedMigrationAppFolderPath)'
    fileType: json
    targetFiles: appsettings.json

- powershell: 'dotnet $(MigratorDllFilename)'
  workingDirectory: '$(UnzippedMigrationAppFolderPath)'
  displayName: Run migrator

- task: AzureRmWebAppDeployment@4
  displayName: Deploy web app
  inputs:
    azureSubscription: '$(Parameters.ConnectedServiceName)'
    appType: '$(Parameters.WebAppKind)'
    WebAppName: '$(Parameters.WebAppName)'
    packageForLinux: '$(System.DefaultWorkingDirectory)/$(Release.PrimaryArtifactSourceAlias)/WebApp/WebApp.zip'
    enableCustomDeployment: true
    ExcludeFilesFromAppDataFlag: false

steps:
- task: AzureAppServiceManage@0
  displayName: Start web app
  inputs:
    azureSubscription: '$(Parameters.ConnectedServiceName)'
    Action: 'Start Azure App Service'
    WebAppName: '$(Parameters.WebAppName)'

Some additional explanations on build pipeline tasks:

• Separating dotnet restore from dotnet build gives informations on time costs.
• The --collect "Code coverage" argument gives the abitility to Azure DevOps to feed the associated tab in build results page, and to a SonarQube server to study code coverage level.
• Whole solution build is not needed for release but good for CI principles.
• If you want to be able to release also to a testing environment during pull request process, remove condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master')) lines. If not, these lines speed up your build by avoiding useless tasks during pull request automatic build.
• Azure DevOps recently moved to ZipDeploy by default when using AzureRmWebAppDeployment@4 task. This locks wwwroot to read-only. This messes up my most recent ABP-based project startup by loading a frustrating HTTP Error 500.30 - ANCM In-Process Start Failure error page and makes me unable to edit web.config file remotely. The task above explicitely asks the good old WebDeploy mode.

[aklscv]

Thank you very much for your detailed explanation. I had something similar in mind but not at this level yet. This will save myself and others a lot of time.

[Stirda]

I updated my comment by adding two things.

First, I added --configuration $(BuildConfiguration) to build task and the two publish tasks. Even if configuration: $(BuildConfiguration) is set, build was done in Debug mode. This causes verbose logs. This seems to be a know DotNetCoreCLI@2 bug : https://github.com/microsoft/azure-pipelines-tasks/issues/9073

I added two tasks to the release pipeline YAML code above too. One task stopping the web app and another starting it. This might prevent DbMigrator for breaking the web app execution.

[luisp88]

Very useful. Just one question: when database is already working and execute dbmigrator, it applies migrations and also dataseeders, and this results in duplicates rows. How can I tell dbmigrator to run only migrations and not dataseeders? or do I have to verify in each dataseeder if table is not empty first?

[Stirda]

when database is already working and execute dbmigrator, it applies migrations and also dataseeders, and this results in duplicates rows (...) do I have to verify in each dataseeder if table is not empty first?

Seeds are intended to be idempotent. So yes you have to check first if each seeded data is present in your database to avoid duplicates. See how Volo.Abp.Identity.Domain seeds the admin user and the admin role.

How can I tell dbmigrator to run only migrations and not dataseeders?

I think the DbMigrator project cannot disable data seeders. But, if you really want your seeds to not be idempotent, you can always override them all.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.