Everything email. Resources that will help your campaigns perform better and make you look like a genius.

Attack of the Clones – Part II

April 24, 2018  •  By: John Kuempel

Environment Cloning with Azure Automation

The Story So Far

In the previous post, we began down the path of cloning our production environment to QA.  In this post, we’ll complete the process by building our project, migrating our database, inserting test data, and running some maintenance operations.

Buildup

An important part of this process is deploying the latest QA build.  We use Visual Studio Online to build and deploy many of our services.  It provides a straight-forward REST API for interacting with projects and builds.  It allows you to do all kinds of things, but we’re only concerned with queuing and monitoring builds.

This is the script we use for queuing and monitoring builds:

workflow queue-team-services-build
{
    param(
        [Parameter(Mandatory=$false)]
        [string] $AzureCredentialAssetName = "AzureCredential",      

        [Parameter(Mandatory=$false)]
        [string] $AzureSubscriptionIdAssetName = "AzureSubscriptionId",

        [Parameter(Mandatory=$true)]
        [string] $VstsAccount,

        [Parameter(Mandatory=$true)]
        [int] $VstsBuildDefinitionId,

        [Parameter(Mandatory=$true)]
        [string] $VstsProject,

        [Parameter(Mandatory=$false)]
        [string] $VstsSourceBranch,

        [Parameter(Mandatory=$false)]
        [string] $BuildConfiguration,

        [Parameter(Mandatory=$false)]
        [string] $ApiEnvironment,

        [Parameter(Mandatory=$false)]
        [string] $DatabaseEnvironment,

        [Parameter(Mandatory=$true)]
        [string] $StorageDestinationKey,

        [Parameter(Mandatory=$false)]
        [string] $Verbosity
    )   
   
   if ($Verbosity) {
       $VerbosePreference = $Verbosity
   }

    InlineScript {
       $url = "https://$using:VstsAccount.visualstudio.com/DefaultCollection/$using:VstsProject/_apis/build/builds?api-version=2.0"

       $body = @{
           "definition" = @{ "id" = $using:VstsBuildDefinitionId }           
       }

       if ($using:VstsSourceBranch) {
           $body.Add("sourceBranch", "$using:VstsSourceBranch")
       }

       $VstsBuildParameters = "{"
       if ($using:BuildConfiguration) {        
           $VstsBuildParameters = $VstsBuildParameters + """BuildConfiguration"": ""$using:BuildConfiguration"","       
       }

       if ($using:Environment) {       
           $VstsBuildParameters = $VstsBuildParameters + """Environment"": ""$using:Environment"","       
       }

       if ($using:DatabaseEnvironment) {
           $VstsBuildParameters = $VstsBuildParameters + """DatabaseEnvironment"": ""$using:DatabaseEnvironment"""
       }

       $VstsBuildParameters = $VstsBuildParameters + """StorageDestinationKey"": ""$using:StorageDestinationKey"""

       $VstsBuildParameters = $VstsBuildParameters + "}"

       if ($VstsBuildParameters -ne "{}") {           
           $body.Add("parameters", $VstsBuildParameters)
       }

       $cred = Get-AutomationPSCredential -Name $using:AzureCredentialAssetName
       Write-Verbose (Add-AzureAccount -Credential $cred)

       $subscriptionId = Get-AutomationVariable -Name $using:AzureSubscriptionIdAssetName
       $null = Select-AzureSubscription -SubscriptionId $subscriptionId

       $headers = @{"Authorization" = "Authorization Token" }
       $bodyJson = ConvertTo-Json $body
       Write-Verbose "Queueing build to $url with body ""$bodyJson"""

       $response = Invoke-WebRequest -Uri $url -Body $bodyJson -ContentType "application/json" -Method Post -ErrorAction Stop -UseBasicParsing -Headers $headers
       Write-Verbose "Response is $response"

       $statusCode = $response.StatusCode
       Write-Verbose "Response code is $statusCode"

       if ($response) {
           if ($statusCode -ne 200) {
               throw "Build request for $using:VstsProject failed with the following status: $response.StatusCode, $response.StatusDescription, $response.Content"
           }

           $payload = ConvertFrom-Json $response.Content
           $buildUrl = $payload.url + "?api-version=2.0"

           Write-Verbose "Monitoring build at $buildUrl"

           $building = $true
           while($building) {
               $response = Invoke-WebRequest -Uri $buildUrl -Method Get -UseBasicParsing -Headers $headers
               $result = ConvertFrom-Json $response.Content

               if ($result.result -eq "succeeded") {
                   Write-Verbose "Build for $using:VstsProject is complete"
                   $building = $false
               } elseif ($result.result -eq "failed") {
                   $failureReason = $result.reason
                   throw "Build for $using:VstsProject failed: $failureReason"
               } else {
                   $status = $result.status
                   Write-Verbose "Build for $using:VstsProject continues: $status!  Sleeping for 60 seconds"
                   Start-Sleep -Seconds 60
               }
           }
       } else {
           throw "Build request for $using:VstsProject failed"
       }
   }
}

The build itself does a lot of heavy lifting with copying blobs, inserting test data, and running data migrations.

Our service also relies on Azure Storage blobs, so we copy those blobs to the test environment to keep everything in sync and avoid having the rebuild the cached data from scratch.  Here’s the script we use for copying the blobs:

Param(
 [Parameter(Mandatory=$true)][string]$SrcAccount,
 [Parameter(Mandatory=$true)][string]$SrcKey,
 [Parameter(Mandatory=$true)][string]$DestAccount,
 [Parameter(Mandatory=$true)][string]$DestKey,
 [Parameter(Mandatory=$true)][string]$Container
)

$srcContext = New-AzureStorageContext -StorageAccountName $SrcAccount -StorageAccountKey $SrcKey
$destContext = New-AzureStorageContext -StorageAccountName $DestAccount -StorageAccountKey $DestKey

$blob = Get-AzureStorageBlob -Container $Container -Context $srcContext | Start-CopyAzureStorageBlob -DestContainer $Container -Context $destContext -Force

$blob | Get-AzureStorageBlobCopyState –WaitForComplete

Inserting the test users requires calling endpoints on the service itself.  We do this through a PowerShell script and store test data in JSON files. Here’s the script:

Param(
 [Parameter(Mandatory=$True)]
 [string]$InputPath,

 [Parameter(Mandatory=$True)]
 [string]$BaseUrl,

 [Parameter(Mandatory=$True)]
 [string]$UserName,

 [Parameter(Mandatory=$True)]
 [SecureString]$Password
)

function CreateRelativeUrl([string]$relativePath)
{
 return (New-Object System.Uri -ArgumentList ($uri.Uri, $relativePath))
}

function GetChildItems($folder)
{
 $jsonFiles = Get-ChildItem ([System.IO.Path]::Combine($InputPath, $folder)) -Filter '*.json'
 if ($jsonFiles -eq $null -or $jsonFiles.Length -eq 0)
 {
  "$folder contains no .json files."
 }

 return $jsonFiles
}

function ReadJsonFile($folder, $file)
{
 $json = ConvertFrom-Json ([System.IO.File]::ReadAllText([System.IO.Path]::Combine($InputPath, $folder, $file)))
 return $json
}

function ConvertToJson($toConvert)
{
 if ($toConvert -is [HashTable])
 {
    $json = '{'
    $i = 0
    foreach($pair in $toConvert.GetEnumerator() | Where { $_.Value } )
    {
      if ($i -gt 0) { $json = $json + ',' }
      $json = $json + '"' + $pair.Key + '":' + (ConvertToJson $pair.Value)
      $i = $i+1
    }

    return $json + '}'
 }
 elseif ($toConvert -is [PSCustomObject])
 {
   $json = '{'
   $i = 0
   foreach($property in $toConvert.psobject.properties)
   {
     if ($i -gt 0) { $json = $json + ',' }
     $json = $json + '"' + $property.Name + '":' + (ConvertToJson $property.Value)
     $i = $i + 1
   }

   return $json + '}'
 }
 else
 {
   return ConvertTo-Json $toConvert
 }
}

function PostObject($objectToPost, $route, $type, $id, $name)
{
 try
 {
   $response = Invoke-WebRequest -UseBasicParsing -Uri $route -Credential $cred -Method Post -Body (ConvertToJson $objectToPost) -ContentType 'application/json'
   if ($response.StatusCode -gt 204)
   { 
     $message = (ConvertFrom-Json $response.Content).message
     return "An error occurred while posting $type {id: $id, name: $name} to $route : $message"
   }
 }
 catch
 {
   return "An exception occurred while posting $type {id: $id, name: $name} to $route : $_"
 }

 return $true
}

function PutObject($objectToPut, $route, $type, $id, $name, $convert=$True)
{
 $body = $objectToPut
 if ($convert -eq $True)
 {
   $body = ConvertToJson $objectToPut
 }

 try
 {
   $response = Invoke-WebRequest -UseBasicParsing -Uri $route -Credential $cred -Method Put -Body $body -ContentType 'application/json'
   if ($response.StatusCode -gt 204)
   {
     $message = (ConvertFrom-Json $response.Content).message
     "An error occurred while putting $type {id: $id, name: $name} to $route : $message"
   }
 }
 catch
 {
    "An exception occurred while putting $type {id: $id, name: $name} to $route : $_"
 }
}

function Data($data, $newData)
{
 $posted = PostObject -objectToPost $newData -route $url -type 'data type' -id $data.key
 if ($posted -ne $true) { $posted }

 $insertUrl = CreateRelativeUrl('relativepath' + $data.key)
 $posted = PostObject -route $insertUrl -type 'insert data' -id $data.key
 if ($posted -ne $true) { $posted }
}

function ProcessData($folder)
{
 $data = ReadJsonFile -folder $folder  -file "data.json"

 foreach ($item in $data)
 {
   $newData = @{
     "key" = $user.key;
     "field" = [System.Guid]::NewGuid().ToString();
   }

   InsertData -data $data -newData $newData
 } 
}

function GetFolders([string]$filter)
{
 return (Get-ChildItem $InputPath -Filter $filter -Attributes D)
}


if ($InputPath -ne $null -And (-Not (Test-Path -Path $InputPath)))
{
 throw "$InputPath does not exist."
}

#this is to bypass any cert issues
add-type @"
 using System.Net;
 using System.Security.Cryptography.X509Certificates;
 public class TrustAllCertsPolicy : ICertificatePolicy
 {
   public bool CheckValidationResult(
     ServicePoint srvPoint, X509Certificate certificate,
     WebRequest request, int certificateProblem)
   {
     return true;
   }
 }
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

$uri = [System.UriBuilder]::new($BaseUrl)
$response = Invoke-WebRequest -UseBasicParsing -Uri $uri -Method Head -ErrorAction Stop
if ($response.StatusCode -ne 200)
{
 throw "$BaseUrl is unreachable."
}

$cred = New-Object PSCredential -ArgumentList ($UserName, $Password)

$dataFolder = GetFolders("data")
if ($dataFolder -eq $null)
{
 "Data folder not found."
}
else
{
 ProcessData($dataFolder)
}

Entity Framework is the ORM layer for the service and handles migrations.  Here’s documentation for its handy migrate.exe tool, which we use to handle database migrations.

Cry Havoc!

After the build and deploy process, there are maintenance tasks that need to be run on the database.  

# perform service maintanence

$username = Get-AutomationVariable "Username"

$password = Get-AutomationVariable "Password"

$credential = New-Object pscredential($userName, (ConvertTo-SecureString $password -AsPlainText -Force))

Invoke-WebRequest -UseBasicParsing -Uri "service endpoint" -Credential $credential -Method Post -TimeoutSec 1500 -ErrorAction Stop

Finally, we can drop the backup we made of the original test database.  The environment has now successfully been cloned and testing can begin.

drop-database `
  -Server $srvrName `
  -ResourceGroupName $dbRsrcGrpName `
  -TargetDatabase $backupDb

Triumph

These posts have shown you how we prepare our testing environment by cloning production data, allowing us to run our tests in an environment as close to production as possible.  Using a combination of Azure Automation, PowerShell, and Visual Studio Online, we’re able to build a sophisticated and comprehensive system that meets our needs. I hope that you can use the information and scripts here to do the same.  Thank you for reading.

Subscribe

Get the inside track on the latest AdTech & MarTech news, trends and strategies.

Blog Form

You direct-sold ads in email that easily translates into money. How? Check out AdServer for Email.

Drive revenue in every email you send. Learn how with RevenueStripe.

Subscribe

Get the inside track on the latest AdTech & MarTech news, trends and strategies.

Blog Form

You direct-sold ads in email that easily translates into money. How? Check out AdServer for Email.

Drive revenue in every email you send. Learn how with RevenueStripe.