Maester is a PowerShell based Microsoft Security test automation framework designed to help you maintain control over your Microsoft tenant's security configuration. Maester is an initiative created by Merill Fernando, Faben Bader, and Thomas Naunheim. In this blog, I will demonstrate the new Maester feature called multi-tenant reporting. This allows you to run your security tests across multiple tenants and view the results in a single report. This setup enables monthly security checks across your Microsoft tenants. 🔥

The first way to generate Maester multi-tenant reporting is to run it locally using PowerShell. To create a multi-tenant report, run Maester against each tenant separately, save the JSON output, then merge the results using Merge-MtMaesterResult.

# Create Maester Test Results folder
md maester-tests
cd maester-tests

# Installing the Maester Tests
Install-MaesterTests

# Run Maester against three tenants and save JSON results
Connect-MgGraph -TenantId $tenantProduction
Invoke-Maester -PassThru -OutputJsonFile ./production.json
Disconnect-MgGraph

Connect-MgGraph -TenantId $tenantDevelopment
Invoke-Maester -PassThru -OutputJsonFile ./development.json
Disconnect-MgGraph

Connect-MgGraph -TenantId $tenantChina -Environment China
Invoke-Maester -PassThru -OutputJsonFile ./china.json
Disconnect-MgGraph

# Generate the multi-tenant HTML report
Merge-MtMaesterResult -Path *.json | Get-MtHtmlReport | Out-File ./MultiTenantReport.html

I start by running Maester against my first tenant, which is the production tenant.

After that, I ran Maester against my development tenant. Both results are now available, which can be verified using the production.json and development.json files. Once both tenants have been checked, you can merge the results using the Merge-MtMaesterResult function.

When opening the Maester report, you can see multiple tenants, which is expected since I ran tests against both my production and demo tenants.

We can now review each individual test per tenant without switching between different Maester test results. 😍

Maester Multi-tenant reporting through CI/CD with Azure DevOps

What we've seen so far is Maester multi-tenant reporting running locally. However, I prefer a more predictable approach, so I'm using Azure DevOps to fully leverage our CI/CD strategy. For this, I use the following YAML pipeline definition. It is based on the Maester documentation and has been adapted to fit my requirements. In the original Maester example, the merged report is published to an Microsoft Azure Web App. In my version, I publish it as a pipeline artifact in Azure DevOps instead. Choose the option that best fits your needs.

trigger: none

parameters:
  - name: tenants
    type: object
    default:
      - name: Production
        serviceConnection: sc-maester-production
        tenantId: <your-production-tenant-id>
        clientId: <your-production-client-id>
        environment: Global
        includeTeams: true
        includeExchange: true
        includeISSP: true
        organizationName: contoso.onmicrosoft.com
      - name: Development
        serviceConnection: sc-maester-development
        tenantId: <your-dev-tenant-id>
        clientId: <your-dev-client-id>
        environment: Global
      - name: China
        serviceConnection: sc-maester-china
        tenantId: <your-china-tenant-id>
        clientId: <your-china-client-id>
        environment: China

variables:
  ResultsDir: $(Pipeline.Workspace)/maester-results

schedules:
- cron: "0 6 * * *"
  displayName: daily at 06:00
  always: true
  branches:
    include:
    - main

jobs:
- job: maester
  timeoutInMinutes: 0
  pool:
    vmImage: ubuntu-latest

  steps:
  - checkout: self
    fetchDepth: 1

  - task: AzurePowerShell@5
    inputs:
      azureSubscription: ${{ parameters.tenants[0].serviceConnection }}
      ScriptType: 'InlineScript'
      pwsh: true
      azurePowerShellVersion: latestVersion
      Inline: |
        Install-Module 'Pester', 'NuGet', 'PackageManagement', 'Microsoft.Graph.Authentication', 'ExchangeOnlineManagement', 'MicrosoftTeams' -Confirm:$false -Force
        Install-Module 'Maester' -RequiredVersion 2.0.68-preview -AllowPrerelease -Confirm:$false -Force
        New-Item -ItemType Directory -Force -Path '$(ResultsDir)'
    displayName: 'Install required modules'

  - ${{ each tenant in parameters.tenants }}:
    - task: AzurePowerShell@5
      inputs:
        azureSubscription: ${{ tenant.serviceConnection }}
        ScriptType: 'InlineScript'
        pwsh: true
        azurePowerShellVersion: latestVersion
        Inline: |
          $includeExchange = '${{ tenant.includeExchange }}'.Trim().ToLower() -eq 'true'
          $includeTeams = '${{ tenant.includeTeams }}'.Trim().ToLower() -eq 'true'
          $includeISSP = '${{ tenant.includeISSP }}'.Trim().ToLower() -eq 'true'
          $TenantId = '${{ tenant.tenantId }}'
          $ClientId = '${{ tenant.clientId }}'
          $Environment = '${{ tenant.environment }}'

          switch ($Environment) {
              'China' {
                  $graphUrl = 'https://microsoftgraph.chinacloudapi.cn'
                  $graphEnvironment = 'China'
                  $outlookUrl = 'https://partner.outlook.cn'
                  $exchangeEnv = 'O365China'
                  $complianceUrl = 'https://ps.compliance.protection.partner.outlook.cn'
              }
              'USGov' {
                  $graphUrl = 'https://graph.microsoft.us'
                  $graphEnvironment = 'USGov'
                  $outlookUrl = 'https://outlook.office365.us'
                  $exchangeEnv = 'O365USGovGCCHigh'
                  $complianceUrl = 'https://ps.compliance.protection.office365.us'
              }
              'USGovDoD' {
                  $graphUrl = 'https://dod-graph.microsoft.us'
                  $graphEnvironment = 'USGovDoD'
                  $outlookUrl = 'https://outlook.office365.us'
                  $exchangeEnv = 'O365USGovDoD'
                  $complianceUrl = 'https://ps.compliance.protection.office365.us'
              }
              'Germany' {
                  $graphUrl = 'https://graph.microsoft.de'
                  $graphEnvironment = 'Germany'
                  $outlookUrl = 'https://outlook.office.de'
                  $exchangeEnv = 'O365GermanyCloud'
                  $complianceUrl = 'https://ps.compliance.protection.outlook.de'
              }
              default {
                  $graphUrl = 'https://graph.microsoft.com'
                  $graphEnvironment = 'Global'
                  $outlookUrl = 'https://outlook.office365.com'
                  $exchangeEnv = 'O365Default'
                  $complianceUrl = 'https://ps.compliance.protection.outlook.com'
              }
          }

          $graphToken = Get-AzAccessToken -ResourceUrl $graphUrl -AsSecureString
          Connect-MgGraph -AccessToken $graphToken.Token -Environment $graphEnvironment -NoWelcome

          if ($includeExchange) {
              Import-Module ExchangeOnlineManagement
              $outlookToken = (ConvertFrom-SecureString -SecureString (Get-AzAccessToken -ResourceUrl $outlookUrl -AsSecureString).Token -AsPlainText)
              Connect-ExchangeOnline -AccessToken $outlookToken -AppId $ClientId -Organization $TenantId -ExchangeEnvironmentName $exchangeEnv -ShowBanner:$false

              if ($includeISSP) {
                $ISSPToken = (ConvertFrom-SecureString -SecureString (Get-AzAccessToken -ResourceUrl $complianceUrl -AsSecureString).Token -AsPlainText)
                Connect-IPPSSession -AccessToken $ISSPToken -Organization '${{ tenant.organizationName }}'
              }
          }

          if ($includeTeams) {
              Import-Module MicrosoftTeams
              $teamsToken = Get-AzAccessToken -ResourceUrl '48ac35b8-9aa8-4d74-927d-1f4a14a0b239' -AsSecureString
              $regularGraphToken = ConvertFrom-SecureString -SecureString $graphToken.Token -AsPlainText
              $teamsTokenPlainText = ConvertFrom-SecureString -SecureString $teamsToken.Token -AsPlainText
              Connect-MicrosoftTeams -AccessTokens @($regularGraphToken, $teamsTokenPlainText)
          }

          $runFolder = Join-Path "$(Agent.TempDirectory)" '${{ tenant.name }}-tests'
          New-Item -ItemType Directory -Force -Path "$runFolder"
          Push-Location $runFolder
          Install-MaesterTests .\tests

          $jsonFile = Join-Path '$(ResultsDir)' '${{ tenant.name }}.json'
          Invoke-Maester -OutputJsonFile $jsonFile -PassThru -Verbosity Normal
          Pop-Location

          # Disconnect all sessions to ensure tenant isolation between steps
          Disconnect-MgGraph -ErrorAction SilentlyContinue
          if ($includeExchange) {
              Disconnect-ExchangeOnline -Confirm:$false -ErrorAction SilentlyContinue
          }
          if ($includeTeams) {
              Disconnect-MicrosoftTeams -ErrorAction SilentlyContinue
          }
      displayName: 'Run Maester tests (${{ tenant.name }})'
      
  - task: PowerShell@2
    inputs:
      targetType: 'inline'
      pwsh: true
      script: |
        $resultsDir = '$(ResultsDir)'

        # Merge all tenant JSON results and generate the report
        $merged = Merge-MtMaesterResult -Path $resultsDir
        $date = (Get-Date).ToString("yyyyMMdd-HHmm")
        $outputDir = Join-Path "$(Agent.TempDirectory)" "report-$date"
        New-Item -ItemType Directory -Force -Path $outputDir
        Get-MtHtmlReport -MaesterResults $merged |
            Out-File -FilePath (Join-Path $outputDir 'index.html') -Encoding UTF8

        $zipPath = Join-Path "$(Agent.TempDirectory)" "MaesterReport$date.zip"
        Compress-Archive -Path (Get-ChildItem -Path $outputDir).FullName -DestinationPath $zipPath

        if (-not (Test-Path $zipPath)) {
            throw "Zip file was not created at: $zipPath"
        }
        Write-Host "##vso[task.setvariable variable=MaesterZipPath]$zipPath"
    displayName: 'Merge results and generate multi-tenant report'

  - publish: $(MaesterZipPath)
    displayName: Publish Maester HTML Report
    artifact: MaesterHtmlReport

Both my production and demo tenants are connected to Azure DevOps, as shown in the service connections in the screenshot below. 💪🏻

For granting the service connections the appropriate permissions, you can use the following PowerShell script.

# Connect to Microsoft Graph
Connect-MgGraph

# Object ID
$clientSpId = ""

# Get the Microsoft Graph Service Principal ID
$graphSp = Get-MgServicePrincipal -Filter "appId eq '00000003-0000-0000-c000-000000000000'"
if (-not $graphSp) { throw "Graph service principal not found" }

# Assign the Policy.Read.All app role to the client service principal

$rolesToAssign = @(
  'DeviceManagementConfiguration.Read.All'
  'DeviceManagementManagedDevices.Read.All'
  'DeviceManagementServiceConfig.Read.All'
  'OnPremDirectorySynchronization.Read.All'
  'OrgSettings-AppsAndServices.Read.All'
  'OrgSettings-Forms.Read.All'
  'Directory.Read.All'
  'DirectoryRecommendations.Read.All'
  'IdentityRiskEvent.Read.All'
  'Policy.Read.All'
  'Policy.Read.ConditionalAccess'
  'PrivilegedAccess.Read.AzureAD'
  'Reports.Read.All'
  'RoleEligibilitySchedule.Read.Directory'
  'RoleManagement.Read.All'
  'SecurityIdentitiesSensors.Read.All'
  'SecurityIdentitiesHealth.Read.All'
  'SharePointTenantSettings.Read.All'
  'ThreatHunting.Read.All'
  'UserAuthenticationMethod.Read.All'
  'DeviceManagementRBAC.Read.All'
  'ReportSettings.Read.All'
)

$roles = $graphSp.AppRoles | Where-Object {
    $_.Value -in $rolesToAssign -and $_.AllowedMemberTypes -contains "Application"
}

if (-not $roles) {
    throw "No matching app roles found on Graph"
}

# Validate client service principal
$clientSp = Get-MgServicePrincipal -ServicePrincipalId $clientSpId -ErrorAction Stop

"clientSpId : $($clientSp.Id)"
"resourceId : $($graphSp.Id)"

foreach ($role in $roles) {

    "Assigning role: $($role.Value)"

    New-MgServicePrincipalAppRoleAssignment `
      -ServicePrincipalId $clientSp.Id `
      -PrincipalId $clientSp.Id `
      -ResourceId $graphSp.Id `
      -AppRoleId $role.Id
}

I've added the pipeline based on the YAML file provided above and started it so it can connect to the different tenants and retrieve the information for our Maester scan.

After waiting a couple of minutes, we can see that the pipeline has finished running and that the pipeline artifact has been published.

It is called MaesterHtmlReport and contains the ZIP-file with the merged Maester HTML report for insights. Which I can now use to check the security posture of my Microsoft tenants. 🔥

Starting now, I will receive a monthly report on the security configuration of my Microsoft tenants. This will allow me to act on the recommendations and enhance the security of my Microsoft tenants. That's all for today! Feel free to share your feedback, and I look forward to our next session. Until then, take care! 💛