From CVE to Closed Ticket: Automating Vulnerability Management with IBM Concert Protect

In 2017, the Equifax breach exposed sensitive data on 147 million people. The attackers got in through a known Apache Struts vulnerability, CVE-2017–5638. A patch had been available for about two months. They stayed in Equifax's systems for 76 days. The security team had been warned about the vulnerability but never patched the affected systems, and the breach went on to cost the company well over a billion dollars.

The lesson is not that Equifax lacked vulnerability data. The problem was that nothing connected that data to the systems running in production and made sure someone acted on it.

IBM Concert addresses that gap. It assesses your risk posture by tracking critical indicators (dimensions) across your application landscape. Concert maps application components, environments and access points from SBOM files or automated ingestion jobs. It then ingests vulnerability scans, calculates a risk score for each finding and automatically opens tickets in Jira, GitHub or ServiceNow.

In this post, you'll integrate Concert with Instana in both directions:

1. A Jenkins pipeline builds the Quote of the Day (qotd) microservice application, pushes the images to the GitHub Container Registry (GHCR) and deploys them to a K3s cluster.
2. Instana monitors the application and sends its SBOM to Concert.
3. A second pipeline scans the images with Trivy and uploads the results to Concert.
4. Concert prioritises the CVEs, opens GitHub issues for any with a risk score of 4 or higher, and serves the results to Instana's CVE sensor.
5. Finally, you patch one vulnerability and verify that the fix shows up in Concert, GitHub and Instana.

The two-way part is what makes this integration worth building. In one direction, Instana tells Concert which images, namespaces and services make up the application, so nobody has to generate and upload SBOMs by hand. In the other, Concert sends its CVE assessments back to Instana, so the SRE team sees vulnerabilities right next to the application it already monitors. Neither team has to switch tools to get the full picture.

log in with concert_username / concert_password.

On the welcome page, choose the Resilience option on the right.

On the next page, click Skip.

Generate the Concert API key

Click the API Key icon in the top right.

Click Generate API key.

Save the value under API key on the Concert API Key line and close the window with X.

Create the Instana API token

log in with Instana admin@instana.local / passw0rd

From the left menu open Settings → Security & access → API tokens and click New API token.

Enter concert-token as the name. Concert only uses read-only APIs, so the default permissions are enough.

Click Save at the bottom and write the token on the Instana API Token line.

Fork the qotd repository

Log in to GitHub and fork the project

Write your fork's URL on the GIT_URL line. It should look like this:

https://github.com/eozkurt/concert-vulnerability-lab

Instana integration

Concert pulls application components, Kubernetes clusters, namespaces, image names and their relationships from Instana automatically, so you no longer generate and upload SBOMs by hand. In the other direction, the CVE sensor in an Instana agent brings Concert's CVE assessments into the Instana UI.

Create the Instana connection in Concert

Select the top Resilience tab and open Administration → Integrations from the left menu.

On the Connections tab click Create connection

and select IBM Instana Observability

Fill in the fields, click Validate connection, and when it shows Successful, click Create.

Define the production environment

In Concert an environment groups related applications and inventory data. The data coming from Instana lands in this environment.

Open Inventory → Environment inventory from the left menu.

Click Define environment → From resources.

Enter production as the Name, choose Production as the Purpose and click Create.

Create the Instana ingestion job

In the Concert tab, open Administration → Integrations from the left menu. On the Ingestion jobs tab click Create ingestion job.

Back up the agent configuration on bluebox

The CVE sensor pulls vulnerability data from Concert and shows it on Instana's Vulnerabilities tab. You enable it in the Instana agent on bluebox. First connect to bluebox from the bastion terminal:

Back up the configuration file before changing it:

Configure the CVE sensor

sudo vi /opt/instana/agent/etc/instana/configuration.yaml

1. In vi, search for the line containing cve with /cve.
2. Leave the sample CVE entry as it is and paste the block below under the # CVE line.
3. Put your Concert API key in api_key, inside single quotes. Concert uses a self-signed certificate, so the last line ignore_cert: true is required.
4. Check the indentation is 2 spaces and save with :wq.

Restart the agent and verify the sensor

The agent should show active (running). Then check the log that the sensor is enabled:

Build and deploy the application

qotd is a microservice application made of the web, quote, ratings, pdf, engraving and author services. The images are built with Buildah, pushed to GHCR and deployed to the K3s cluster on demo-apps. Instana monitors the application and hands its SBOM to Concert.

Confirm the application is not deployed yet

From the bastion terminal, run the commands one by one:

You should see a message that no resources were found in the qotd namespace.

Initialise Jenkins

Open the Jenkins.. On Unlock Jenkins, enter the jenkins_password value and click Continue.

On Customize Jenkins, choose Install suggested plugins

When the plugins are installed, click Skip and continue as admin.

On Instance Configuration, leave the URL as it is and click Save and Finish.

Click Start using Jenkins.

Create the "Build and Deploy" pipeline

On Welcome to Jenkins!, click Create a job

Enter Build and Deploy as the name, select Pipeline and click OK

On Configure, select Pipeline on the left. Leave Definition as Pipeline script, paste the code below into Script and click Save.

Fill in the variables and run the pipeline

Go back to Configure → Pipeline on the left.

1. In Script, fill in GIT_URL, GHCR_USER and GHCR_PAT from notes and click Save.

Click Build Now on the left.

When it finishes you should see a green tick next to the build number.

The first build can take up to 10 minutes because it downloads all dependencies and builds several images; later builds are a bit faster thanks to caching.

Verify the deployment

All pods should be Running:

Define the qotd application in Instana

switch to the Instana tab.

Open Platforms → Kubernetes and on the Clusters tab check that the demo-apps cluster is listed. Instana has already discovered the nodes, services, namespaces and pods.

Go to Applications and choose New application perspective from + ADD in the bottom right.

Step 1: Select model: choose Kubernetes or containers and click Next

Step 2: Specify application: use Add Filter, search namespace, select Kubernetes → Namespace → Name and pick qotd. Choose All downstream services and click Next

Step 3: Provide details: enter qotd as the name, choose All calls and click Create.

Turn on Live in the top right; the Summary page fills within a couple of minutes.

Look at the Vulnerabilities tab: it is empty because Concert has not reported anything yet

Have Concert pull the data from Instana

In the Concert tab, on Administration → Integrations → Ingestion jobs, choose Run now from the three-dot menu of instana-ingestion.

Wait a few seconds and refresh; the status should be Success.

On Inventory → Evidence Store, sort Added on descending; you should see SBOMs ingested for qotd and All Services.

Find the qotd application on Inventory → Application inventory.

Enter qotd's business details

Click qotd and select version 1.0.0. The Topology tab shows the components and their relationships.

On the Overview tab, click the pencil (Edit) icon

Enter the values below, choose Yes for Add business details, fill in the business fields and click Save.

Review the SBOM, build artifacts and CVE tabs

On SBOMs, find the latest SBOM from Instana; click its name to explore the details, then Back in the top left.

On Build artifacts, see the images derived from the SBOM

CVE findings is empty for now; no scan has been uploaded yet…

Vulnerability scanning

Set up the GitHub connection and the issue-opening automation rule in Concert, then run the second Jenkins pipeline that scans the images with Trivy. Verify the results in Concert, GitHub and Instana.

Create the GitHub connection in Concert

In the Concert tab, open Administration → Integrations from the left menu. On Connections, click Create connection and pick the GitHub tile.

Fill in the fields (Description can stay empty), click Validate connection and then Create.

Create the issue-creation automation rule

Concert calculates a risk score and priority for every CVE. This rule opens a GitHub issue for every CVE in qotd with a risk score of 4 or more.

On the Automation rules tab, click Create automation rule on the right.

Fill in the fields as below. For Labels, click the refresh icon and tick bug

Create the "Vulnerability Scan" pipeline..

switch to the Jenkins tab.

Click New Item, enter Vulnerability Scan, select Pipeline and click OK.

Fill in the variables and run the scan

In Configure → Pipeline, fill in CONCERT_API_KEY, GHCR_USER and GHCR_PAT and click Save.

Click Build Now and wait for the green tick.

The images are scanned remotely, so this can take up to 5 minutes depending on network speed.

Review the vulnerabilities in Concert

In the Concert tab, open Dimensions → Vulnerability. Note the number of unique CVEs and findings across all images.

Sort Highest priority in findings descending and click a CVE with priority 1 or 2.

In Details, read the description plus the Detailed attack vector explanation and Active exploit and remediation that watsonx fills in.

On Findings, see the affected images; expand a row to read watsonx's Recommended mitigation strategy.

On Applications impacted, you should see qotd.

On the Blast radius tab in the top left, see how the vulnerability reaches packages, build artifacts, applications and environments.

Verify the GitHub issues

On Dimensions → Vulnerability, use the filter icon in the top right to select the qotd application.

Sort Highest risk score of this finding descending; you will see several CVEs with a risk score of 4 or more.

In a new tab, open the Issues tab of your fork: Concert has opened one issue per CVE finding.

See the vulnerabilities in Instana

After a couple of minutes, refresh Applications → qotd → Vulnerabilities in Instana; the vulnerabilities Trivy found are listed.

Click a CVE to see its details.

Patching the vulnerability

Fix the zlib vulnerability (CVE-2022–37434) in the qotd-engraving-service image by upgrading the Alpine packages in its Dockerfile, run the pipelines again, and verify the fix in Concert, GitHub and Instana.

Record the baseline in Concert

Select the top Resilience tab and open Dimensions → Vulnerability.

Search for CVE-2022-37434 and select qotd in the filter.

Record the baseline in Instana

On Applications → qotd → Vulnerabilities, search for CVE-2022-37434 with the search icon.

Patch the Dockerfile

In your fork on GitHub, open this folder:

concert-vulnerability-lab/qotd/qotd-engraving-service/

Open Dockerfile and edit it with the pencil icon

Add this line right after the first line (FROM node:14.16.0-alpine)

The file should end up like this:

Click Commit changes… in the top right, choose Commit directly to the main branch and click Commit changes.

Run the pipelines again

On the Jenkins dashboard, click the play button on the right of Build and Deploy and wait for it to finish (~10 min).

When it succeeds, click the play button for Vulnerability Scan (~3–5 min).

Verify the fix in Concert

On Dimensions → Vulnerability, search for CVE-2022-37434 again.

The findings should drop from 6 to 5; the qotd-engraving-service finding has been updated.

This lab ties the whole lifecycle of a vulnerability into one chain: build, observe, scan, contextual prioritisation, ticket, patch and automatic closure. Concert's value is not listing CVEs. It weighs each CVE against the application's criticality, its environment and its blast radius, puts it in front of the right person, and verifies the patch actually landed. Those are exactly the links Equifax was missing.

In production you can go one step further: trigger the scan pipeline after every build, connect your organisation's ticketing tool (Jira, ServiceNow) instead of GitHub, and add an ingestion job that pulls ticket status back.