Skip to content
Colin Wren
Twitter

Bi-Directional Traceability in Jira With X-Ray

— Testing, Automation, JavaScript, Bash, Software Development — 5 min read

footsteps in the sand
Photo by Marten Bjork on Unsplash

A while ago I wrote about using Notion to build a test reporting structure that allows for bi-directional traceability from User Story to Test Execution and vice-versa.

I then turned that blog into a talk — ‘Test Reporting — Why I Hate Piecharts’ and gave it at my local Ministry of Testing meet-up where I argued the case against test reports being just a number of tests run and a pass rate.

When I started building up a backlog in Jira for the app I’m currently working on (JiffyCV) I knew that I wanted to have bi-directional traceability between the user stories I was adding and the automated test suite I’d be eventually writing to check everything was working.

The decision I had to make was then — which Jira based test reporting tool to use, Zephyr or X-Ray?

I’ve used Zephyr in the past and found it to be really powerful but the custom ZQL language that it uses to query test cases was a little bit of a put off.

X-Ray on the other hand uses Jira’s JQL so you can build custom views easily anywhere you can use JQL queries.

Getting Started with X-Ray

X-Ray uses Jira tasks as a backing for the entities it uses:

  • Test — A test case to verify behaviour
  • Test Set — A grouping of tests
  • Precondition — Set up for a test case
  • Test Execution — The actual running of a test case
  • Test Plan — A plan of what to test

If you don’t already have the relevant issue types created you’ll need to create them and configure X-Ray’s mapping, if you’re using a Next-Gen project then you might find that this involves some work as you don’t start with many issue types.

Once you have the mapping in place you’ll then be able to create test cases for your user stories.

X-Ray supports three types of test case structures; Gherkin, Manual and Unstructured which can be used to define how to carry out the test and be used in automation.

I’m not a big fan of Gherkin so I chose to go with Unstructured format and link the description to the name of the test in Jest, the test runner I’m using in my project.

Making things traceable

Having floating test cases won’t give you much benefit but you can link them to your user stories and epics using Jira’s issue linking tool and defining a ‘tests’ and ‘tested by’ relationship between the test case and the user story/epic.

You can define the relationship to use for traceability in X-Ray’s settings if you want to use a custom one but the default ‘tests’/’tested by’ relationship should cover most use cases.

Once the relationship between the user story/epic and the test case is created you should then see the Coverage section of the user story populated with the test case, over time this will show the status of test executions as they are run.

In Jira Next-Gen projects there’s one new feature that I really like as someone acting as Product Owner, which is the Roadmap view.

The Roadmap view shows the epics in a Gantt chart and shows you how much of the epic’s stories have been completed.

If you create a relationship between the test case and the epic then you can also see the test coverage of all stories under the epic which brings your test reporting into this higher level and gives a really powerful insight into the quality of what you’re building.

Automating test reporting

The basic relationship setup will allow you to carry out your manual tests and provide traceability on those but the real benefit of a tool like X-Ray is that you can import test run data and automate this process to run tests across multiple environments.

This automation is something I had done previously with Zephyr, running tests against 20 environments in parallel and using Jira as a dashboard to show the status of the tests for reporting this to the team.

X-Ray offers a number of means for automating test case and test execution creation, allowing you to create in bulk via an import or to create on the fly via calls to a set of API endpoints.

If you’re running a suite of tests then I would recommend the bulk import approach as you only need to hit the API once which keeps your test run fast, if you do hit up the API as you run your tests then you’ll have the overhead of the network requests which will add time.

For my project I’m using an upload of a JUnit report from Jest generated using the jest-junit test reporter which once configured correctly allowed me to link a test case in Jest to an Acceptance Criteria in my user stories.

Creating JUnit Reports

I found with jest-junit I had some additional work to do in order to set it up how I wanted it, here’s the config I used:

1module.exports = {
2  reporters: [
3    'default',
4    [
5      'jest-junit',
6      {
7        suiteName: 'Tests',
8        classNameTemplate: '{classname}', titleTemplate: '{title}'
9      }
10    ]
11  ],
12}

By default the titleTemplate includes the classname which for my use meant I had duplicated titles which threw the test linking off.

Authenticating with X-Ray to upload test results

In order to use the X-Ray API to upload test results you need to first create an API key, this can be done by going to:

  1. Settings (cog icon in top right, not to be confused with project settings)
  2. Apps (bottom of list)
  3. API Keys (under the X-Ray heading)

Once you’ve created a client_id and client_secret you’ll then use those to create an auth token which will be used when talking to the API.

You can use the following cURL command to get an authentication token:

1token=$(curl -H "Content-Type: application/json" -X POST --data @"xray_auth.json" https://xray.cloud.xpand-it.com/api/v1/authenticate| tr -d '"')

The xray_auth.json file looks like:

1{"client_id": "[ADD CLIENT ID HERE]", "client_secret": "[ADD CLIENT SECRET HERE]"}

Once you’ve got a token you can then use it in your calls to upload your tests:

1curl -H "Content-Type: text/xml" -X POST -H "Authorization: Bearer $token"  --data @"junit.xml" https://xray.cloud.xpand-it.com/api/v1/import/execution/junit?projectKey=[PROJECT KEY]

The junit.xml file is the output from jest-junit or however you’ll be generating your JUnit format test report.

When running this setup in CI I stored the client_id and client_secret in the CI’s secret store and used echo to construct the xray_auth.json file before calling the cURL to generate the token, this allows for the credentials to be stored and used securely without committing to git.

Test Coverage Reports

Within Jira you can make use of the test coverage data to help prioritise your backlog items in order to fix technical debt & bugs and you can also use it to see the health of a particular piece of functionality as it progresses through different environments.

Ultimately how you use the test coverage will be dependent on your situation but for my app I’m using it to automate my Acceptance Criteria testing and then using the test results across a range of devices to ensure that my app is working and to highlight where additional development effort may be needed to support older or more obscure set ups.

Out of the box X-Ray offers a number of traceability reports and even allows for custom templates for the PDF test run reports, I don’t use these though as I’m of the opinion that test reports are never actually used and just go into inboxes to die.

For me the real power of a tool like X-Ray is to be able to use dashboards and other dynamic reporting tools to see the data, query it and explore the links between different tickets to uncover the real quality issues impacting the product instead of just checking that some tests have been run.