I know what you’re thinking. “Why Mocha? Why not use AVA? Why not find other alternatives?”

Yes, there are considerations taken for those questions, I’d rather cover that in another post.

This time, let’s assume we’re using the following tech and tools:

1. API written in Node.js/Typescript
2. Mocha
3. Circle CI
4. nyc test coverage tool
5. And a long set of tests, I mean really long.

Setting up Parallelism in Circle CI

This is not the same --parallel flag option in Mocha. Parallelism in Circle CI allows you to split your test files and run them in separate executors running at the same time therefore reducing test time. Here’s what it looks like:

In the image, we can see that there are 8 executors running in parallel. Each executor runs a different set of test files depending on how they were split across executors.

To enable this, we simply set parallelism attribute of a job in our .circleci/config.yml to the number of executors we want. In the case depicted in the picture above, parallelism is set to 8. We also have to use some commands using circleci cli to split our test files:

• circleci tests glob
• circleci tests split

Let’s put this into action, shall we? Here’s a simple .circleci/config.yml file that utilizes parallelism.

version: 2.1

jobs:
  checkout_source:
    docker:
      - image: circleci/node:14.15.1-stretch
    steps:
      - checkout
      - persist_to_workspace:
          root: .
          paths:
            - .
  install_dependencies:
    docker:
      - image: circleci/node:14.15.1-stretch
    steps:
      - attach_workspace:
          at: .
      - restore_cache:
          keys:
            - dependencies-{{ checksum "package-lock.json" }}
      - run:
          name: Install dependencies
          command: npm install
      - save_cache:
          key: dependencies-{{ checksum "package-lock.json" }}
          paths:
            - node_modules
      - persist_to_workspace:
          root: .
          paths:
            - node_modules/*
  test:
    docker:
      - image: circleci/node:14.15.1-stretch
    parallelism: 8
    steps:
      - attach_workspace:
          at: .
      - run:
          name: Run tests
          command: |
            circleci tests glob 'test/**/*.spec.ts' |
            circleci tests split |
            xargs npx mocha
workflows:
  build_workflow:
    jobs:
      - checkout_source
      - install_dependencies:
          requires:
            - checkout_source
      - test:
          requires:
            - install_dependencies

What the command circleci tests glob 'test/**/*.spec.ts' does is that it lists all the files that matches the glob pattern 'test/**/*.spec.ts'. It’s pretty much like what the command ls does when you set a glob pattern as argument. The command circleci tests split then takes this list of files and splits them in such a way that each executor will have ideally well-distributed test files.

There are 3 splitting techniques that Circle CI is using and you can choose which one to use:

1. Default, files are split alphabetically by their filenames.
2. circleci tests split --split-by=filesize – Files are split by their sizes.
3. circleci tests split --split-by=timings – Files are split by test timings. When you use this option, you’ll need to store your timing data in artifacts.

For simplicity, in this example, we’re just splitting files using the default technique.

Congrats! You now have parallelism enabled in your Circle CI tests. Time to celebrate? Not quite. When you’re using a test coverage tool, you stumble upon a slight issue.

Setting up nyc test coverage while parallelism is enabled

Since we are splitting the test files and are distributing them to a number of executors, each executor will only be running a partial set of your whole test files. That means your test coverage, for each executor, will plummet down and coverage checking will consequently fail.

Here’s my approach to work around this thing:

1. Run nyc in all tests for each executor but setting check-coverage to false. We don’t want to check coverage levels right now, we only need coverage data.
2. For each executor, store coverage data to an arbitrary (with some level of file organization) directory.
3. Merge coverage data from different directories generated by each executor.
4. And run coverage check against the merged coverage data.

Again, to the config files!

We’re going to add nyc command and set the right --temp-dir flag to control where coverage data are stored. In our .circleci/.config.yml, our test command will now become:

circleci tests glob 'test/**/*.spec.ts' |
circleci tests split |
xargs npx nyc --temp-dir=./.nyc_output_temp/$CIRCLE_JOB-$CIRCLE_NODE_INDEX/nyc_output mocha

What this does is it instructs nyc to store coverage data in ./.nyc_output_temp/$CIRCLE_JOB-$CIRCLE_NODE_INDEX/nyc_output. You can set your own temporary directory, but it should be unique per executor to avoid overlaps/conflicts in merging the generated test coverage results. In here, I used the following built-in environment variables in Circle CI:

1. $CIRCLE_JOB – Name of the job. In this case, it’s test.
2. $CIRCLE_NODE_INDEX – Each executor for a specific job has it’s own index.

Now that we have test coverage data stored in the temporary directory, we just need to add a step in our job to make these files available to jobs downstream in the workflow. The following step tells CircleCI to persist the files in .nyc_output_temp/* directory, so that these files are available in our next job where we merge coverage test data and do the coverage check.

- persist_to_workspace:
		root: .
    paths:
      - .nyc_output_temp/*

Assuming all tests passed, we can now merge the coverage data and do the coverage check. To merge the coverage data, we can just run the following code:

'use strict';

const { spawnSync } = require('child_process');
const path = require('path');
const glob = require('glob');
const makeDir = require('make-dir');
const rimraf = require('rimraf');

process.chdir(__dirname);
rimraf.sync('.nyc_output');
makeDir.sync('.nyc_output');

// Merge coverage data from each executor.
glob.sync('.nyc_output_temp/*/nyc_output').forEach((nycOutput) => {
  console.log(`Merging: ${nycOutput}`);
  const cwd = path.dirname(nycOutput);
  const { status, stderr } = spawnSync(
    'npx nyc',
    [
      'merge',
      'nyc_output',
      path.join(__dirname, '.nyc_output', path.basename(cwd) + '.json'),
    ],
    {
      encoding: 'utf8',
      shell: true,
      cwd,
    },
  );

  if (status !== 0) {
    process.exit(status);
  }
});

The code actually runs nyc merge command to merge all coverage data from each directory generated by each executor. Once we have the merged coverage data, we can now run the coverage check.

nyc report --reporter=lcov --reporter=text-summary --check-coverage

This command runs a coverage check under .nyc_output where all coverage data were merged when we ran the code above. This methodology of merging test coverage data is not limited to Circle CI tests only. You can also use this when you need to run different tests separately and want to run test coverage check after all different tests are ran.

Conclusion

That’s it! You can now enable parallelism in Circle CI without removing the ability to check test coverage. For sure, there are a few other ways to solve the same problem, but I have found this to be more robust and straightforward.

Here’s a public GitHub Gist were I put all the necessary configuration files: https://gist.github.com/ccpacillos/3778615ec84e3a494111a29430996829