Migrating from CucumberJS 2.x to 3.x

On the 8th of August the CucumberJS team (read Charlie Rudolph) released version 3.0.0, so why did it take me so long to write a migration post? Well the answer is simple. Release 3 didn’t had that many changes but the changes that were there were pretty breaking (at least for a lot of users including me).

Example project

As an example project I will refer to this boilerplate from my previous post called from compiler to boilerplate. That boilerplate holds a default setup with

Protractor
CucumberJS 3
TypeScript
and since 23rd of September also:
- a reporting plugin for protractor (protractor-multiple-cucumber-html-reporter-plugin)
- rerun flaky tests with protractor-flake
- and much more

Why so long?

First of all I was taking a vacation and they released it during my first week. When I tried to upgrade I saw that protractor-cucumber-framework (I’m only using CucumberJS in combination with Protractor) was not compatible with version 3.

This brings me to the second reason and it had to do with the breaking change CucumberJS introduced by removing the registerHandler. protractor-cucumber-framework relied heavily on it so it costed Darrin Holst a week to fix this (I’m still grateful to him for this).

The third reason had to do with the breaking change CucumberJS introduced by removing the registerListener. The registerListener was used by a lot of people to download the JSON-report output after all features were done. Personally I used this method a lot to manipulate the JSON-report output before creating a beautiful report with multiple-cucumber-html-reporter. But again Darrin Holst saved me with a nice feature in protractor-cucumber-framework he already built into version 3.1.2. I used that feature to create an easy way of generating reports with Protractor and CucumberJS by using a self-created plugin called protractor-multiple-cucumber-html-reporter-plugin which I’ll be explaining later.

The biggest changes

I think you can pick up the biggest changes by reading the CHANGELOG.md of CucumberJS but I’ll discuss them below.

Removed the pretty formatter

The pretty formatter has been removed. All errors are now reported in a pretty format instead. The progressformatter is now the default. If you migrate and forget to change the formatter you will get the following error:

Unhandled rejection Error: Cannot find module '/Users/wswebcreation/protractor-cucumber-typescript-boilerplate/pretty'

You can fix this by using one of the other formatters.

Removed the registerHandler

The registerHandler has been removed. This feature gave you the possibility to hook onto an event within the Cucumber lifecycle with:

BeforeFeatures / -Feature / -Scenario / -Step
FeaturesResult / ScenarioResult / StepResult
AfterStep / -Scenario / -Feature / -Features

Before- / -Result / AfterFeatures

The Before- / -Result / AfterFeatures have been replaced by BeforeAll / AfterAll. The difference between them is that the Before- / -Result / AfterFeatures will be passed the associated object as the first argument. TheBeforeAll / AfterAlldon’t have this (anymore). Take for example the BeforeFeatures, you had the complete features object like this

[
  {
    "description": "The description of feature two",
    "keyword": "Feature",
    "line": 1,
    "name": "The name of feature one",
    "tags": [],
    "uri": "/Users/wswebcreation/protractor-cucumber-typescript-boilerplate/e2e-tests/features/feature.one.feature",
    "scenarios": [
      [
        "Object"
      ]
    ]
  },
  {
    "description": "The description of feature two",
    "keyword": "Feature",
    "line": 2,
    "name": "The name of feature two",
    "tags": [],
    "uri": "/Users/wswebcreation/protractor-cucumber-typescript-boilerplate/e2e-tests/features/feature.two.feature",
    "scenarios": [
      [
        "Object"
      ]
    ]
  }
]

There is no replacement for this. BeforeAll / AfterAll should only be used for setup / teardown that needs to be done before or after all scenarios.

The rest of the eventHandlers

The rest of the eventHandlers are also removed. The difference with the above described eventHandlers was only that they would return a level deeper of the associated object of the feature / scenario / step.

Minimized Hooks (scenario) result

Hooks are used for setup / teardown the environment or do other magical stuff before and after each scenario. The first argument was alway a ScenarioResult like for example the below result for the After hook:

{
  "duration": 2,
  "failureException": null,
  "scenario": {
    "feature": {
      "description": "  The scenario's have scripted pending steps\n  So the report should hold pending steps",
      "keyword": "Feature",
      "line": 1,
      "name": "Pending scenario's specified V2",
      "tags": [],
      "uri": "/Users/wswebcreation/protractor-cucumber-typescript-boilerplate/e2e-tests/features/pending.feature",
      "scenarios": ["Object"]
    },
    "keyword": "Scenario",
    "lines": [
      5
    ],
    "name": "Johnny wants to see pending scenario\"s",
    "tags": [],
    "uri": "/Users/wswebcreation/protractor-cucumber-typescript-boilerplate/e2e-tests/features/pending.feature",
    "line": 5,
    "description": "undefined",
    "steps": [
      ["Object"],
      ["Object"],
      ["Object"]
    ]
  },
  "status": "pending",
  "stepResults": [
    {
      "attachments": [],
      "duration": 2,
      "step": ["Object"],
      "stepDefinition": ["Object"],
      "status": "passed"
    },
    {
      "attachments": [],
      "duration": 0,
      "step": ["Object"],
      "stepDefinition": ["Object"],
      "status": "pending"
    },
    {
      "step": ["Object"],
      "stepDefinition": ["Object"],
      "status": "skipped"
    },
    {
      "step": ["Object"],
      "stepDefinition": ["Object"],
      "status": "skipped"
    }
  ]
}

With version 3 you will only get this back from the Before / After hook

// The Before result
{
  "sourceLocation": {
    "uri": "e2e-tests/features/pending.feature",
    "line": 5
  }
}

// The After result
{
  "sourceLocation": {
    "uri": "e2e-tests/features/pending.feature",
    "line": 5
  },
  "result": {
    "duration": 3,
    "status": "pending"
  }
}

This means that you now need to be very resourceful to find a solution if you used more information in past.

For example, I had an After hook which saved a screenshot when a scenario failed. I saved the screenshot in the JSON-report output and if needed I saved the screenshot to a folder. The name of the screenshot was based on the name of the scenario. Because now the name of the scenario isn’t given back I needed to have a link to the running scenario that could be used to link the screenshot to a scenario / feature. The only reference is the URI of the feature file which I can use as a link between the screenshot and the feature / scenario. What I now do is that I abstract the name of the feature from the URI with a regular expression and add a unique value to it so it will not be overwritten when more than 1 scenario fails in 1 feature file.

// Because the scenario name isn't available in CucumberJS 3 I now use the format
// `${Date.now()}.${featureName}.${browserName}.png`
const featureName = testCase.sourceLocation.uri.match(/([^\/]+)(?=\.\w+$)/)[0]
    .replace(/[^a-zA-Z0-9\s]/g, '').replace(/\s/g, '-')
    .toLowerCase().substr(0, 100);
const fileName = `${Date.now()}.${featureName}.${browser.browserName}.png`;

Removed the registerListener

Since 2015 I’ve been using CucumberJS and also started using the registerListener to save the JSON-report output to a file for reporting. Between 2015 and 2017 I’ve enriched the JSON-report output with all kinds of extra data, splitting the features into separate files, and so on, but I never bothered about finding a new / better way to save and manipulate the JSON-report output. Until on the 8th of August when I migrated to version 3.

When you are using CucumberJS as a standalone framework you can provide a JSON formatter with a unique filename per instance through the command line. This was not possible with Protractor + Cucumber + protractor-cucumber-framework, at least, that was what I thought.

There is a possibility to provide the format in the cucumberOpts like this

format: 'json:.tmp/results.json'

But when you work with multiple browsers (Chrome, Firefox, Safari,..) divided over multiple instances and shardTestFiles: true (all features can be divided over multiple browser instances, instead of running them all in 1 browser) the output would overwrite itself resulting in not seeing all the reports. Again, that was what I thought.

When I looked into the protractor-cucumber-framework source code, see here, I saw that Darrin Holst already had added a new feature in version 3.1.2 that covered that part. What it basically does is that it saves a JSON-report output for each NodeJS instance that has been run with a name that consists out of:

the provided name (in our example results)
the NodeJS process identifier (PID)

so you would get results.1234.json. This will make the files unique enough so they will not overwrite themselves.

My first challenge, and I think for more people, has been resolved. The second one, manipulate / extending the JSON-report files before generating a HTML report was a bigger challenge because I could not manipulate / extend the JSON-report object before it was saved. When I contacted Charlie Rudolph about this he advices to put this in a post process. That made me thinking.

After some digging I found out that the solution itself was simple. The connection between the running instance and the report was the PID. I used that logic and wrote a Protractor plugin called protractor-multiple-cucumber-html-reporter-plugin to easily integrate the JSON-report output to a beautiful report. I hope you like the solution ;-).

Rewrite of the custom formatter and the custom snippet syntax

I’ve never used the custom formatter and custom snippet syntax APIs before, but based on the change log there has been a major rewrite of them in support of the event protocol. When I also look at the removal of the eventHandler I would advise you to thoroughly look into the updated documentation, see the above links.

Conclusion

As said in the beginning, there were not many changes, but the changes that were made were pretty breaking a CucumberJS based framework. I understand all the choices the CucumberJS team made, despite the fact that I do not always agree, but that’s more based on a habit ;-). The changes will encourage you to be more resourceful, check all the dependencies you are using and appreciate all the nice work contributors in the open source community already did for you.

So I hope people won’t be negative towards the (breaking) changes and embrace the challenges and changes. I would like to thank Charlie Rudolph and Darrin Holst for having the patience and the time to help us all out and not only making stuff better, but also keep encouraging us to be more resourceful!

Credits:

Business vector created by Dooder – Freepik.com

wswebcreation

Find me on:

Categories

Archives