tl;dr Nock overrides http(s).request functions, which needs to be done before other code stores a reference to it.

We ran into an issue where Nock wasn’t recording api calls to a new service dependency we introduced into the BytesMatter codebase. The solution eventually turned out to be quite straightforward, but investigating it taught us that sometimes code subtleties mean that tools are not always as straightforward to use as the “copy-paste” tutorials suggest.

Background

We were introducing influxdb, and specifically the javascript influxdb-client for storing operational user-site time series data. Nock was going to be new to the project because we used a different approach to test our other dependency, mongodb. For that we used mongodb-memory-server. So in general our test runs look like this:

1. start mongodb-memory-server
2. start our BytesMatter express app in process (note this is important for being able to record)
3. run tests
4. teardown

The plan

We wanted to use nock in its default form of manually mocking requests, instead of the nock.back approach. This is a matter of preference because it makes for simpler or more obvious debugging when all the test code and data is visually present in a test file.

With that in mind, we used the very cool nock.recroder.rec function to figure out what http calls that the influxdb-client is making under the hood. This meant we had a one-off task off

1. adding the recorder code
2. copying the output
3. removing the recorder code

At this point we can trim down the nock call to only the meaningful data we require.

See an Example to set up mock below:

TO START RECORDING: The nock.recorder.rec call can easily be added to your test before the api call to the service. Alternatively, nock is flexible enough it can be added directly into service code and you can record while actually using your code (via the UI for example). Keep in mind doing it this way comes with the caveat of having possible side effect by adding “test code” to the api itself.

nock.recorder.rec({
dont_print: true,
use_separator: true,
output_objects: true,
});

Then after the api call is finished, we can log all the service calls that nock has recorded

var fixtures = nock.recorder.play();
console.log('FIXTURES', fixtures);
nock.restore(); // stop recording

Example test file with the temporary recording code:

nock.recorder.rec({
dont_print: true,
use_separator: true,
output_objects: true,
});

return new Promise(async(resolve, reject) => {
    api.post(`/api/alerts?property=${journeyProperty._id}`)
        .set('Accept', 'application/json')
        .set('x-auth-token', journeyToken)
        .send({
            name: 'CreateAlert',
            operator: 'lt',
            threshold: 1234,
            filters: {
                metric: 'page_load',
                browser: 'chrome',
            },
            emailNotification: {
                users: [user.id],
            },
            created: date,
            })
        .expect(201)
        .then(async() => {
            var fixtures = nock.recorder.play();
            console.log('FIXTURES', fixtures);
            nock.restore(); // stop recording
        )};

Now we run the particular test(npm test in our case ): In this case we were expecting to see calls to localhost:8181(our express app running in process) and localhost:8086(InfluxDB)

Actual Output:

FIXTURES [
  {
    scope: 'http://localhost:8181',
    method: 'POST',
    path: '/api/alerts?property=60ae6657b9beafd1118e853a',
    body: {
      ...
    },
    status: 201,
    response: 'Created',
    rawHeaders: [
      ...
    ],
    reqheaders: undefined,
    responseIsBinary: false
  }
]

WHAAAT! we are only seeing calls to localhost:8081 i.e the call being made from the test to our API that we were spinning up just for the test. It was very frustrating that nock.recoder.rec was not showing the Influx DB calls.

Our initial thought was that perhaps the influxdb-client was not using http to makes its request and hence nock not recording it. After reading documentation we ruled this out.

Our next step was to add a very basic node-fetch into the execution path to see if that got picked up. This worked as expected - like a charm.

So at this point we knew:

• the influxdb-client definitely used HTTP requests under the hood
• we had set up nock at least correctly enough that the dummy fetch request we insterted was being recorded.

Time to look at some code!

From the nock README:

Nock works by overriding Node’s http.request

Ok, great. How about the influxdb-client? Digging into their code we saw in the NodeHttpTransport.ts constructor:

constructor(...) {
  ...
  this.requestApi = http.request
  ...
}

and in our express server initialisation code:

const influxDBs = new InfluxDB({ url, token });

Remember our steps when running tests:

1. start mongodb-memory-server
2. start our BytesMatter express app in process (note this is important for being able to record)
3. run tests
4. teardown

So the problem: Influx DB client initialisation was before nock started recording. The influxdb-client was holding a reference to the original http.request which it picked up during initialisation. We were inviting Nock to the party too late!

We changed the code to use a just-in-time cached getter function, instead of instantiating during server initialisation. Essentially:

let influxDBs;
const getClient = () => {
  if (!influxDBs) {
    client = new InfluxDB({ url, token });
  }

  return influxDBs;
}

...
const getChecks = () => {
  const influxDB = getClient();
  const checksAPI = new ChecksAPI(influxDB);
}

Now we run the test again and voila the calls to localhost:8086 are recorded. Here is a (shortened) sample of the output.

FIXTURES [
  {
    "scope": "http://beta:8086",
    "method": "GET",
    "path": "/api/v2/orgs?org=abc",
    "body": "",
    "status": 200,
    "response": {
      "links": {
        "self": "/api/v2/orgs"
      },
      "orgs": [
        {
          "links": {
            "buckets": "/api/v2/buckets?org=abc",
          },
          "id": "00000111234",
          "name": "abc",
          "description": "",
        }
      ]
    },
  },
]

Once we had the recordings, the next part of mocking all the requests/response with nock was very straightforward. Example:

    nock('http://localhost:8086')
      .get('/api/v2/orgs?org=abc')
      .reply(200, nockResponseOrgBytesMatter);

Here, the variable nockResponseOrgBytesMatter is the json response that we recieved from the nock recording but manipulated to use it for tests.

With our new setup we have the following benefits: a) No network calls to Influx DB making the tests faster! b) We have a mechanism to record new calls for tests c) tests for our new dependency run in our AWS pipeline and is part of the build pipeline

Final thoughts

• This was both a frustrating and rewarding issue to work through. As is often the case, the final solution was simple but the investigation took a while
• As I mentioned above, for mongodb testing we use mongo-memory-server which I still think is an ideal and preferable setup. It means you’re testing with the “real thing” while still enjoying test isolation and repeatability. However, influx doesn’t seem to have an equivalent, so I am more than happy with this set up!

Learnings:

1. Nock is amazing but implementations are not always straightforward. We ended up changing production code to suit a test, which is not great. I’m sure we could figure out a way to move the nock code earlier, but our solution will have very little impact, so until proven otherwise, we’ll deem this good enough. Fix the next problem instead of struggling with diminishing rate of returns looking for a perfect fix.
2. It is important to be aware of the flow of code in your solution. It helps with diagnosing issues as well as impact of any changes.

Thanks for reading!