Why did Nock not record all the api requests?
6 minutes readtl;dr Nock overrides http(s).request
functions, which needs to be done before other code stores a reference to it.
We ran into an issue where Nock wasn’t recording api calls to a new service dependency we introduced into the BytesMatter codebase. The solution eventually turned out to be quite straightforward, but investigating it taught us that sometimes code subtleties mean that tools are not always as straightforward to use as the “copy-paste” tutorials suggest.
Background
We were introducing influxdb, and specifically the javascript influxdb-client for storing operational user-site time series data. Nock was going to be new to the project because we used a different approach to test our other dependency, mongodb. For that we used mongodb-memory-server. So in general our test runs look like this:
- start mongodb-memory-server
- start our BytesMatter express app in process (note this is important for being able to record)
- run tests
- teardown
The plan
We wanted to use nock in its default form of manually mocking requests, instead of the nock.back
approach. This is a matter of preference because it makes for simpler or more obvious debugging when all the test code and data is visually present in a test file.
With that in mind, we used the very cool nock.recroder.rec
function to figure out what http calls that the influxdb-client is making under the hood. This meant we had a one-off task off
- adding the recorder code
- copying the output
- removing the recorder code
At this point we can trim down the nock call to only the meaningful data we require.
See an Example to set up mock below:
TO START RECORDING:
The nock.recorder.rec
call can easily be added to your test before the api call to the service. Alternatively, nock is flexible enough it can be added directly into service code and you can record while actually using your code (via the UI for example). Keep in mind doing it this way comes with the caveat of having possible side effect by adding “test code” to the api itself.
nock.recorder.rec({
dont_print: true,
use_separator: true,
output_objects: true,
});
Then after the api call is finished, we can log all the service calls that nock has recorded
var fixtures = nock.recorder.play();
console.log('FIXTURES', fixtures);
nock.restore(); // stop recording
Example test file with the temporary recording code:
nock.recorder.rec({
dont_print: true,
use_separator: true,
output_objects: true,
});
return new Promise(async(resolve, reject) => {
api.post(`/api/alerts?property=${journeyProperty._id}`)
.set('Accept', 'application/json')
.set('x-auth-token', journeyToken)
.send({
name: 'CreateAlert',
operator: 'lt',
threshold: 1234,
filters: {
metric: 'page_load',
browser: 'chrome',
},
emailNotification: {
users: [user.id],
},
created: date,
})
.expect(201)
.then(async() => {
var fixtures = nock.recorder.play();
console.log('FIXTURES', fixtures);
nock.restore(); // stop recording
)};
Now we run the particular test(npm test
in our case ):
In this case we were expecting to see calls to localhost:8181(our express app running in process) and localhost:8086(InfluxDB)
Actual Output:
FIXTURES [
{
scope: 'http://localhost:8181',
method: 'POST',
path: '/api/alerts?property=60ae6657b9beafd1118e853a',
body: {
...
},
status: 201,
response: 'Created',
rawHeaders: [
...
],
reqheaders: undefined,
responseIsBinary: false
}
]
WHAAAT! we are only seeing calls to localhost:8081 i.e the call being made from the test to our API
that we were spinning up just for the test. It was very frustrating that nock.recoder.rec
was not showing the Influx DB calls.
Our initial thought was that perhaps the influxdb-client was not using http to makes its request and hence nock not recording it. After reading documentation we ruled this out.
Our next step was to add a very basic node-fetch into the execution path to see if that got picked up. This worked as expected - like a charm.
So at this point we knew:
- the influxdb-client definitely used HTTP requests under the hood
- we had set up nock at least correctly enough that the dummy
fetch
request we insterted was being recorded.
Time to look at some code!
From the nock README:
Nock works by overriding Node’s http.request
Ok, great. How about the influxdb-client? Digging into their code we saw in the NodeHttpTransport.ts constructor:
constructor(...) {
...
this.requestApi = http.request
...
}
and in our express server initialisation code:
const influxDBs = new InfluxDB({ url, token });
Remember our steps when running tests:
- start mongodb-memory-server
- start our BytesMatter express app in process (note this is important for being able to record)
- run tests
- teardown
So the problem: Influx DB client initialisation was before nock started recording. The influxdb-client was holding a reference to the original http.request
which it picked up during initialisation. We were inviting Nock to the party too late!
We changed the code to use a just-in-time cached getter function, instead of instantiating during server initialisation. Essentially:
let influxDBs;
const getClient = () => {
if (!influxDBs) {
client = new InfluxDB({ url, token });
}
return influxDBs;
}
...
const getChecks = () => {
const influxDB = getClient();
const checksAPI = new ChecksAPI(influxDB);
}
Now we run the test again and voila the calls to localhost:8086 are recorded. Here is a (shortened) sample of the output.
FIXTURES [
{
"scope": "http://beta:8086",
"method": "GET",
"path": "/api/v2/orgs?org=abc",
"body": "",
"status": 200,
"response": {
"links": {
"self": "/api/v2/orgs"
},
"orgs": [
{
"links": {
"buckets": "/api/v2/buckets?org=abc",
},
"id": "00000111234",
"name": "abc",
"description": "",
}
]
},
},
]
Once we had the recordings, the next part of mocking all the requests/response with nock was very straightforward. Example:
nock('http://localhost:8086')
.get('/api/v2/orgs?org=abc')
.reply(200, nockResponseOrgBytesMatter);
Here, the variable nockResponseOrgBytesMatter
is the json response that we recieved from the nock recording but manipulated to use it for tests.
With our new setup we have the following benefits: a) No network calls to Influx DB making the tests faster! b) We have a mechanism to record new calls for tests c) tests for our new dependency run in our AWS pipeline and is part of the build pipeline
Final thoughts
- This was both a frustrating and rewarding issue to work through. As is often the case, the final solution was simple but the investigation took a while
- As I mentioned above, for mongodb testing we use
mongo-memory-server
which I still think is an ideal and preferable setup. It means you’re testing with the “real thing” while still enjoying test isolation and repeatability. However, influx doesn’t seem to have an equivalent, so I am more than happy with this set up!
Learnings:
- Nock is amazing but implementations are not always straightforward. We ended up changing production code to suit a test, which is not great. I’m sure we could figure out a way to move the nock code earlier, but our solution will have very little impact, so until proven otherwise, we’ll deem this good enough. Fix the next problem instead of struggling with diminishing rate of returns looking for a perfect fix.
- It is important to be aware of the flow of code in your solution. It helps with diagnosing issues as well as impact of any changes.
Thanks for reading!