Meteor Unit Testing With Testdouble.js

Written by Pete Corey on May 2, 2016.

It’s been several months since my first post on unit testing with Meteor 1.3, and Meteor 1.3 has finally been officially released!

With this release, Meteor’s new recommending testing solution is the all-in-one meteor test command. When you run your application with meteor test, it spins up your application server, but only loads your test files and modules explicitly imported by your tests. In theory, you can run your unit tests, integration tests, and (with an extra flag), your end-to-end tags with this tool.

Unfortunately, there is a problem. meteor test is slow! Spinning up a server is fantastic for end-to-end and integration tests, but is a massive hinderance when trying to write fast unit tests.

As I mentioned in my last post, I can easily find myself lost on Twitter by the time my server restarts.

Dependency Injection Woes

Recently, I’ve been getting feedback from people who read my Unit Testing With Meteor 1.3.

They like the idea of using Mocha directly and bypassing meteor test for faster unit test turnarounds, but find the dependency injection technique I described to be too much work in practice. But without dependency injection, how do we get around Meteor’s “magic imports”?

Error: Cannot find module 'meteor/meteor'
at Function.Module._resolveFilename (module.js:338:15)
at Function.Module._load (module.js:289:25)
at Module.require (module.js:366:17)
at require (module.js:385:17)

This error may be familiar to you if you’ve ever tried to evaluate Meteor-style modules outside the context of the Meteor build tool.

Hooking and Stubbing

Thankfully, there are ways around these tricky imports that don’t require a cumbersome dependency injection architecture.

What if we could just hook into each import and require call, and return our own objects whenever we detect an import of the "meteor/meteor" module? We could even return a Meteor object filled with stubbed or mocked methods so we can make assertions about how and when those methods are called!

The good news is that this is entirely possible within Node.js; there are actually multiple ways of accomplishing this.

In this post on the Meteor forums, Stephan Meijer discussed one technique for doing exactly this using require-hook and Sinon.js. While this technique works beautifully, I’ve recently been playing with replacing both of these modules with Testdouble.js in my testing infrastructure.

Enter Testdouble.js

Testdouble.js lets us easily replace modules with fully stubbed replacements. For example, replacing the "meteor/meteor" with an object with stubbed out methods and call functions is as simple as:

import td from "testdouble";
let Meteor = td.object(["methods", "call"]);
td.replace("meteor/meteor", { Meteor });

After this point, any imports of the "meteor/meteor" module will be given our test double, rather than the real "meteor/meteor" module.

We can even use Testdouble.js to make assertions about how our stubbed methods have been used:

...
thisMethodCallsFooWithBar();
td.verify(Meteor.call("foo", "bar"));

If thisMethodCallsForWithBar did not call the "foo" Meteor method with an argument of "bar" (e.g., Meteor.call("foo", "bar");), our td.verify assertion would fail.

Fast, Simple Unit Testing

I’ve created an example Meteor application that demonstrates these ideas. Check it out on Github. Be sure to take a look at the module under test and the tests themselves.

You can run the Mocha unit tests once:

npm test

Or you can tell Mocha to watch your project for changes and re-run your test suite on each change:

npm test -- -w

I’ve also included a short video showing how I might interact with my test runner. I make a change to a test, instantly notice that the test fails, find the problem, and notice that the test turns green as soon as I make the fix.

As you can see, the results of my test suite appear nearly instantly after every file change.

Sweet relief!

Credit Where Credit Is Due

Credit for this post goes to Stephan Meijer and the amazing work that he posted to the Meteor forums.

Stephan outlines how to mock Meteor dependencies using both require-hook and Sinon.js, and the Testdouble.js technique described above. He even found and documented a workaround for an issue related to ES6 import hoisting that was preventing Testdouble.js from properly replacing modules.

Awesome work, Stephan!

Blaze Meets Clusterize.js

Written by Pete Corey on Apr 18, 2016.

Recently, I’ve been working on a Meteor project that deals with lots of data. Most of this data is rendered in “cards” that populate a vertically scrolling list.

These cards need to be very quickly scannable, sortable, and filterable by any users of the application. To do this quickly, we need to publish all of this data to the client and let the UI handle its presentation; we can’t rely on techniques like infinite scrolling or on-the-fly subscriptions.

This situation led to an interesting problem with Blaze and an even more interesting solution leveraging Clusterize.js and a client-side cache. Let’s dig into it!

Blazingly Slow

The naive Blaze solution to presenting a bunch of UI components is to simply render each of these cards within an {{#each}} block:

{{#each data in cards}}
  {{> card data}}
{{/each}}

Unfortunately, as we start to render (and re-render) more and more cards in our list, or application slows to a crawl. After lots of profiling, debugging, and researching I came to the conclusion that Blaze simply isn’t designed to handle this much rendering and re-rendering.

Arunoda of MeteorHacks (partially) explains the issue in this article and its corresponding blog post.

Enter Clusterize.js

For our situation, a better approach was to use Clusterize.js to efficiently manage and render the massive list of cards.

Rather than dumping all of our cards into the DOM at once, Clusterize.js only renders the small portion of the cards that are currently visible. As you scroll through the list, those DOM elements are recycled and replaced with the newly visible cards. This efficient use of the DOM makes Clusterize a much more effective option when dealing with large sets of scrolling data.

Unfortunately, using Clusterize.js with Blaze wasn’t the most straight-forward process. Here’s a breakdown of how I approached the problem.

I didn’t want this Clusterize.js implementation code to permeate the rest of my front-end code, so I decided to abstract all of the Clusterize-specific complexity I was about to introduce into its own private Blaze component. This component introduced some boilerplate DOM elements required by Clusterize and an onRendered hook required to initialize the plugin:

<template name="clusterize">
  <div id="scrollArea">
    <div id="contentArea" class="clusterize-content">
    </div>
  </div>
</template>
Template.clusterize.onRendered(function() {
  // Initialize Clusterize.js here...
  this.clusterize = undefined;
});

The component was designed to accept a cursor and a template name. Each document returned by the cursor was associated with a single card that needed to be rendered with the given template. We could use the component like this:

{{> clusterize cursor=getCardDocuments
               template="card"
               options=getClusterizeOptions}}

Where getCardDocuments was a helper that returned a cursor, and getClusterizeOptions returned an options object to be passed into Clusterize.js.

Basic Rendering

The most straight forward way of using Clusterize.js is to render our cards in the DOM using a Blaze ``{:.language-javascript} tag, and then initialize the plugin:

<div id="contentArea" class="clusterize-content">
  {{#each document in cursor}}
    {{> Template.dynamic template=template data=document}}
  {{/each}}
</div>

Unfortunately, this leads to the same problems that started this whole mess. Naively rendering lots of templates in Blaze is inherently slow!

Another technique would be to manage the rendering process ourselves and give Clusterize.js a list of raw HTML strings to manage and render:

Template.cachedClusterize.onRendered(function() {
  this.autorun(() => {

    // Any time data changes, re-run
    let data = Template.currentData();
    if (!data) {
      return;
    }

    // Build the HTML for each patient card
    let template = Template[data.template];
    let rows = data.cursor.fetch().map(function(document) {
      return Blaze.toHTMLWithData(template, document);
    });

    // Update or initialize Clusterize.js
    if (this.clusterize) {
      this.clusterize.update(rows);
    }
    else {
      this.clusterize = new Clusterize(_.extend({
        rows: rows,
        scrollElem: this.$("#scrollArea")[0],
        contentElem: this.$("#contentArea")[0]
      }, data.options));
    }

  });
});

This seems like a step in the right direction, but as the cursor changes, you might notice that our component takes quite a bit of time to re-render each of the cards before passing the raw HTML off to Clusterize.js…

There has to be a faster way!

Cached Rendering

Thankfully, speeding up this implementation was fairly straight-forward. The key insight is that we don’t want to waste time re-rendering a card if we’ve already rendered it in the past. This sounds like an ideal job for a cache!

In this case, I decided to use a simple LRU cache (specifically, lru-cache) to cache my rendered templates. This cache can be set up in your application in a variety of ways depending on your current Meteor version.

I decided that a simple, but effective caching strategy would be to store each card’s rendered HTML string in the cache, indexed by the card’s _id.

This change makes the Clusterize.js render method slightly more complex:

...
let rows = data.cursor.fetch().map(function(document) {
  // Has this card already been rendered?
  let html = TemplateCache.get(document._id);
  if (html) {
    return html;
  }

  // Render the card and save it to the cache...
  html = Blaze.toHTMLWithData(template, document);
  TemplateCache.set(document._id, html);
  return html;
});
...

Now, if we ever try to re-render a card that’s already been rendered on the client, we’ll find that card in the cache and instantly return the card’s rendered HTML.

This greatly improves the speed of our Clusterize.js component as we change the set of cards we’re trying to render.

Cache Invalidation

Unfortunately, our Clusterize.js component in its current form has some major issues.

If we ever update any data on a document that should be reflected on that document’s card, we’ll never see that change. Because that card has already been rendered and cached, it’ll never be re-rendered. We’re stuck looking at old, stale data in our cards list.

To deal with this situation, we need to clear any cache entries for a card whenever its corresponding document is changed. The most straight-forward way of doing this is through an observe handler on the cursor provided to our component:

...
// Invalidate our cache whenever a doucment changes
data.cursor.observe({
  changed: function(id) {
    TemplateCache.del(id);
  }
});
...

Bam! We now have incredibly fast, dynamically updating cards in our Clusterize.js managed list!

Next Steps

What I described here is a fairly simplified version of the Clusterize.js component I finally landed on.

This version doesn’t handle “client-side joins” within your rendered cards. It also doesn’t handle changes made to documents on the server, while that document doesn’t exist in the client’s cursor. These downfalls can easily be addressed with slightly more sophisticated invalidation rules and caching schemes.

At the end of the day, Clusterize.js was a life saver. With some minor massaging, it was able to step in and replace Blaze to do some majorly impressive feats of rendering.

CollectionFS Safety Considerations

Written by Pete Corey on Apr 4, 2016.

The ability to upload files is a core piece of functionality in many web applications. Because of this, many libraries have sprung up around the topic of managing and facilitating file uploads. Arguably the most popular Meteor file upload package is CollectionFS.

CollectionFS is ubiquitous throughout the Meteor community due to its extensive functionality and its ease of use. In fact, it’s so easy to use that many developers simply drop it into their application and move on to their next feature without considering the implications that file uploading might have.

From a security and performance perspective, there are several things you should consider and make conscious choices about before adding file uploading to your application:

  • File size limiting
  • File count limiting
  • File type restrictions
  • File handling and processing

Let’s dive into each of these topics and explore why they’re so important.

File Size Limiting

The size of files being uploaded into your system quickly becomes important when you begin working with those files. Are you processing the files that are uploaded in any way? In doing that processing, are you attempting to load the entire file’s contents into memory?

Loading a massive file into memory can quickly lead to performance issues and server crashes. Node.js applications have a default maximum of 1.76GB of available memory. If a user were to upload a file that’s around 1.76GB or larger, it would lead to the server crashing and the application being completely unavailable during the restart.

Thankfully, restricting an upload’s file size is a very simple process when using CollectionFS. The following code creates a Files collection and uses the filter object to specify a maximum file size of 100MB (1e+8 bytes).

Files = new FS.Collection("files", {
  stores: [ ... ],
  filter: {
    maxSize: 1e+8
  }
});

Any files larger that 100MB will be rejected by this filter.

File Count Limiting

Along with limiting the size of files being uploaded, you should also limit the number of files uploadable by users of your system.

Imagine that you’re using CollectionFS to upload files into an S3 bucket. Left unchecked, a malicious user might upload hundreds or thousands of very large files into this bucket, drastically increasing the amount of storage space being used.

Without some kind of alerting, you may not notice this until your next AWS billing cycle where you’ll find a notably increased S3 bill!

Adding a maximum limit to the number of files in your CollectionFS stores is accomplished by added a custom insert rule to your file collection:

Files.allow({
  insert: function() {
    return Files.find().count() <= 100;
  }
});

In this example, file uploads will be rejected if there are already 100 files uploaded to your stores.

We could easily tweak this example to allow a maximum number of files per user or per any arbitrary group of users.

File Type Restrictions

When files go up, they usually come down. Any system that allows for the uploading of files usually intends for those files to be downloaded and used at some point in the future.

However, what if a user could upload any kind of file they wanted? Imagine the repercussions of a user uploading malicious scripts, executables or viruses to your applications. Those files might be downloaded and run by some other user, leading to a compromise of their system.

Most applications only work with a small set of file types. It’s good security practice to only allow files of those types to be uploaded. The best way to restrict the file types allowed into your system is to allow a set of expected file extensions:

Files = new FS.Collection("files", {
  stores: [ ... ],
  filter: {
    allow: {
      extensions: ["pdf", "doc", "docx"]
    }
  }
});

This example only allows PDFs and Word documents to be uploaded.

Filtering on file extension is considered safer than filtering on content type. The content type of an uploaded is provided by the client and can easily be spoofed.

Additionally, always whitelist expected file extensions, rather than blacklisting disallowed extensions. Blacklisting creates the possibility that a harmful file extension was forgotten, and could still be uploaded.

File Handling and Processing

When working with uploaded files, always act defensively. Never assume that the file is well-formed or that it will conform to your assumptions. Bugs in file processing algorithms have historically led to some severe vulnerabilities.

CollectionFS’s transformWrite runs in an unprotected thread of execute. This means that any uncaught exceptions that bubble up out of this method will escalate all the way to the event loop and crash the application. Once the server restarts, CollectionFS will notice that the transformation was not success and will re-attempt to transform the file, crashing the server in the process.

This kind of repeated crashing can leave your application completely inaccessible to users until the file having problems is removed from your CollectionFS store. A malicious user may intentionally create a crash loop to deny service to your application.

Final Thoughts

Working with files can be a dangerous proposition. Thankfully, CollectionFS and its associated storage drivers takes some of the danger out of our hands. In most circumstances we don’t have to worry about things like directory traversal vulnerabilities, or the possibilities of arbitrary code execution.

As we’ve seen, there are still things we need to be considerate of. If you follow these suggestions and spent time thoroughly analyzing your file upload system, you should have nothing to worry about.