Offline GraphQL Queries with Redux Offline and Apollo

Written by Pete Corey on Jul 24, 2017.

Ironically, in our ever more connected world, demands for offline capabilities of web applications are growing. Our users (and clients) expect to use rich internet applications while online, offline, and in areas of questionable connectivity.

This can be… difficult.

Let’s dive into how we can build a reasonably powerful offline solution using React and a GraphQL data layer powered by Apollo Client. We’ll split this article into two parts. This week, we’ll discuss offline querying. Next week, we’ll tackle mutations.

Redux Persist and Redux Offline

Under the hood, Apollo Client is powered by Redux. This means that the entire Redux ecosystem of tools and libraries are available for us to use in our Apollo application.

In the world of Redux offline support, there are two major players: Redux Persist, and Redux Offline.

Redux Persist is a fantastic, but bare bones, tool designed to store and retrieve (or “rehydrate”) a redux store to and from localStorage (or any other supported storage engine).

Redux Offline expands on Redux Persist and adds additional layers of functionality and utility. Redux Offline automatically detects network disconnections and reconnections, lets you queue up actions and operations while offline, and automatically retries those actions once reconnected.

Redux Offline is the batteries-included option for offline support. 🔋

Offline Queries

Out of the box, Apollo Client works fairly well in partially connected network situations. Once a query is made by the client, the results of that query are saved to the Apollo store.

If that same query is made again with any fetchPolicy other than network-only, the results of that query will be immediately pulled from the client’s store and returned to the querying component. This means that even if our client is disconnected from the server, repeated queries will still be resolved with the most recent results available.

Unfortunately, as soon as a user closes our application, their store is lost. How can we persist the client’s Apollo store through application restarts?

Redux Offline to the rescue!

The Apollo store actually exists within our application’s Redux store (under the apollo key). By persisting the entire Redux store to localStorage, and rehydrating it every time the application is loaded, we can carry over the results of past queries through application restarts, even while disconnected from the internet!

Using Redux Offline with an Apollo Client application doesn’t come without its kinks. Let’s explore how to get these two libraries to work together.

Manually Building a Store

Normally, setting up an Apollo client is fairly simple:


export const client = new ApolloClient({
    networkInterface
});

The ApolloClient constructor would create our Apollo store (and indirectly, our Redux store) automatically for us. We’d simply drop this new client into our ApolloProvider component:


ReactDOM.render(
    <ApolloProvider client={client}>
        <App />
    </ApolloProvider>,
    document.getElementById('root')
);

When using Redux Offline, we’ll need to manually construct our Redux store to pass in our Redux Offline middleware. To start, let’s just recreate what Apollo does for us:


export const store = createStore(
    combineReducers({ apollo: client.reducer() }),
    undefined,
    applyMiddleware(client.middleware())
);

Our new store uses the reducer and middleware provided by our Apollo client, and initializes with an initial store value of undefined.

We can now pass this store into our ApolloProvider:


<ApolloProvider client={client} store={store}>
    <App />
</ApolloProvider>

Perfect. Now that we have control over the creation of our Redux store, we can wire in offline support with Redux Offline.

Basic Query Persistence

Adding Redux Offline into the mix, in its simplest form, consists of adding a new piece of middleware to our store:


import { offline } from 'redux-offline';
import config from 'redux-offline/lib/defaults';

export const store = createStore(
    ...
    compose(
        applyMiddleware(client.middleware()),
        offline(config)
    )
);

Out of the box, this offline middleware will automatically start persisting our Redux store into localStorage.

Don’t believe me?

Fire up your console and pull up this localStorage entry:


localStorage.getItem("reduxPersist:apollo");

You should be given a massive JSON blob that represents the entire current state of your Apollo application.

Awesome!

Redux Offline is now automatically saving snapshots of our Redux store to localStorage. Any time you reload your application, this state will be automatically pulled out of localStorage and rehydrated into your Redux store.

Any queries that have resolutions living in this store will return that data, even if the application is currently disconnected from the server.

Rehydration Race Conditions

Unfortunately, store rehydration isn’t instant. If our application tries to make queries while Redux Offline is rehydrating our store, Strange Things™ can happen.

If we turn on autoRehydrate logging within Redux Offline (which is an ordeal in and of itself), we’d see similar errors when we first load our application:

21 actions were fired before rehydration completed. This can be a symptom of a race condition where the rehydrate action may overwrite the previously affected state. Consider running these actions after rehydration: …

The creator of Redux Persist acknowledges this and has written a recipe for delaying the rendering of your application until rehydration has taken place. Unfortunately, his solution relies on manually calling persistStore, which Redux Offline does for us behind the scenes.

Let’s come up with another solution.

We’ll start by creating a new Redux action called REHYDRATE_STORE, and a corresponding reducer that sets a rehydrated flag in our Redux store to true:


export const REHYDRATE_STORE = 'REHYDRATE_STORE';

export default (state = false, action) => {
    switch (action.type) {
        case REHYDRATE_STORE:
            return true;
        default:
            return state;
    }
};

Now let’s add our new reducer to our store and tell Redux Offline to trigger our action when it finishes rehydrating the store:


export const store = createStore(
    combineReducers({
        rehydrate: RehydrateReducer,
        apollo: client.reducer()
    }),
    ...,
    compose(
        ...
        offline({
            ...config,
            persistCallback: () => {
                store.dispatch({ type: REHYDRATE_STORE });
            },
            persistOptions: {
                blacklist: ['rehydrate']
            }
        })
    )
);

Perfect. When Redux Offline finishes hydrating our store, it’ll trigger the persistCallback function, which dispatches our REHYDRATE_STORE action, and eventually updates the rehydrate field in our store.

Adding rehydrate to our Redux Offline blacklist ensures that that piece of our store will never be stored to, or rehydrated from localStorage.

Now that our store is accurately reflecting whether or not rehydration has happened, let’s write a component that listens for changes to our new rehydrate field and only renders its children if rehydrate is true:


class Rehydrated extends Component {
    render() {
        return (
            <div className="rehydrated">
                {this.props.rehydrated ? this.props.children : <Loader />}
            </div>
        );
    }
}

export default connect(state => {
    return {
        rehydrate: state.rehydrate
    };
})(Rehydrate);

Finally, we can wrap our <App /> component in our new <Rehydrate> component to prevent our application from rendering until rehydration has taken place:


<ApolloProvider client={client} store={store}>
    <Rehydrated>
        <App />
    </Rehydrated>
</ApolloProvider>

Whew.

Now our application will happily wait until Redux Offline has completely rehydrated our store from localStorage before continuing on to render and make any subsequent GraphQL queries or mutations.

Quirks and Notes

There are a few quirks and things to take note of when using Redux Offline with Apollo client.

First, it’s important to note that the examples in this article are using version 1.9.0-0 of the apollo-client package. Fixes were introduced to Apollo Client in version 1.9 that resolved some strange behaviors when combined with Redux Offline.

Another oddity related to this setup is that Redux Offline doesn’t seem to play nicely with the Apollo Client Devtools. Trying to use Redux Offline with the Devtools installed can sometimes lead to unexpected, and seemingly unrelated errors.

These errors can be easily avoided by not connecting to the Devtools when creating your Apollo client instance:


export const client = new ApolloClient({
    networkInterface,
    connectToDevTools: false
});

Stay Tuned

Redux Offline should give you basic support for query resolution for your Apollo-powered React application, even if your application is re-loaded while disconnected from your server.

Next week we’ll dive into handling offline mutations with Redux Offline.

Stay tuned!

Recurring Tasks in Elixir

Written by Pete Corey on Jul 17, 2017.

Running periodic, or recurring tasks is a common undertaking for any web application. The stacks I’ve used in the past have all relied heavily on external databases and job queues to accomplish this task.

Elixir is a little different.

Thanks to Elixir’s Erlang heritage and the power of OTP, we’re given the option to opt out of relying on an external database and manage our recurring tasks from entirely within our application.

The Fruit Printer

Before we implement our recurring task runner, we should have a task that we want to repeat. Let’s pretend that we want to print out a random item from a list of @fruits:


@fruits ["🍉", "🍊", "🌽", "🍒", "🍇", "🌶"] # TODO: Is corn a fruit?

def print_fruit, do: IO.puts("fruit: #{Enum.fetch!(@fruits, :random.uniform(6))}")

To keep up with our voracious appetite for fruit, we want to print a fruit emoji to the console every two seconds. This is our recurring task.

How do we accomplish this?

The first thing we need to do is build our fruit printer into a GenServer process:


defmodule HelloRecurring.FruitPrinter do
  use GenServer

  @fruits ["🍉", "🍊", "🌽", "🍒", "🍇", "🌶"] # TODO: Is corn a fruit?

  def start_link, do: GenServer.start_link(__MODULE__, [])

  def print_fruit, do: IO.puts("fruit: #{Enum.fetch!(@fruits, :rand.uniform(6))}")

end

We’ll also want to supervise our new fruit printing operation:


defmodule HelloRecurring do
  use Application

  import Supervisor.Spec

  def start(_type, _args) do
    Supervisor.start_link([worker(HelloRecurring.FruitPrinter, [])],
                          [strategy: :one_for_one, name: HelloRecurring.Supervisor])
  end

end

Here we’re simply adding a FruitPrinter as a child of our supervision tree and telling the supervisor to restart the FruitPrinter child process if it ever dies for any reason.

At this point, our FruitPrinter GenServer isn’t doing us much good. It’s running, but it’s not printing fruit. We can still manually print fruit by calling FruitPrinter.print_fruit, but this would run the print_fruit function within the current process, not the GenServer’s process.

Not good enough!

We want our FruitPrinter to automatically print its own fruit every two seconds!

Our solution comes in the form of standard process messages. Let’s wire our FruitPrinter up to print_fruit whenever it receives a :print_fruit message:


def handle_info(:print_fruit, state) do
  print_fruit()
  {:noreply, state}
end

Now we can send a :print_fruit message to our FruitPrinter process with either Process.send/3, or Process.send_after/4.

Printing Fruit Forever and Ever

Sending delayed messages with Process.send_after/4 will be the key component to implementing our recurring task.

The general idea behind building out a recurring task runner in Elixir is that the task itself should be a GenServer process that schedules sending messages to itself signaling it to carry out its task.

Putting that plan into action, once our FruitPrinter is started, we can schedule a :print_fruit message to be sent to itself in two seconds:


def init(state) do
  schedule()
  {:ok, state}
end

def schedule, do: Process.send_after(self(), :print_fruit, 2000)

It’s important to note that we need to schedule our :print_fruit message in the GenServer’s init callback, rather than the start_link callback, because start_link is called under the context of the supervising process. The init callback is called once the process is created, and self() will point to our FruitPrinter, not the supervisor.

Next, we’ll add another call to schedule() in our handle_info callback. This will ensure that every handled :print_fruit message will schedule another :print_fruit message to be sent two seconds in the future:


def handle_info(:print_fruit, state) do
  schedule()
  print_fruit()
  {:noreply, state}
end

Spinning up the application, you’ll notice a constant stream of fruit being printed at a steady rate of once every two seconds.

Delicious victory.

Even When Things Go Wrong

Astute readers may have noticed a bug in our initial print_fruit function.

We’re using :rand.uniform(6) to pick a random index out of our list of @fruits. Unfortunately, :rand.uniform/1 produces a random number between 1 and n, not 0 and n - 1, as we assumed. This means that any given call to print_fruit has a one in six chance of crashing with an out of bounds error.

Whoops!

Interestingly, this bug hasn’t affected our recurring task. If we run our application for long enough to see this error raise its head, we’ll notice that two seconds after our FruitPrinter process crashes, it’s up and running again trying to print another random fruit.

Because our FruitPrinter is being supervised, any failures that result in a crash of the process will cause the supervising process to create a new FruitPrinter in its place. This new FruitPrinter will schedule a :print_fruit message in its init callback, and will continue working as expected.

Back to the problem at hand, the proper way to implement our print_fruit function would be with Elixir’s Enum.random/1 function:


def print_fruit, do: IO.puts("fruit: #{Enum.random(@fruits)}")

That’s better. We certainly don’t want bugs in our fruit.

Final Thoughts

While this type of entirely in-application recurring process may not be a solution for every problem out there, it’s a powerful option in the Elixir environment.

The robustness given to us by the concept of supervisors and the “let it crash” mentality gives us a clear advantage over similar patterns in other languages (i.e. setTimeout in Node.js).

Before you go reaching for an external tool, I’ve found that it’s often beneficial to ask yourself, “can I do this with just Elixir?”

Detecting NoSQL Injection

Written by Pete Corey on Jul 10, 2017.

The entire premise behind my latest project, Inject Detect, is that NoSQL Injection attacks can be detected in real-time as they’re being carried out against your application.

But how?

In this article, I’ll break down the strategy I’m using for detecting NoSQL Injection in MongoDB-based web applications.

At a high level, the idea is to build up a set of expected queries an application is known to make, and to use that set to detect unexpected queries that might be to result of a NoSQL Injection attack.

Let’s dig into the details.

Expected Queries

Every web application has a finite number of queries it can be expected to make throughout the course of its life.

For example, a shopping cart application might query for single orders by _id:


Orders.findOne(orderId);

Similarly, it might query for a number of orders created in the past three days:


Orders.find({createdAt: {$gte: moment().subtract(3, "days").toDate()}});

These queries aren’t limited to fetching data. When a user “deletes” an order, the application may want to set a flag on the order in question:


Orders.update({userId: this.userId}, {$set: {deleted: true}});

Each of these individual queries can be generalized based on the shape of the query and the type of data passed into it.

For example, we expect the Orders.findOne query to always be called with a String. Similarly, we expect the Orders.find query to be passed a Date for the createdAt comparison. Lastly, the Orders.update query should always be passed a String as the userId.


Orders.findOne({_id: String});

Orders.find({createdAt: {$gte: Date}});

Orders.update({userId: String}, ...);

An application might make thousands of queries per day, but each query will match one of these three generalized query patterns.

Unexpected Queries

If our application makes a query that does not fall into this set of expected queries, we’re faced with one of two possibilities:

  1. We left a query out of our set of expected queries.
  2. Our application is vulnerable to a NoSQL Injection vulnerability.

Imagine our application makes the following query:


Orders.findOne({_id: { $gte: "" }});

A query of this pattern (Orders.findOne({_id: {$gte: String}})) doesn’t exist in our set of expected queries. This means that this is either an expected query that we missed, or our application is being exploited.

It’s unlikely that our application is trying to find a single Order who’s _id is greater than or equal to an empty string.

In this case, it’s far more likely that someone is exploiting our Orders.findOne({_id: String}) query and passing in an orderId containing a MongoDB query operator ({$gte: ""}) rather than an expected String.

We’ve detected NoSL Injection!

By watching for queries that fall outside our set of expected queries, we can detect NoSQL Injection as it happens.

Similar Expected Queries

Basing our NoSQL Injection detection strategy around expected and unexpected queries has an added bonus.

Because we have a set of all expected queries for a given application, unexpected queries that are the result of NoSQL Injection attacks can often be linked back to the query being exploited.

To illustrate, in our previous example we detected an unexpected query against our application:


Orders.findOne({_id: {$gte: ""}});

Inspecting our set of expected queries, it’s obvious that the most similar expected query is the Orders.findOne query:


Orders.findOne({_id: String});

As the application owner, we know that we need to enforce more stringent checks on the type of the provided orderId.

Based on this information, an application owner or developer can deduce which specific query is being exploited within their application and respond quickly with a fix.

Final Thoughts

Progress on Inject Detect continues to move along at a steady pace. Once finished, Inject Detect will automatically apply this type of real-time NoSQL Injection detection to every query made by your Meteor application.

If you’re interested in learning more, be sure to sign up for the Inject Detect newsletter.