Controlling a Bitcoin Node with Elixir

Written by Pete Corey on Sep 4, 2017.

I’ve been bit by the Bitcoin bug, and I’ve been bit hard. To satiate my thirst for knowledge, I’ve been reading the fantastic Mastering Bitcoin (affiliate link) book by Andreas Antonopoulos, and diving into the brave new world of Bitcoin development.

Mastering Bitcoin does a fantastic job of outlining the technical underpinnings of Bitcoin, but I wanted to solidify my understanding with some hands-on experience.

Writing a simple Elixir application to communicate and control a Bitcoin Core full node through its JSON-RPC interface seems like a fantastic “hello world” exercise. Let’s get to it!

You’ll Need a Full Node

The first step in communicating with a Bitcoin Core full node is getting our hands on one. While publicly available nodes with wide open JSON-RPC interfaces are few and far between, it’s fairly simple to run our own Bitcoin Core node locally.

Assuming we’ve installed the bitcoind daemon on our system, we’ll need to configure it with a bitcoin.config file:


rpcuser=<username>
rpcpassword=<password>

The <username> and <password> values we define in our configuration will be used to authenticate ourselves when making requests to the Bitcoin node.

Once we’ve created our configuration file, we can spin up our full node:


bitcoind -conf=<path to bitcoin.config> -daemon

Once started, our full node daemon will begin connecting to peer nodes, downloading, and verifying blocks from the blockchain.

We can verify that everything is working as expected:


bitcoin-cli getinfo

This command should return some basic information about the node, including the node’s "version", and the number of "blocks" it’s received and verified. It may take several days to download and verify the entire blockchain, but we can keep continue on with our project in the meantime.

The Bitcoin Node’s JSON-RPC

Our Bitcoin full node implements a JSON-based RPC API which can be used to retrieve information about the Bitcoin blockchain, and to interact with the node itself.

Interestingly, the bitcoin-cli tool that we used to get information about the node leverages this JSON-RPC API. You can fetch a list of all of the available RPC commands on the node by calling bitcoin-cli help, or by browsing through the Bitcoin Wiki.

The node’s JSON-RPC accepts incoming commands through an HTTP server, which means that we can manually craft these RPC commands and bypass the bitcoin-cli tool entirely.

For example, we can run getinfo manually with curl:


curl --data-binary '{"jsonrpc":"1.0","method":"getinfo","params":[]}' \
     http://<user>:<pass>@localhost:8332/

Similarly, we can execute these commands from any programming environment with an HTTP client, like Elixir!

Setting Up Our Elixir Application

Now that we have a strategy for communicating with our Bitcoin full node, let’s start building out our Elixir application.

First, we’ll create a new Elixir project and update our mix.exs to add dependencies on poison, which we’ll need to encode and decode JSON objects, and httpoison, our go-to Elixir HTTP client.


defp deps do
  [
    {:httpoison, "~> 0.13"},
    {:poison, "~> 3.1"}
  ]
end

Now that we’ve laid out the scaffolding for our application, let’s turn our attention towards talking with our Bitcoin node.

We’ll start by gutting our HelloBitcoin module, and stubbing out a new getinfo function:


defmodule HelloBitcoin do

  def getinfo do
    raise "TODO: Implement getinfo"
  end

end

To keep things simple, we’ll interact with this module through iex -S mix. As a sanity check, let’s verify that everything is working correctly before moving on to the next section.

Calling our HelloBitcoin.getinfo stub should raise a runtime exception:


iex(1)> HelloBitcoin.getinfo
HelloBitcoin.getinfo
** (RuntimeError) TODO: Implement getinfo
    (hello_bitcoin) lib/hello_bitcoin.ex:4: HelloBitcoin.getinfo/0

Perfect. Progress through failure.

Constructing the GetInfo Command

Let’s start to flesh out our getinfo function.

To recap, our goal is to send a POST HTTP request to our Bitcoin node’s HTTP server (usually listening on http://localhost:8332), passing in a JSON object that holds the command we’re trying to execute and any required parameters.

It turns out this is incredibly easy with httpoison:


def getinfo do
  with url     <- Application.get_env(:hello_bitcoin, :bitcoin_url),
       command <- %{jsonrpc: "1.0", method: "getinfo", params: []},
       body    <- Poison.encode!(command),
       headers <- [{"Content-Type", "application/json"}] do
    HTTPoison.post!(url, body, headers)
  end
end

We start by pulling our url from the bitcoin_url key in our application’s configuration. This needs to be set in config/config.exs and should point to your local node:


config :hello_bitcoin, bitcoin_url: "http://<user>:<password>@localhost:8332"

Next, we build a map that represents our JSON-RPC command. In this case, our method is "getinfo", which requires no params. Finally, we construct the body of our request by JSON encoding our command with Poison.encode!.

Calling HelloBitcoin.getinfo should give us a successful 200 response from the Bitcoin node, along with the JSON encoded response to our getinfo command:


%HTTPoison.Response{
  body: "{\"result\":{\"version\":140200,\"protocolversion\":70015,\"walletversion\":130000,\"balance\":0.00000000,\"blocks\":482864,\"timeoffset\":-1,\"connections\":8,\"proxy\":\"\",\"difficulty\":888171856257.3206,\"testnet\":false,\"keypoololdest\":1503512537,\"keypoolsize\":100,\"paytxfee\":0.00000000,\"relayfee\":0.00001000,\"errors\":\"\"},\"error\":null,\"id\":null}\n",
  headers: [{"Content-Type", "application/json"}, {"Date", "Thu, 31 Aug 2017 21:27:02 GMT"}, {"Content-Length", "328"}],
  request_url: "http://localhost:8332",
  status_code: 200
}

Beautiful.

Let’s decode the resulting JSON in body and return the result:


HTTPoison.post!(url, body)
|> Map.get(:body)
|> Poison.decode!

Now our call to HelloBitcoin.getinfo returns the result returned by bitcoind in a more usable format:


%{"error" => nil, "id" => nil,
  "result" => %{"balance" => 0.0, "blocks" => 483001, "connections" => 8,
    "difficulty" => 888171856257.3206, "errors" => "",
    "keypoololdest" => 1503512537, "keypoolsize" => 100, "paytxfee" => 0.0,
    "protocolversion" => 70015, "proxy" => "", "relayfee" => 1.0e-5,
    "testnet" => false, "timeoffset" => -1, "version" => 140200,
    "walletversion" => 130000}}

You’ll notice that the "result", the data we actually want, is wrapped in a map containing metadata about the request itself. This metadata includes a potential error string, and the id of the request.

Let’s refactor our getinfo function to include some error handling, and to return the actual data we care about in the case of an error-free response:


with url <- Application.get_env(:hello_bitcoin, :bitcoin_url),
     command <- %{jsonrpc: "1.0", method: "getinfo", params: []},
     {:ok, body} <- Poison.encode(command),
     {:ok, response} <- HTTPoison.post(url, body),
     {:ok, metadata} <- Poison.decode(response.body),
     %{"error" => nil, "result" => result} <- metadata do
  result
else
  %{"error" => reason} -> {:error, reason}
  error -> error
end

Now our getinfo function will return an {:ok, result} tuple containing the result of our getinfo RPC call if everything goes well. In the case of an error we’ll receive an {:error, reason} tuple, explaining the failure.

Generalizing Commands

We could implement another Bitcoin RPC command, like getblockhash, in a nearly identical fashion:


def getblockhash(index) do
  with url <- Application.get_env(:hello_bitcoin, :bitcoin_url),
       command <- %{jsonrpc: "1.0", method: "getblockhash", params: [index]},
       {:ok, body} <- Poison.encode(command),
       {:ok, response} <- HTTPoison.post(url, body),
       {:ok, metadata} <- Poison.decode(response.body),
       %{"error" => nil, "result" => result} <- metadata do
    {:ok, result}
  else
    %{"error" => reason} -> {:error, reason}
    error -> error
  end
end

Calling our new getblockhash with an index of 0 gives us the hash of the Bitcoin genesis block, as we would expect.


HelloBitcoin.getblockhash(0)

{:ok, "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f"}

While it’s great that this works, you’ll notice that there’s a huge amount of code duplication going on here. Our getblockhash function is nearly identical to our getinfo function.

Let’s abstract out the common functionality into a new bitcoin_rpc helper function:


defp bitcoin_rpc(method, params \\ []) do
  with url <- Application.get_env(:hello_bitcoin, :bitcoin_url),
       command <- %{jsonrpc: "1.0", method: method, params: params},
       {:ok, body} <- Poison.encode(command),
       {:ok, response} <- HTTPoison.post(url, body),
       {:ok, metadata} <- Poison.decode(response.body),
       %{"error" => nil, "result" => result} <- metadata do
    {:ok, result}
  else
    %{"error" => reason} -> {:error, reason}
    error -> error
  end
end

Now we can redefine our getinfo and getblockhash functions in terms of this new bitcoin_rpc helper function:


def getinfo, do: bitcoin_rpc("getinfo")

def getblockhash(index), do: bitcoin_rpc("getblockhash", [index])

Our bitcoin_rpc now acts as a fully functional and complete Bitcoin RPC interface. We can easily implement any of the Bitcoin RPC commands using this helper function.

If you’re curious and want to interact with a Bitcoin node yourself, the full source for this HelloBitcoin project is available on GitHub.

Wrap Up

In hindsight, this was a long article explaining a relatively simple idea. The Bitcoin full node software exposes a JSON-RPC interface that can easily be accessed by your favorite language or stack, such as Elixir.

I’m incredibly excited about Bitcoin development, and I’m planning on spending more time diving deeper into this world in the future.

If you’re interested in the technical ideas behind Bitcoin, or are interested in Bitcoin development, I highly recommend you read Mastering Bitcoin (affiliate link).

Inject Detect is Launching Soon

Written by Pete Corey on Aug 28, 2017.

It’s been a long and winding road, but development and testing on the initial release of my latest project, Inject Detect, is done.

Finished!

I first announced Inject Detect back in March of this year. At that time, I estimated that the project would be finished and released mid-year.

At that point in time, I estimated that three to four months of development time was ample for the project I intended to build. Unfortunately, as software projects often do (especially when personal projects), scope creeped and timelines extended.

Take a look at the project timeline below:

As you can see, development took longer than expected. Work on the project largely took place in the evenings and in free blocks of time scheduled around my existing client work.

Not only that, but Inject Detect underwent three large refactors during its first iteration. Check out this GitHub graph of insertions and deletions for a visualization of my inner turmoil:

In hindsight, I should have prioritized shipping over perfecting my engineering vision.

That being said, I’m happy I took the time to explore alternative implementations of my idea. Along the way I learned a great deal about event sourcing, building distributed systems, Elixir, Phoenix, GraphQL, and React.

The development of Inject Detect is finished, but it’s not currently available to the public as of yet.

If you’re interested in getting your hands on Inject Detect as soon as possible, be sure to sign up for the Inject Detect newsletter where I’ll be first announcing the release in the coming weeks!

Advanced MongoDB Query Batching with DataLoader and Sift

Written by Pete Corey on Aug 21, 2017.

Last week we dove into the gloriously efficient world of batching GraphQL queries with DataLoader.

In all of the examples we explored, the queries being batched were incredibly simple. For the most part, our queries consisted of _id lookups followed by a post-processing step that matched each result to each _id being queried for.

This simplicity is great when working with examples, but the real world is a much more complicated, nuanced place.

Let’s dive into how we can work with more complicated MongoDB queries, and use sift.js to map those queries back to individual results in a batched set of query results.

A More Complicated Example

Instead of simply querying for patients or beds by _id, let’s set up a more complicated example.

Imagine that we’re trying to find if a patient has been in a particular set of beds in the past week. Our query might look something like this:


return Beds.find({
    patientId: patient._id,
    bedCode: { $in: bedCodes },
    createdAt: { $gte: moment().subtract(7, "days").toDate() }
});

If this query were used as a resolver within our GraphQL patient type, we would definitely need to batch this query to avoid N + 1 inefficiencies:


{
    patients {
        name
        recentlyInBed([123, 234, 345]) {
            bedCode
            createdAt
        }
    }
}

Just like last time, our new query would be executed once for every patient returned by our patients resolver.

Batching with DataLoader

Using DataLoader, we can write a loader function that will batch these queries for us. We’d just need to pass in our patient._id, bedCodes, and our createdAt date:


return loaders.recentlyInBedLoader({
    patientId: patient._id,
    bedCodes,
    createdAt: { $gte: moment().subtract(7, "days").toDate()
});

Now let’s implement the recentlyInBedLoader function:


export const recentlyInBedLoader = new DataLoader(queries => {
    return Beds.find({ $or: queries }).then(beds => {
        // TODO: ???
    });
});

Because we passed our entire MongoDB query object into our data loader, we can execute all of our batched queries simultaneously by grouping them under a single $or query operator.

But wait, how do we map the results of our batched query batch to the individual queries we passed into our loader?

We could try to manually recreate the logic of our query:


return queries.map(query => {
    return beds.filter(bed => {
        return query.patientId == bed.patientId &&
            _.includes(query.bedCodes, bed.bedCode) &&
            query.createdAt.$gte <= bed.createdAt;
    });
});

This works, but it seems difficult to manage, maintain, and test. Especially for more complex MongoDB queries. There has to be a better way!

Sift.js to the Rescue

Sift.js is a library that lets you filter in-memory arrays using MongoDB query objects. This is exactly what we need! We can rewrite our loader function using sift:


export const recentlyInBedLoader = new DataLoader(queries => {
    return Beds.find({ $or: queries }).then(beds => {
        return queries.map(query => sift(query, beds));
    });
});

Perfect!

Now we can write loaders with arbitrarily complex queries and easily map the batched results back to the individual queries sent into the loader.

Sift can actually be combined with DataLoader to write completely generic loader functions that can be used to batch and match any queries of any structure against a collection, but that’s a post for another day.