Build Your Own Code Poster with Elixir

On Feb 13, 2017 by Pete Corey

I recently finished up a long engagement with one of my clients, AdmitHub. To celebrate the work we had done together, I wanted to give them a going away gift.

I liked the idea of giving them a Commits.io poster, but I wasn’t comfortable handing out access to my client’s private repository to a third party.

Not to be deterred, I decided to use this opportunity to practice Elixir and build my own poster generator!

In the process of making the AdmitHub poster, I also made an Elixir poster to celebrate my ever increasing love for the language.

High-level Strategy

If we approach the problem of generating a code poster from a high level, we can break it into three distinct parts.

First, we’ll want to load a source image and a blob of source code. These pieces of data are the raw building blocks for our final poster.

Next, we’ll need to merge this data together. Our ideal outcome is an SVG filled with <text> elements. Each <text> element will contain one or more characters, or code points, from our source code blob, colored to match the corresponding pixel in our source image.

Once we’ve merged this data together in memory, we need to generate our final SVG and save it to disk.

With our SVG in hand, we can use a tool like Adobe Illustrator or Inkscape to render it into a more printer friendly format and then send it off to be printed!

Loading Our Source Data

Our application will make use of the Imagineer elixir library to load our source image. As an example, we’ll be using a beautiful version of the Elixir logo as designed by Bruna Kochi.

Before loading our image into our application, we need to consider a few things.

We’ll be using the Source Code Pro font to render the code in our poster. It’s important to realize that the characters in Source Code Pro, while monospaced, aren’t perfectly square. It turns out that the ratio of each character’s width to its height is 0.6.

This means that if we’re inserting a character of code for every pixel in our source image, and we want our final poster to maintain the aspect ratio of the original logo, we’ll need to scale the width of our source image by a factor of 1.667.

The total width and height of our source image also effects the outcome of our poster. More pixels means more (and smaller) code characters. A width and height of 389px by 300px gives good results.

Check out the scaled and resized source image to the right.

Once we’ve scaled and resized our source image, loading it into our Elixir application is a breeze:


{:ok, image} = Imagineer.load(image_path)

Within the resulting image struct, we’ll find a pixels field that holds all of the raw pixel data for the image in a two-dimensional array of tuples. Each tuple holds the RGB value for that specific pixel.


Now that we’ve loaded our source image, we’ll need to load the code we want to render onto our poster.

Rather than letting a library do the heavy lifting for us, we’ll take a more hands-on approach here.

We’ll use File.read! to load the file at code_path into memory, strip out any excess whitespace with the join_code helper function, and finally split the resulting string into a list of individual code points:


code = code_path
|> File.read!
|> join_code
|> String.codepoints

The join_code function simply replaces all newlines with leading and trailing spaces with a single space character, and replaces all non-space whitespace characters (tabs, etc…) with spaces:


def join_code(code) do
  code
  |> String.trim
  |> String.replace(~r/\s*\n+\s*/, " ")
  |> String.replace(~r/\s/," ")
end

And with that, we’ve successfully loaded both our source image, and our source code!

Merging Pixels and Code Points

The real meat of our application is in merging these two disparate sets of data.

Our goal is to build an in-memory data structure that places each code point from our source code blob in its correct position on the poster, and colors it according to the pixel corresponding to that same position.

Imagineer delivers pixel data in the form of a list of rows of pixels. Each pixel is a tuple of RGB values. The structure of this data influences how we’ll structure our solution.

Ultimately, we’ll map over each row of pixels and reduce each row down to a list of tuples representing each <text> element in our final poster.


The reduction of each row is probably the most interesting part of our application. When reducing an individual pixel from a row of pixels, there are three possible scenarios.

This might be the first pixel we’ve encountered in a row. In that case, create a new <text> element:


def merge_pixel_into_row(fill, character, x, y, []) do
  [{:text, %{x: x, y: y, fill: fill}, character}]
end

The current pixel might match the fill color of the previous pixel. In that case, append the current character to the body of the previous <text> element:


def merge_pixel_into_row(fill, character, _, _, 
                         [{:text, element = %{fill: fill}, text} | tail]) do
  [{:text, element, text <> character} | tail]
end

Notice that we’re pattern matching the current fill color to the fill color of the previously seen text element. Awesome!

Lastly, the current pixel might be a different color. In that case, create and append a new <text> element to the head of the list:


def merge_pixel_into_row(fill, character, x, y, pixels) do
  [{:text, %{x: x, y: y, fill: fill}, character} | pixels]
end

After flattening the results of our map/reduce, we finally have our resulting list of correctly positioned and colored <text> elements!

Building Our SVG

Now that we’ve built up the representations of our <text> elements, we can finally construct our SVG.

We’ll use the xml_builder Elixir module to generate our final SVG. Generating an SVG with xml_builder is as simple as passing an :svg tuple populated with our text elements and any needed attributes into XmlBuilder.generate:


{:svg,
 %{
   viewBox: "0 0 #{width*ratio} #{height}",
   xmlns: "http://www.w3.org/2000/svg",
   style: "font-family: 'Source Code Pro'; font-size: 1; font-weight: 900;",
   width: final_width,
   height: final_height,
   "xml:space": "preserve"
 },
 text_elements}
|> XmlBuilder.generate

Notice that we’re defining a viewBox based on the source image’s width, height, and our font’s ratio. Similarly, we’re setting the final width and height based on the provided final_width and final_height.

The "xml:space": "preserve" attribute is important. Without preserving whitespace, text elements with leading with space characters will be trimmed. This results in strangely chopped up words in the final poster and is a difficult issue to track down (trust me).

Lastly, we pass in the list of :text tuples we’ve generated and stored in the text_elements list.

The Final Poster

After the final SVG is generated and saved to disk, it can be loaded into Illustrator or Inkscape and rendered to a PNG for printing.

A 10500 by 13500 pixel image at 300 dpi looks fantastic when printed onto a 20x30 inch poster.

Overall, I’m very happy with the outcome of the final poster. It beautifully commemorates the work my client and I accomplished together over the past two years, and will hang proudly in my office and theirs.

The Elixir poster turned out very nicely as well. If you’re interested or want to print your own poster, you can download the full resolution output image here.


The application takes approximately one minute to generate a 20x30 inch poster, depending on the color variation per row. This performance can almost definitely be improved and may be the subject of a future post.

Was Elixir the tool for this project? Probably not. Was it still a fun project and a good learning experience? Absolutely.

If you’re interested in seeing more of the code, be sure to check out the entire project on Github.

Playing the Game of Life with Elixir Processes

On Feb 6, 2017 by Pete Corey

I’ve always been fascinated with Conway’s Game of Life, and I’ve always wondered what the Game of Life is like from an individual cell’s perspective.

The constant threat of dying of loneliness, three person mating rituals, and the potential to achieve immortality! What a strange and beautiful world…

In this article, we’ll use Elixir processes, supervision trees, and the Registry to implement Conway’s Game of Life from this interesting perspective.

The Game of Life

Most basic implementations of the Game of Life represent the universe as a large two-dimensional grid. Each spot in the grid is either “alive”, or “dead”.

Once every “tick”, the simulation will loop over each cell in the universe. If the cell is dead and has exactly three living neighbors, it will be born into the next generation. If the cell is living and has two or three neighbors, it will live on during the next generation. Otherwise, the cell will die or remain dead.

When let loose on an initial configuration of living cells, the continuous application of these rules can create an incredibly complex, seemingly organic system.

The Architecture

Rather than following the standard approach of using a finite two-dimensional array to represent our universe, let’s flip this problem upside down.

Let’s represent each living cell as an active Elixir process living somewhere on on our server. Each cell process will hold it’s location as an {x, y} tuple as its state.

We’ll need some way of finding a cell’s neighbors, which means we’ll need to be able to look up a cell based on its given {x, y} location. This sounds like a classic process discovery problem and it gives us an excellent excuse to try out Elixir’s new Registry module.

Our cells will be fairly independent; they’ll manage their own position, and determine when it’s time to die or give birth to another cell. However, we’ll need some additional outside process to tell each cell in the universe when to “tick”. We’ll call this controller process the “Universe”.

“Life calls the tune, we dance.”

Given those basic components, we can draw up a basic dependency tree of our application. We’ll need a top level supervisor managing our universe and cell registry, along with a supervisor to dynamically manage each cell process.

The Supervisors

Before we dive into the meat of our application, let’s take a quick look at how we’ll implement our application’s supervisors.

Our top level supervisor will be called Universe.Supervisor. It simply spins up a single instance of the Universe worker, the Cell.Supervisor supervisor, and the Cell.Registry which is an instance of Elixir’s Registry module:


children = [
  worker(Universe, []),
  supervisor(Cell.Supervisor, []),
  supervisor(Registry, [:unique, Cell.Registry])
]
supervise(children, strategy: :one_for_one)

Notice were’s using a :one_for_one supervision strategy here. This means that all of our children will be immediately started, and if a child ever fails, that process (and only that process) will be restarted.


Our Cell.Supervisor, the supervisor that manages all dynamically added cell processes, is a little more interesting.

Instead of immediately spinning up child processes, we create a template describing the the type of process we’ll be supervising in the future:


children = [
  worker(Cell, [])
]
supervise(children, strategy: :simple_one_for_one, restart: :transient)

The :simple_one_for_one strategy informs the system that we’ll be dynamically adding and removing children from this supervision tree. Those children will be Cell worker processes.

The :transient restart strategy means that if the Cell process is killed with a :normal or :shutdown message, it will not be restarted. However, if the Cell processes experiences a problem and dies with any other message, it will be restarted by the Cell.Supervisor.


Our Cell.Supervisor module also has a function called children:


def children do
  Cell.Supervisor
  |> Supervisor.which_children
  |> Enum.map(fn
    {_, pid, _, _} -> pid
  end)
end

The children function returns all living cell processes currently being supervised by Cell.Supervisor. This will be useful when we need to tick each cell in our Universe module.

The Universe

Our Universe module is the driving force in our Game of Life simulation. It’s literally what makes the cells tick.

If we had to tell Universe what to do in plain English, we might say:

Get all living cells. Asynchronously call tick on each one. Wait for all of the ticks to finish. Kill, or reap, all cells that will die from loneliness, and create, or sow, all of the cells that will be born.

Now let’s compare those instructions with our the code in our tick handler:


get_cells()
|> tick_each_process
|> wait_for_ticks
|> reduce_ticks
|> reap_and_sow

Perfect. I might even go so far as to say that the Elixir code is more readable than plain English.

As we dig into each of these functions, we’ll find that they’re still very descriptive and understandable. The get_cells function simply calls the Cell.Supervisor.children function we defined earlier:


defp get_cells, do: Cell.Supervisor.children

The tick_each_process function maps over each cell process and calls Cell.tick as an asynchronous Task:


defp tick_each_process(processes) do
  map(processes, &(Task.async(fn -> Cell.tick(&1) end)))
end

Similarly, wait_for_ticks maps over each asynchronous process, waiting for a reply:


defp wait_for_ticks(asyncs) do
  map(asyncs, &Task.await/1)
end

reduce_ticks, along with the helper function accumulate_ticks, reduces the response from each call to Cell.tick into a tuple holding a list of cells to be reaped, and a list of cells to be sown:


defp reduce_ticks(ticks), do: reduce(ticks, {[], []}, &accumulate_ticks/2)

defp accumulate_ticks({reap, sow}, {acc_reap, acc_sow}) do
  {acc_reap ++ reap, acc_sow ++ sow}
end

Lastly, reap_and_sow does exactly that: it kills cells marked for death, and create cells queued up to be born:


defp reap_and_sow({to_reap, to_sow}) do
  map(to_reap, &Cell.reap/1)
  map(to_sow,  &Cell.sow/1)
end

Take a look at the entire Universe module on Github.

The Cell

We’ve seen that while Universe is the driver of our simulation, it defers most of the computational work and decision making to individual cells. Let’s dive into our Cell module and see what’s going on.

The Cell.reap and Cell.sow methods we saw in Universe are fairly straight-forward:

The reap function simply calls Supervisor.terminate_child to remove the given cell process from the Cell.Supervisor tree.


def reap(process) do
  Supervisor.terminate_child(Cell.Supervisor, process)
end

Similarly, sow calls Supervisor.start_child to create a new process under the Cell.Supervisor tree, passing in the cell’s position as its initial state:


def sow(position) do
  Supervisor.start_child(Cell.Supervisor, [position])
end

The real magic of our Game of Life simulation happens in the cell’s tick function.

During each tick, a cell needs to generate a list of cells to reap (which will either be an empty list, or a list containing only itself), and a list of cells to sow.

Generating the to_reap list is easy enough:


to_reap = position
|> do_count_neighbors
|> case do
     2 -> []
     3 -> []
     _ -> [self()]
   end

We count the number of living neighbors around the cell. If the cell has two or three neighbors, it lives on to the next generation (to_reap = []). Otherwise, it dies from loneliness (to_reap = [self()]).

The do_count_neighbors functions does what you might expect. Given a cell’s position, it finds all eight neighboring positions, filters out all dead neighbors, and then returns the length of the resulting list of living neighbors:


defp do_count_neighbors(position) do
  position
  |> neighboring_positions
  |> keep_live
  |> length
end

After we’ve generated our to_reap list, our cell needs to generate a list of cells to be born.

From an individual cell’s perspective, this is a process of looking for any dead (unoccupied) neighboring positions and filtering out those that do not have enough living neighbors to be born into the next generation:


to_sow = position
|> neighboring_positions
|> keep_dead
|> keep_valid_children

The keep_valid_children function goes through the provided list of unoccupied positions, filtering out positions with a neighbor count not equal to three:


defp keep_valid_children(positions) do
  positions
  |> filter(&(do_count_neighbors(&1) == 3))
end

This means that only dead cells with exactly three neighbors (one of which is the current ticking cell) will be born into the next generation.


Now that we’ve generated out to_reap and to_sow lists, our cell process is finished ticking.

We can send our reply back to the universe, being sure to preserve position as our current state:


{:reply, {to_reap, to_sow}, position}

Take a look at the entire Cell module on Github.

Finding Neighbors with Registry

When generating both the to_reap and to_sow lists, cells were required to determine if neighboring cells were living or dead.

This was done with the keep_live and keep_dead functions, respectively:


defp keep_live(positions), do: filter(positions, &(lookup(&1) != nil))

defp keep_dead(positions), do: filter(positions, &(lookup(&1) == nil))

The key here is that we’re calling lookup on each position. The lookup function translates a cell’s position into a PID for that cell’s active process.


def lookup(position) do
  Cell.Registry
  |> Registry.lookup(position)
  |> Enum.map(fn
    {pid, _valid} -> pid
    nil -> nil
  end)
  |> Enum.filter(&Process.alive?/1)
  |> List.first
end

Here is where the Registry shines.

We’re using Registry.lookup to find a process in our Cell.Registry based on a given {x, y} position.

Registry.lookup will give us a list of {pid, value} tuples (or an empty list). Since we only want the pid, we can pull it out of the tuple.

Next, we filter the resulting PIDs with Process.alive?. After reaping a cell with Supervisor.terminate_child, the cell’s process will be removed from the Cell.Supervisor supervisor, but the process may not be fully removed from the Cell.Registry.

This means our cells can potentially interact with “ghost neighbors”; neighboring cells who are in the process of dying, but are not quite completely dead.

Adding a Process.alive? filter prevents our cell from interacting with this ghost neighbors (and prevents a very frustrating, subtle bug).

Running the Simulation

Now that we’ve built our process-driven simulation, it’s time to test it out.

We can fire up our Game of Life environment by starting an interactive Elixir shell:


iex -S mix

Next, let’s spawn three cells in a row. This will create a “blinker” pattern:


Cell.sow({0, 0})
Cell.sow({1, 0})
Cell.sow({2, 0})

Now let’s fire up Erlang’s observer to get a high level view of our universe:


:observer.start

We can see the three cells we just added to the universe below the Cell.Supervisor supervision tree. Also notice that those processes are linked to the Cell.Registry process.

To test out our Cell.Supervisor, let’s manually kill one of our cell processes. Send a kill exit message to one of the cells, and notice that after the process dies, another process immediately takes its place.

This means that any unintended errors in a Cell process won’t bring down our entire life simulation. Awesome!


Now that our initial conditions are set up, let’s tick our universe:


Universe.tick

Switching back to our observer, we can see that two of the three cell processes have been removed and two new processes have been added. If we look at the state of these new processes, we’ll see that they live at positions {1, 1}, and {1, -1}, as expected.

If we tick our universe again, we would see that those two processes would be killed, and two new processes would be added in their place. Their positions would oscillate back to {0, 0} and {2, 0}. Notice that the process for the cell living at position {0, 1} is still alive and well.

We can tick our universe as many times as we want:


1..10_000
|> Enum.map(fn n -> Universe.tick end)

After all of the ticks are processed, we can switch back to our observer and see that we still have three living cells, as expected.

Let’s restart our universe and try again with a more interesting pattern. Let’s try a “diehard” pattern, which is a methuselah that dies after 130 generations:


[
                                                  {6, 2},
  {0, 1}, {1, 1},
          {1, 0},                         {5, 0}, {6, 0}, {7, 0},
]
|> Enum.map(&Cell.sow/1)

1..130
|> Enum.map(fn
              n -> Universe.tick
                   :timer.sleep(500)
            end)

If you watch your observer as it slowly runs through the each tick, you’ll see that the number of active processes skyrockets and then eventually fades to zero.

Final Thoughts

Truth be told, the goal of this project was to get deeper hands-on experience with Elixir processes, supervision trees, and the new Registry functionality.

At the end of the day it was an excellent experience. I learned quite a few important lessons the hard way. If you’re interested in learning Elixir or how to “think in processes”, I highly recommend you take on a similar project.

While this Game of Life implementation isn’t the fastest or most efficient, it does come with its interesting benefits.

It’s incredibly resilient. Every cell process, and the universe process can fail and restart seamlessly. Catastrophic failure can only take place if either the Universe.Supervisor, or the Cell.Supervisor fail, which is unlikely to happen.

It’s concurrent and parallel out of the box. Our asynchronous calls to Cell.tick are distributed across every CPU on our node. The Erlang VM automatically takes full advantage of its environment and orchestrates the running of these parallel processes.


As far as future work for this project, I have lots of ideas.

I’d like to give cells more independence, and remove the need for the Universe driver module. I imagine each cell automatically trying to progress into future generations as soon as all of its neighboring cells have all necessary information to do so.

I’d also like to spread the simulation across multiple nodes. I imagine a massive Game of Life simulation running on dozens of EC2 instances, orchestrated through an edeliver powered release.

Lastly, I’d like to give the simulation a simple web-based user interface, and potentially the ability to run multiple simulations at once.

If you’d like to take this project out for a test run, or get a better look at the source, be sure to check it out on Github!

My Favorite Pattern Without a Name

On Jan 30, 2017 by Pete Corey

My recent dives into Elixir have been changing how I write software. More and more I find myself writing in a functional style, even in languages where its not the norm.

As my style of programming changes, I find myself re-using patterns and ideas across many different languages.

This article is about one of the patterns I use most often; interestingly, a pattern without a clear name.

As my friend Dean Radcliffe pointed out, I'm probably describing a fluent interface as described by Martin Fowler. Today I learned.

An Example

I often find myself wanting to chain together many function calls, passing in the results of previous calls into the next function in the chain.

Sometimes, the next function in the chain only needs the result of the previous function. Sometimes, it needs the result of a function further back in the chain.

To make things more real, let’s consider an example. Imagine we’re building a system that receives and responds to text messages from users. In this example, we’ll have three functions:


def get_user(phone_number) do
  ...
end

def get_response(message) do
  ...
end

def send_response(user, response) do
  ...
end

The get_user function takes in the sender’s phone number and returns a corresponding user object. The get_response function takes in the sender’s text message and returns a response. send_response will take in a response and the user to send that response to.

One way of calling these three methods together might look like this:


user = get_user(sms.from)
response = get_response(sms.message)
send_response(user, response)

This is a fine solution. However, we’re binding user and response for no other reason than to pass them into send_response.

Additionally, if our methods evolve over time to need more inputs, or more intermediary values, our function calls will get more and more tangled up.

The Pattern in Elixir

Instead of writing each function to take in exactly the arguments it needs, and returning a single value, let’s broaden our scope.

Let’s create a larger, all-encompassing state object that will contain all inputs, intermediary values, and outputs of our function chain.

In our case, we’ll need the sms input, the user object, and our system’s response:


%{sms: sms, user: nil, response: nil}

We can then rewrite our get_user, get_response, and send_response methods to take our state object as an argument, and return a (potentially modified) state object as a result:


def get_user(state = %{sms: %{from: from}}) do
  ...
  %{state | user: ...}
end

def get_response(state = %{sms: %{message: message}}) do
  ...
  %{state | response: ...}
end

def send_response(state = %{user: user, response: response}) do
  ...
end

Now we can cleanly chain together all of our function calls:


%{sms: sms, user: nil, response: nil}
|> get_user
|> get_response
|> send_response

Beautiful!

What’s great is that we can easily inject new functions into our chain without worrying about juggling inputs and outputs. Let’s add a function to the end of our chain that logs the entire interaction:


%{sms: sms, user: nil, response: nil}
|> get_user
|> get_response
|> send_response
|> log_interaction

In a similar vein, we can easily add debug logging (or anything else) between each step of the chain to get an in-depth look at what’s happening between each call in the chain:


%{sms: sms, user: nil, response: nil}
|> get_user
|> IO.inspect    # debug logging
|> get_response
|> IO.inspect    # debug logging
|> send_response
|> IO.inspect    # debug logging

Now in Javascript

We can apply this same pattern in Javascript. The Lodash library gives us an excellent set of tools for chaining together function calls. Let’s put them to work.

Imagine we have a similar set of functions:


function get_user({sms: {from}}) {
  ...
}

function get_response({sms: {message}}) {
  ...
}

function send_response({user, response}) {
  ...
}

We can use Lodash to run through our chain, just like we did in Elixir:


_.chain({sms, user: undefined, response: undefined})
 .thru(get_user)
 .thru(get_response)
 .thru(send_response)
 .value();

Similarly, we can tap into any point of this chain to add debug logging:


_.chain({sms, user: undefined, response: undefined})
 .thru(get_user)
 .tap(console.debug)    // debug logging
 .thru(get_response)
 .tap(console.debug)    // debug logging
 .thru(send_response)
 .tap(console.debug)    // debug logging
 .value();

What’s in a Name

There are lots of names that we can throw at this pattern. Some of them stick better than others, but I’m not fully satisfied with any of them.

I asked in the Elixir slack channel if anyone had a name for this kind of thing:

Is there a generic name for building a large, all encompassing state object, and then passing it through a lot of different functions that might only work on a subset of that state? What Plug does, for example.

I find myself doing that kind of thing a lot, and I wonder if there’s some kind of name/pattern that describes it.

Dependency injection?

Yeah, maybe. I’d describe it as maybe DI + method chaining. DI seems like a weird thing to talk about in a purely functional setting. Of course dependencies will be injected.

I was trying to look at “popular" non-Elixir projects (like Redux or Om) which use the same concept to see if they identify the pattern by a name, but I’m not seeing anything.

Dependency injection (DI) might apply in a way, but I usually think of dependency injection in terms of passing around functionality, not data. In a pure functional context, everything could be considered dependency injection in that sense.

I mentioned “method chaining”, which seems to describe a nice side-effect of the pattern, not the pattern itself.


Maybe the best pattern name would be “Middleware”, as seen in libraries like Express, React, and Plug.

The pattern used by these libraries is exactly what I’ve been describing, but I’m not sure about the name. The term “middleware” was originally used to describe something very different than what we’ve been talking about.

On top of that, there’s some disagreement amongst software developers as to whether “middleware” is the term we should be applying to this pattern (or if this is even a pattern at all).

Either way, I’ll probably refer to this pattern as middleware when talking with other developers, and I’ll definitely continue using my favorite pattern without a name.