Simplifying Elixir Releases With Edeliver

Written by Pete Corey on Jan 16, 2017.

In our previous two posts, we’ve built releases manually with Distillery, and deployed an upgrade release to an existing Elixir application. As we’ve seen, this was a very hands-on process.

Thankfully, there is another tool that we can use to simplify our release process. In this post, we’ll dive into using edeliver to build and release Elixir applications.

Our Example Application

To help walk ourselves through the process of using edeliver, let’s create a new Phoenix application that we can deploy to a “production server”.

Let’s start by creating a new, basic Phoenix project:


mix phoenix.new hello_edeliver --no-ecto

We’ll be deploying this application as is, so let’s move onto the process of installing and configuring our release tools.

First, we’ll add dependencies on edeliver, and distillery, and add edeliver to our projects list of applications:


def application do
  [...,
   applications: [..., :edeliver]]
end

defp deps do
  [
   ...
   {:edeliver, "~> 1.4.0"},
   {:distillery, ">= 0.8.0", warn_missing: false}]
end

Note: edeliver is a high-level tool designed to orchestrate the creation, deployment, and management of Elixir releases. Under the covers, it can use Distillery, exrm, relx, or rebar to build release bundles.

Our application won’t use any “production secrets”, so to simplify this introduction to edeliver, let’s remove our dependency on the config/prod.secrets.exs file by commenting out its import_config line in config/prod.exs:


# import_config "prod.secrets.exs"

Finally, while we’re in config/prod.exs, let’s add a few vital configuration options to our HelloEdeliver endpoint:


config :hello_edeliver, HelloEdeliver.Endpoint,
  http: [port: {:system, "PORT"}],
  url: [host: "...", port: {:system, "PORT"}],
  server: true,
  root: ".",
  version: Mix.Project.config[:version],
  cache_static_manifest: "priv/static/manifest.json"

And with that, our example application should be ready for release.

Before we move on, let’s fire up our development server with mix phoenix.server and make sure that everything looks as we’d expect.

Configuring Distillery and edeliver

Now that our application is ready, we need to spend some time configuring our deployment tools.

First, let’s create our Distillery configuration file (rel/config.exs) with the release.init mix task:


mix release.init

The default configuration options provided by release.init should be fine for our first deployment.

Next, let’s take a look at our edeliver configuration file (.deliver/config). We’ll update the provided fields and point edeliver to a remote “build host” and “production host”:


APP="hello_edeliver"

BUILD_HOST="ec2-52-87-163-123.compute-1.amazonaws.com"
BUILD_USER="ec2-user"
BUILD_AT="/home/ec2-user/hello_edeliver/builds"

PRODUCTION_HOSTS="ec2-54-172-3-38.compute-1.amazonaws.com"
PRODUCTION_USER="ec2-user"
DELIVER_TO="/home/ec2-user"

At this point, we could add any number of production hosts, or add another environment entirely, such as a staging environment.


Because we’re building a Phoenix application, we’ll need to build and digest all of our static assets before building our release.

We’re building our release bundle on our remote build host, so we’ll need to instruct edeliver to do these things for us. Thankfully, edeliver comes with a host of pre and post hooks that we can use to accomplish this task.

Let’s add a “pre compile” hook to build (npm install, npm run deploy) and digest (mix phoenix.digest) our static assets:


pre_erlang_clean_compile() {
  status "Installing NPM dependencies"
  __sync_remote "
    [ -f ~/.profile ] && source ~/.profile
    set -e

    cd '$BUILD_AT'
    npm install $SILENCE
  "

  status "Building static files"
  __sync_remote "
    [ -f ~/.profile ] && source ~/.profile
    set -e

    cd '$BUILD_AT'
    mkdir -p priv/static
    npm run deploy $SILENCE
  "

  status "Running phoenix.digest"
  __sync_remote "
    [ -f ~/.profile ] && source ~/.profile
    set -e

    cd '$BUILD_AT'
    APP='$APP' MIX_ENV='$TARGET_MIX_ENV' $MIX_CMD phoenix.digest $SILENCE
  "
}

By default, edeliver will store all release tarballs in the .deliver/releases folder. Let’s preemptively exclude this folder from revision control:


echo ".deliver/releases/" >> .gitignore

The last thing we need to do to prepare for our first release is to commit all of our changes!

Preparing Our Hosts

When using edeliver, we’ll always have at least two different types of remote environments.

The first is our build host. This is where edeliver actually builds our release. In our case, it’s running a variation of a mix release command. As we saw in a previous article, it’s important that the build host be nearly identical to the production host in terms or architectures, etc…

Our build host has several dependencies. We’ll need to install Git, Erlang & Elixir, and Node. Thankfully, we’ll only need to provision a single build host.

The second type of host is our deployment target. These machines will be running our application in environments such as staging or production. We can have any number of these hosts living in the wild.

Our deployment machines have no dependencies. We don’t even need to install Erlang - our release will bring it along for us.

However, we do need to set any required environment variables on these hosts. Out of the box, a Phoenix application needs a PORT value. We can export this from our ~/.profile:


export PORT=4000

If you’re using Amazon EC2 hosts, you’ll most likely authenticate with your remote hosts by passing along a *.pem file when you establish your SSH connection.

While edeliver doesn’t explicitly support this kind of authentication, adding a Host/IndentityFile entry in your ~/.ssh/config file for each of your remote hosts will authorize edeliver to communicate with these hosts:

Host ec2-foo.compute-1.amazonaws.com
  IdentityFile ~/my_identity_file.pem

Host ec2-bar.compute-1.amazonaws.com
  IdentityFile ~/my_identity_file.pem

Making Our Release

Once our application is ready, our deployment tools are configured, and our hosts are provisioned, we can build and deploy our release.

The first step is to instruct edeliver to build our release on our build host:


mix edeliver build release

Success! If everything went well, we should find our newly built release tarball in our release store (.deliver/releases).

Next, let’s push our initial release up to our production server:


mix edeliver deploy release to production

Another success! We’ve deployed our initial release to our production environment.

Now let’s fire up the Phoenix application in production:


mix edeliver start production

We can check that everything went well by using edeliver to ping our application:


mix edeliver ping production

If our application is up and running, we should receive a pong reply to our ping.

At this point, we should be able to navigate to our production host and find our application running at the port we’ve specified.

Final Thoughts

From my limited experience, edeliver is a fantastic tool.

While it does require up-front work (more work than using Distillery on its own), that work is purely up-front. Once you’ve provisioned your build host and set up your edeliver configuration file, building and deploying releases to any number of hosts is smooth sailing.

I’m excited to work more with edeliver. Expect an upcoming post on deploying hot upgrade releases to see how the process compares with just using Distillery.

Upgrade Releases With Distillery

Written by Pete Corey on Jan 9, 2017.

Now that we’ve deployed our first Elixir application using Distillery, let’s dive into the process of building and deploying a “hot upgrade” for our application!

We’ll move through the process of making a change to our application, using Distillery to build our upgrade, and then deploying the upgrade with zero downtime.

Making Our Changes

With the goal of keeping things simple, let’s make a small change to our application’s index.html.eex file. We’ll add a new <h3> tag that shows we’ve upgraded our application:


<div class="jumbotron">
  <h2><%= gettext "Welcome to %{name}", name: "Phoenix!" %></h2>
  <h3>Version 0.0.2!</h3>
  ...
</div>

Next, we’ll need to upgrade our project’s version in our mix.exs file:


def project do
  [app: :hello_distillery,
   version: "0.0.2",
   ...

It’s important to upgrade our project’s version so distillery can correctly build an upgrade/downgrade patch for our application.

Problems With Our Build

And with that, we should be ready to build our upgrade release. The process for building an upgrade is very similar to that of building a normal release:


MIX_ENV=prod mix do compile, phoenix.digest, release --env=prod --upgrade

Unfortunately, due to how we configured our project in the last article, this upgrade build will fail:


==> Assembling release..
==> Building release hello_distillery:0.0.2 using environment prod
==> Failed to build release:
    Hot upgrades will fail when include_erts: false is set,
    you need to set include_erts to true or a path if you plan to use them!

Because we installed Erlang on our production machine, we set include_erts to false in our release configuration file, indicating that we didn’t want to include the Erlang runtime in our final build.

Distillery is complaining that when building upgrade releases, include_erts either needs to be true (which will include the Erlang runtime installed on our development machine in the release), or a path pointing to the Erlang runtime we want to include in the release.

A Tale of Two Erlangs

My development machine is a Macbook, and my production machine is an Amazon Linux EC2 instance. This means that including my development machine’s version of Erlang in the release is not an option.

This means that we’ll have to copy the instance of Erlang we installed on our production server onto our development server and point to it with include_erts.

The Distillery walkthrough touches in this in the “Deploying Your Release” section:

If you are deploying to a different OS or architecture than the build machine, you should either set include_erts: false or include_erts: "path/to/cross/compiled/erts".

The latter will require that you have built/installed Erlang on the target platform, and copied the contents of the Erlang lib directory somewhere on your build machine, then provided the path to that directory to include_erts.

Following this advice, we’ll copy the Erlang runtime from our production server into a folder on our development machine (~/al-erlang):


scp -i ~/hello_distillery.pem -r \
    ec2-user@ec2...amazonaws.com:/usr/local/lib/erlang \
    /Users/pcorey/al-erlang

Now we can change our include_erts to point to our newly downloaded ~/al-erlang directory:


set include_erts: "/Users/pcorey/al-erlang"

And finally, we can build our upgrade release:


MIX_ENV=prod mix do compile, phoenix.digest, release --env=prod --upgrade

Hot Deploying Our Upgrade

Now that we’ve successfully built our upgrade release, we can deploy it to our production server.

The first thing we’ll need to do is ssh into our production server and create a 0.0.2 folder in our application’s releases directory:


mkdir ~/releases/0.0.2/

Next, we’ll hop back over to our development server and copy our newly built release tarball into that new directory:


scp -i /hello_distillery.pem \
    _build/prod/rel/hello_distillery/releases/0.0.2/hello_distillery.tar.gz \
    ec2-user@ec2-...amazonaws.com:/home/ec2-user/releases/0.0.2/ \
    hello_distillery.tar.gz

Lastly, we’ll switch back to our production server and run the upgrade command, passing in the new version of our application that we just uploaded:


./bin/hello_distillery upgrade 0.0.2

If everything went well, we should be able to refresh our application in the browser and see our "Version 0.0.2!" message!

Final Thoughts

Minus a few burs and rough edges, Distillery is a fantastic tool for building and deploying Elixir/Phoenix applications.

I imagine that all of the sticking points I encountered can be smoothed out with a combination of well-designed build scripts and building releases on a machine with the same architecture as the production environment.

Looking to the future, it looks like edeliver does exactly that. Expect to see an article in the next few weeks about simplifying the deployment process with edeliver!

Deploying Elixir Applications with Distillery

Written by Pete Corey on Dec 26, 2016.

While churning through the exercises in Programming Elixir (affiliate link), I came across a section dedicated to deploying Elixir applications using the Elixir Release Manager (exrm).

Browsing through the Elixir Release Manager project, I noticed that it is being replaced by Distillery:

I would highly recommend using [Distillery] moving forward, as most of my efforts will be put towards that project from now on.

Rather than learning how to use a deprecated tool, I decided it might be a better investment of my time to learn how to deploy Elixir applications using Distillery.

Below is a guide for doing a basic release with Distillery. For the most part, the instructions are very similar to doing a release with exrm, with a few minor differences.

Creating Our Application

Before we start, we’ll need an Elixir project we want to deploy. To keep things simple, we’ll use a bare-bones Phoenix application. We can create our application (called HelloDistillery) with the phoenix.new Mix task:

mix phoenix.new --no-ecto hello_distillery

Once we’ve got our application set up and tested locally (mix phoenix.server), we can start the Distillery deployment process.

Installing Distillery

The first step of deploying an Elixir project with Distillery is to add Distillery as a dependency. In our mix.exs file, add a dependency on :distillery version 1.0:


{:distillery, "~> 1.0"}

Now tell Mix to pull down the new dependency:

mix deps.get

Once installed, Distillery creates a new release Mix task. If we try to run it, we’ll be told that we need to do some initial configuration:

mix release
==> You are missing a release config file. Run the release.init task first

Following that advice, we can generate our initial config file with the release.init Mix task:

mix release.init

Review the newly generated rel/config.exs file. For our purposes, the defaults should be fine.

Configuring Our Release

The last step before we build our first release is to do some final configuration on the environment we’ll be deploying.

In our case, we’ll be deploying the prod environment. If we look in our config/prod.exs file, we’ll see a comment block addressing releases:


# ## Using releases
#
# If you are doing OTP releases, you need to instruct Phoenix
# to start the server for all endpoints:
#
# config :phoenix, :serve_endpoints, true
#
# Alternatively, you can configure exactly which server to
# start per endpoint:
#
#     config :hello_distillery, HelloDistillery.Endpoint, server: true

Following this advice and the advice in the Distillery Phoenix Walkthrough, we’ll add a few configuration options to our HelloDistillery endpoint, and update its url:


config :hello_distillery, HelloDistillery.Endpoint,
  ...
  url: [ ... ],
  server: true,
  root: ".",
  version: Mix.Project.config[:version]

Once we’ve made those configuration changes, we’re ready to build and deploy our release!

Building Our Release

Building our release with Distillery is an easy process.

If we’re deploying a Phoenix application, we’ll need to be sure that we’ve bundled our front-end assets. We can do that with the following command:

brunch build --production

Next, we’ll run a series of Mix tasks. First, we’ll compile our project. Next, we’ll compress and digest all of our static files. Finally, we’ll build our prod release:

MIX_ENV=prod mix do compile, phoenix.digest, release --env=prod

The mix release task builds our release and places it in the _build folder. To make sure everything went well, we can run the release locally using the following command:

PORT=8080 ./_build/prod/rel/hello_distillery/bin/hello_distillery foreground

If the release was successful, we should see our application’s server logs in the console, and we should be able to access your application at http://localhost:8080.

Deploying Our Release

At this point, I’ll assume that you’ve already provisioned the machine you’ll be using for your production environment.

As a side note, I decided to deploy my application to an Amazon EC2 instance running 64 bit Amazon Linux.

Because we’re using the default configuration value of include_erts: false, we’ll need to be sure that Erlang is installed on the machine we’ll be deploying our release to.

Following these instructions, I was able to easily build Erlang from source and install it on my production machine. You may also have luck with a pre-compiled Erlang package.

Be sure to use the same version of Erlang on your production machine as the machine you built your release on. If you do not, you may encounter obtuse errors when trying to run your release.

Once our machine is provisioned and Erlang is installed, we’ll copy our release tarball from our development machine up to our production machine:

scp -i ~/hello_distillery.pem \ 
       _build/prod/rel/hello_distillery/releases/0.0.1/hello_distillery.tar.gz \
       ec2-user@ec2-...amazonaws.com:/home/ec2-user

Be sure to tweak this command to suite your needs. I’m authenticating with a pem file stored in my home directory, and deploying to the home directory of the default ec2-user of my Amazon EC2 instance.

Once our release tarball is pushed up to our production server, we’ll need to extract it and run the release.

On the production server, extract the newly uploaded tarball:

tar -xzf ~/hello_distillery.tar.gz

Now is the moment of truth. Run the release binary using the foreground option:

PORT=8080 ./bin/hello_distillery foreground

If everything went well, we should see our application’s server logs in the console. Additionally, we should be able to access our application at our production machine’s host on port 8080 (granted you’ve properly opened that port).

Now that we know everything works, kill the server process. This time we’ll run the hello_distillery application as a daemon:

PORT=8080 ./bin/hello_distillery start

Once we’ve started the application, we can ensure that it’s running with a ping:

./bin/hello_distillery ping
> pong

And with that, we’ve deployed a basic Phoenix application using Distillery!

Final Thoughts

This was a simple run-through of a happy-path first deployment using Distillery.

I highly recommend you check out Distillery’s documentation, especially the Walkthrough and Phoenix Walkthrough sections. They go into much more detail about the various aspects of deploying with Distillery.

Overall, the process of deploying an application with Distillery was very similar to the Elixir Release Manager process described in Programming Elixir.

If you’d like to see Distillery in action, check out this short webm of entire entire deployment process: