In this episode of Running in Production, Marcelo Lebre goes over building a remote job platform on remote.com. They serve about 100k+ requests a day and it’s all hosted on a small AWS ECS cluster. It’s been up and running since early 2019.

We covered a lot of ground, from using Elasticsearch to developing API based applications to Elixir’s ecosystem and everything in between. One takeaway is to be mindful of over engineering your code base and try to focus on the things that are important to your application.

Topics Include

• 3:15 – Taking over the old remote.com site and rebuilding it from scratch
• 4:03 – Motivation for rewriting everything in Elixir and Phoenix
• 6:25 – Phoenix quickly became a go-to tool for Marcelo
• 7:27 – He first used it in another project to deal with 1000s of requests per minute
• 8:08 – It was pretty easy to get the rest of his team on-board with Elixir
• 9:40 – Writing a monolithic code base using single responsibility principles
• 11:47 – The code base’s lib/ folder has 500+ files and many thousands of lines of code
• 12:59 – Phoenix contexts are being used but with an added level of SRPs
• 14:26 – This will allow Marcelo to easily break up his code base later if needed
• 16:15 – The value in writing about things after having a lot of real world experience
• 20:28 – The Phoenix app is API based with a React front-end
• 21:36 – NextJS is also being used to have server rendered pages
• 22:31 – Letting developers pick and choose their dev style leads to increased productivity
• 23:36 – The front-end team doesn’t need to know about the Phoenix back-end
• 24:52 – There’s 3 different git repos (back-end, front-end and the new app)
• 25:13 – The front-end devs can get going easily due to excellent documentation
• 27:19 – Docker is being used in staging and production but not development
• 29:43 – PostgreSQL, Redis and Elasticsearch are being used and it’s hosted on AWS ECS
• 30:16 – Not super happy with AWS ECS and he wants to switch to Kubernetes eventually
• 30:51 – Gitlab is used for continuous integration and deployment
• 31:17 – Reasons for going with Elasticsearch (full text search and caching)
• 35:59 – Live View isn’t being used but he’s keeping an eye on it
• 37:40 – The core of what’s needed to make Live View amazing is there
• 38:48 – Everyone would still be using IE 5 if no one tried things out in practice
• 40:53 – The Elixir community is very helpful if you get stuck
• 41:09 – Keeping the ops side under control by using AWS for the DB, cache and ES
• 42:27 – After looking at Heroku, Google Cloud, Azure and others - AWS looked good
• 43:26 – Serving 100k to a million requests per day for $4,000 a month before Phoenix
• 45:26 – The moment Remote switched to Phoenix the bill dropped to $500 / a month
• 45:56 – The entire staging environment runs with ~512 MB of memory
• 46:01 – The production environment runs on 2-4 instances with 2 GB of memory each
• 46:36 – Exq is being used to process jobs and it’s running in its own service
• 47:35 – 2 web servers and 1 worker is enough to serve things with a ~150ms response time
• 48:24 – Redis is being used to cache a few things (nothing like Rails level caching)
• 49:16 – Elixir / Phoenix are incredibly efficient with memory usage and releasing it
• 51:11 – AppSignal’s notifications help with giving a general peace of mind
• 52:28 – Searching and applying to jobs is free but posting jobs require a payment
• 54:44 – Payments are handled with Stripe using the stripity_stripe library
• 55:40 – Walking through a code deploy from development to staging to production
• 57:04 – The Docker images are being stored on Gitlab’s registry service
• 57:23 – Secret management is done with Gitlab’s secret management tool
• 58:47 – Elixir releases are created using Docker multi-stage builds
• 1:00:05 – Having small Docker images helps a lot due to their deployment strategy
• 1:00:21 – Everything is backed up and the cache can be recreated from the DB
• 1:02:05 – Mail Gun is used for transactional emails and customer.io is used too
• 1:02:39 – Sometimes you get used to a service or tool always existing
• 1:03:34 – Best tips? Be careful about over engineering and keep your eyes on the prize
• 1:06:18 – Mistakes? Not enough tests, too many tests, finding a perfect structure, etc.
• 1:06:53 – Coming to acceptance that Elixir and Phoenix has a fairly new ecosystem
• 1:07:52 – Check out the remote.com blog or follow Marcelo on Twitter @marcelo_lebre

Links

• https://en.wikipedia.org/wiki/Single_responsibility_principle
• https://www.getpostman.com/
• https://dockyard.com/blog/2018/12/12/phoenix-liveview-interactive-real-time-apps-no-need-to-write-javascript
• https://martinfowler.com/bliki/BlueGreenDeployment.html
• https://edgeguides.rubyonrails.org/caching_with_rails.html#russian-doll-caching
• https://customer.io/

• phoenix →
• elixir →
• react →
• appsignal →
• aws →
• cloudflare →
• docker →
• ecs →
• elasticache →
• elasticsearch →
• gitlab-ci →
• mailgun →
• nextjs →
• postgres →
• rds →
• redis →
• stripe →

• https://github.com/akira/exq
• https://github.com/code-corps/stripity_stripe

Support the Show

This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend.

• Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included.

• Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.