In this episode of Running in Production, Julien Blanchard goes over building Smart Music which uses a combination of Rails, Phoenix and .NET Core. It has roughly half a million users and it’s all hosted on AWS with EKS, ECS and Elastic Beanstalk. It’s been up and running since 2016.

There’s around 20 developers working on the project. We talked about managing git repos with a few apps, TDD, using GraphQL with Phoenix, contexts, multiple databases with Rails, InfluxDB, GitHub Actions and tons more.

Topics Include

• 2:41 – Roughly half a million users are on the platform (~1.5k requests a minute at times)
• 3:27 – What Rails, Phoenix and .NET Core are being used for
• 5:38 – End users of the site interact with the Rails app and .NET Core is for authentication
• 6:10 – It’s an API back-end driven app and React / EmberJS is used on the front-end
• 6:35 – Motivation for using Phoenix and Elixir for the data ingesting app
• 9:28 – About 20 developers work on all of the different parts of the site
• 9:55 – Organizing the git repos for each of the apps
• 10:34 – The back-end code has many tens of thousands of lines of code
• 11:04 – TDD is something their company practices and they like it a lot
• 12:24 – A JS front-end makes sense for this app since the UI is live and dynamic
• 13:17 – Trying to visualize a live sheet music application that helps you learn notes
• 14:02 – Maybe Phoenix LiveView could have been used, but they prefer what they chose
• 14:33 – The TL;DR on GraphQL and why in this case it works better than a RESTful API
• 17:55 – Docker isn’t being used in dev, but Kubernetes is being used in production
• 18:29 – PostgreSQL, InfluxDB and Redis are used to manage the data and for caching
• 19:32 – They knew from the start that InfluxDB would be needed to store the time data
• 20:33 – Redis is being used as a cache through AWS ElastiCache
• 21:49 – nginx is sitting in front of the Rails application with Elastic Beanstalk
• 22:44 – Motivation for picking AWS over Google Cloud and other providers
• 23:40 – AWS Aurora is being used to manage PostgreSQL
• 24:51 – They are using the Rails 6.x feature to select multiple databases
• 25:33 – Rails is very nice when it comes to getting community driven features merged in
• 26:08 – Julien also really likes Phoenix and here’s how they use contexts
• 28:50 – File uploads are sent directly to S3 using the ex_aws Elixir library
• 30:02 – For Kubernetes, they are using the managed EKS service from AWS
• 31:07 – (2) pretty beefy boxes with 8 GB of memory power the EKS cluster (overkill)
• 31:36 – They are still feeling out the resource usage of their services
• 32:18 – (20)’ish EC2 instances power the Elastic Beanstalk set up for the Rails app
• 32:54 – CloudFront is being used as a CDN for book covers but not much else
• 33:38 – Walking us through a code deploy from development to production
• 34:46 – Getting rid of Jenkins is the next step but GitHub Actions is a bit insecure currently
• 35:49 – GitHub Actions is a great tool and it’s being used for more than just CI
• 36:44 – You can use GitHub Actions to run tasks periodically (separate from git pushes)
• 37:27 – Dealing with big database migrations with scheduled down time
• 38:54 – New Relic and the ELK stack take care of metrics and logging
• 39:18 – Sentry.io (self hosted version) is being used to track exceptions
• 39:42 – The time series data doesn’t end up getting logged by these tools
• 40:20 – Braintree is used as a payment gateway to handle credit card and PayPal payments
• 41:44 – Transactional emails are being sent through AWS SES
• 42:24 – In app notifications are coming soon to SmartMusic (websockets, etc.)
• 44:05 – Another use case for websockets and events will be for collaboration features
• 44:49 – The databases are backed up daily and S3 is very redundant in itself for user files
• 45:31 – VictorOps handles alerting if a service happens to go down
• 45:58 – New Relic is being used in a few of the applications
• 46:55 – Handling bot related issues with nginx’s allow / deny IP address feature
• 48:46 – Best tips? Make a solid proof of concept in your new tech before switching to it fully
• 50:36 – Biggest mistake? Trying to use your old coding habits in a different language
• 51:27 – Dealing with N + 1 queries with GraphQL using DataLoader
• 52:58 – Ecto Multi is awesome for ensuring multiple things happen successfully
• 54:10 – Check out Julien’s blog, @julienXX on GitHub and he’s on Source Hut

Links

• https://en.wikipedia.org/wiki/Practice_(learning_method)#Deliberate_practice
• https://www.w3.org/2017/12/musicxml31/
• https://emberjs.com/
• https://guides.rubyonrails.org/active_record_multiple_databases.html
• https://github.com/dependabot
• https://www.elastic.co/what-is/elk-stack
• http://nginx.org/en/docs/http/ngx_http_access_module.html
• https://martinfowler.com/bliki/BlueGreenDeployment.html
• https://stackoverflow.com/questions/97197/what-is-the-n1-selects-problem-in-orm-object-relational-mapping
• https://sourcehut.org/

• rails →
• ruby →
• graphql →
• phoenix →
• elixir →
• aurora →
• aws →
• braintree →
• dotnet-core →
• eks →
• github-actions →
• influxdb →
• jenkins →
• kubernetes →
• new-relic →
• nginx →
• postgres →
• react →
• redis →
• s3 →
• sentry →
• ses →
• victorops →

• https://github.com/ex-aws/ex_aws
• https://github.com/absinthe-graphql/absinthe
• https://github.com/graphql/dataloader

Support the Show

This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend.

• Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included.

• Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.