Running in Production
Running in Production
#deployment #elixir #flask #hosting #node #rubyonrails #phoenix #production #python #techstacks #webdevelopment #softwarehow-to #technology #news #technews #business #careers

Hear about how folks are running their web apps in production. We'll cover tech choices, why they chose them, lessons learned and more.

Mux Is an API Based Platform That Lets You Process and Stream Videos

May 18, 2020 1:20:48 77.58 MB Downloads: 0

In this episode of Running in Production, Dylan Jhaveri talks about building an API driven video platform called Mux. It uses Phoenix, Elixir and Go to handle billions of video views a month. It’s hosted on AWS and GCP with Kubernetes and has been up and running since early 2016.

Dylan covers how video streaming works, processing billions of events a month, taking advantage of Elixir and Phoenix’s features, providing a zero downtime public API, continuously deploying their products, working with massive databases, metered billing and tons more.

Topics Include

  • 1:14 – How online streaming video works with HLS and where Mux fits into the picture
  • 7:51 – Mux lets you post a video to their API and they give you an HLS playback URL
  • 8:24 – Mux has been up and running since January 2016 and went through YCombinator
  • 8:37 – Mux Data is another service they offer, it’s like New Relic but for video data
  • 12:04 – They process billions of video views per month through Mux Data
  • 12:36 – You could use Mux as a lower level alternative to Vimeo or Wistia
  • 13:33 – Sometimes embedding iframes can be problematic and Mux can help in this area
  • 14:35 – About 45 people work at Mux and half are involved with engineering
  • 15:03 – Motivation for using Phoenix and Elixir, even when they were very new tools
  • 16:52 – Their main public API is an out of the box Phoenix app
  • 17:52 – They have a real-time dashboard that is powered by websockets and channels
  • 20:28 – Some of Mux’s customers have millions of concurrent video views through that
  • 20:42 – Will you switch to using Live View? Probably not since they are so API driven
  • 21:51 – A dozen or so Go microservices and Kafka handle processing the videos
  • 23:25 – Go is a great fit for super CPU intensive tasks such as video encoding
  • 24:03 – The video processing infrastructure was very well thought out early on
  • 24:50 – The public API is RESTful and there’s ~40-50 endpoints with a few private endpoints
  • 26:14 – Cookie based auth is done in a browser but there’s tokens for API access
  • 26:47 – The exq library is used for processing jobs asynchronously in Elixir land
  • 27:22 – exq runs within a supervisor of your app, not a dedicated OS level service
  • 28:21 – Prometheus is used for metrics but it’s not hooked into Elixir Telemetry (yet)
  • 29:26 – Kubernetes and Docker drive their production infrastructure
  • 29:47 – Buildkite is used for their CI / CD pipeline
  • 32:08 – Deployments are very automated, a human only needs to merge to a specific branch
  • 32:53 – The video processing microservices are in 1 mono repo, but there’s 2 other repos
  • 33:33 – There’s PR approvals in place but all developers can merge to the production branch
  • 34:39 – Code reviews are really important and you need to trust your developers
  • 35:41 – The Elixir app has a PostgreSQL billing DB and also uses ClickHouse (SQL based)
  • 37:53 – ClickHouse lets them store billions of rows and access everything quickly
  • 40:58 – You do write SQL queries with ClickHouse but it doesn’t work with Ecto out of the box
  • 41:44 – The Elixir API runs on AWS with an AWS load balancer sitting in front of it all
  • 42:20 – The video infrastructure runs on Google Cloud
  • 42:56 – How many servers do you run in total? Hard to tell really, but it’s a lot of compute
  • 43:44 – Despite being on AWS, they are not using Amazon’s managed Kubernetes (EKS)
  • 44:01 – All payments go through Stripe, including the metered billing which they hand rolled
  • 45:06 – Instead of billing based on bandwidth, Mux bills by minutes watched
  • 46:06 – SendGrid is used for transactional emails, Sentry for errors and Opsgenie for paging
  • 46:48 – All sorts of CI / CD related information gets sent over to a Slack channel
  • 47:08 – Developers are broken out into 4 cross functional teams
  • 48:31 – There’s 2 flavors of SDKs that Mux has (REST API wrappers and video players)
  • 50:21 – They currently have 22 different video players to account for across many platforms
  • 50:36 – Efficiently creating so many different SDKs by having a core library for each language
  • 54:20 – It’s sort of like having a core payment library and supporting Stripe, PayPal, etc.
  • 54:41 – The SDK team needs to be aware of many different languages and players
  • 55:16 – Another key metric to track is the video upscale and downscale percentages
  • 56:47 – As of today Mux is focused on supplying service quality metrics
  • 58:08 – There’s a lot of data stored but it all gets rolled over after 90 days
  • 58:42 – The API is deployed all the time, but there’s zero down time deploys
  • 59:45 – There’s been one day in the past there they had to put the API in read-only mode
  • 1:00:19 – The data is backed up, but Dylan isn’t sure how often (but it happens, he swears!)
  • 1:00:42 – Video thumbnails can be picked out from any timestamp, even animated GIFs too
  • 1:02:21 – For now you need to supply your own closed captions to Mux
  • 1:03:52 – Captions are downloaded, cached locally until processed and then backed up too
  • 1:04:38 – Smoke tests and various alarms help detect issues in production (they use Flink)
  • 1:06:25 – Uptime is important, Mux has high profile clients where downtime is not an option
  • 1:06:52 – Rate limiting is done at the Elixir level for API calls with the ex_rated library
  • 1:07:25 – It’s a reasonable idea to always assume users are out to get you
  • 1:07:52 – For video rate limiting, it’s up to the CDN and they use a few different CDNs
  • 1:09:33 – You could build a live streaming service like Twitch with Mux’s API
  • 1:13:19 – The Elixir API doesn’t get billions of calls a month but it’s a still a lot
  • 1:16:37 – Best tips? Video is hard and it keeps getting more and more complicated
  • 1:18:15 – Fortunately the video player SDK’s churn isn’t too high due to the HTML5 spec
  • 1:19:14 – You can email Dylan or contact him on Twitter, also Mux is hiring too!

Links

📄 References ⚙️ Tech Stack 🛠 Libraries Used

Support the Show

This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend.

  • Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included.
  • Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.