Elixir Wizards is an interview-format podcast, focused on engineers who use the Elixir programming language. Initially launched in early 2019, each season focuses on a specific topic or topics, with each interview focusing on the guest's experience and opinions on the topic. Elixir Wizards is hosted by Eric Oestrich and Sundi Myint of SmartLogic, a dev shop that’s been building custom software since 2005 and running Elixir applications in production since 2015. Learn more about how SmartLogic uses Phoenix and Elixir. (https://smartlogic.io/phoenix-and-elixir?utm_source=podcast)
"You've Got a Job to Do" with Joel Meador
Today on Elixir Wizards Office Hours, SmartLogic Engineer Joel Meador joins Dan Ivovich to discuss all things background jobs.
The behind-the-scenes heroes of app performance and scalability, background jobs take center stage as we dissect their role in optimizing user experience and managing heavy-lifting tasks away from the main application flow. From syncing with external systems to processing large datasets, background jobs are pivotal to successful application management.
Dan and Joel share their perspectives on monitoring, debugging, and securing background jobs, emphasizing the need for a strategic approach to these hidden workflows.
Key topics discussed in this episode:
- The vital role of background jobs in app performance
- Optimizing user experience through background processing
- Common pitfalls: resource starvation and latency issues
- Strategies for effective monitoring and debugging of task runners and job schedulers
- Data integrity and system security in open source software
- Background job tools like Oban, Sidekiq, Resque, Cron jobs, Redis pub sub
- CPU utilization and processing speed
- Best practices for implementing background jobs
- Keeping jobs small, focused, and well-monitored
- Navigating job uniqueness, locking, and deployment orchestration
- Leveraging asynctask for asynchronous operations
- The art of continuous improvement in background job management
Links mentioned in this episode:
https://redis.io/
Oban job processing library https://hexdocs.pm/oban/Oban.html
Resque Ruby library for background jobs https://github.com/resque
Sidekiq background processing for Ruby https://github.com/sidekiq
Delayed Job priority queue system https://github.com/collectiveidea/delayed_job
RabbitMQ messaging and streaming broker https://www.rabbitmq.com/
Mnesia distributed telecommunications DBMS https://www.erlang.org/doc/man/mnesia.html
Task for Elixir https://hexdocs.pm/elixir/1.12/Task.html
ETS in-memory store for Elixir and Erlang objects https://hexdocs.pm/ets/ETS.html
Cron - https://en.wikipedia.org/wiki/Cron
Donate to Miami Indians of Indiana https://www.miamiindians.org/take-action
Joel Meador on Tumblr https://joelmeador.tumblr.com/
Special Guest: Joel Meador.