Discover the future of software from the people making it happen.Listen to some of the smartest developers we know talk about what they're working on, how they're trying to move the industry forward, and what you can learn from it. You might find the solution to your next architectural headache, pick up a new programming language, or just hear some good war stories from the frontline of technology.Join your host Kris Jenkins as we try to figure out what tomorrow's computing will look like the best way we know how - by listening directly to the developers' voices.
Batch Data & Streaming Data in one Atom (with Jove Zhong)
Every database has to juggle the need to process new data and to query old data. That task falls to any system that “does stuff and remembers stuff”. But it’s quite hard to really optimise one system for both use cases. There are different constraints on new and old data, and as a system gets larger and larger, those differences multiply to breaking point. That’s something Twitter’s engineers were figuring out in the 2010s.One solution that came up in those years was the Lambda Architecture. A two-pronged approach that recognises the divide between new and old data, and works hard to blend the two together seamlessly in userspace. But that seamless blending is easier said than done. It’s nearly all bespoke work.What if you could get it off the shelf? Let someone else do the work of combining two different kinds of database into one neat package? That's the question of the week as we look at the recently open-sourced project Proton, and its attempt to be the Lambda Architecture in a box…–Proton Docs: https://docs.timeplus.com/protonProton Source: https://github.com/timeplus-io/protonTimeplus: https://www.timeplus.com/Kris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Twitter: https://twitter.com/krisajenkins–#podcast #softwareengineering #databases #dataengineering
Advanced Memory Management in Vale (with Evan Ovadia)
Rust changed the discussion around memory management - this week's guest hopes to push that discussion even further.This week we're joined by Evan Ovadia, creator of the Vale programming language and collector of memory management techniques from far and wide. He takes us through his most important ones, including linear types, generation references and regions, to see what Evan hopes the future of memory management will look like.If you've been interested in Rust's borrow-check and want more (or want different!) then Evan has some big ideas for you to sink your teeth into.–Vale: https://vale.dev/The Vale Discord: https://discord.com/invite/SNB8yGHEvan’s Blog: https://verdagon.dev/homeEvan’s 7DRL Entry: https://verdagon.dev/blog/higher-raii-7drl7DRL: https://7drl.com/https://verdagon.dev/grimoire/grimoireWhat Colour Is Your Function?: https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/42, the language: https://forty2.is/Verona Language: https://www.microsoft.com/en-us/research/project/project-verona/Austral language: https://austral-lang.org/Surely You’re Joking, Mr Feynman! (book): https://www.goodreads.com/book/show/35167685-surely-you-re-joking-mr-feynmanEvan on Twitter: https://twitter.com/verdagonFind Evan in the Vale Discord: https://discord.com/invite/SNB8yGHKris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Twitter: https://twitter.com/krisajenkins–#software #programming #podcast #valelang
Bringing Pure Python to Apache Kafka (with Tomáš Neubauer)
The “big data infrastructure” world is dominated by Java, but the data-analysis world is dominated by Python. So if you need to analyse and process huge amounts of data, chances are you’re in for a less-than-ideal time. The impedance mismatch will probably make your life hard somehow. So there are a lot of projects and companies trying to solve that problem. To bridge those two worlds seamlessly, and many of the popular solutions see SQL as the glue. But this week we’re going to look at another solution - ignore Java, treat Kafka as a protocol, and build up all the infrastructure tools you need with a pure Python library. It’s a lot of work, but in theory it would make Python the one language for data storage, analysis and processing, at scale. Tempting, but is it feasible? Joining me to discuss the pros, cons, and massive scope of that approach is Tomáš Neubauer. He started off doing real time data analysis for the Maclaren’s F1 team, and is now deep in the Python mines effectively rewriting Kafka Streams in Python. But how? How much work is actually involved in porting those ideas to Python-land, and how do you even get started? And perhaps most fundamental of all - even if you succeed, will that be enough to make the job easy, or will you still have to scale the mountain of teaching people how to use the new tools you’ve built? Let's find out.– Quix Streams on Github: https://github.com/quixio/quix-streamsQuix Streams getting started guide: https://quix.io/get-started-with-quix-streamsQuix: https://quix.io/ Tomáš on LinkedIn: https://www.linkedin.com/in/tom%C3%A1%C5%A1-neubauer-a10bb144Tomáš on Twitter: https://twitter.com/TomasNeubauer0Kris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Twitter: https://twitter.com/krisajenkins --#podcast #softwaredevelopment #datascience #apachekafka #streamprocessing
Taking Erlang to OCaml 5 (with Leandro Ostera)
Erlang wears three hats - it’s a language, it’s a platform, and it’s an approach to making software run reliably once it’s in production. Those last two are so interesting I sometimes wonder why those ideas haven’t been ported to every language going. How much work would it be?This week we’re going to dig right down into that question with Leandro Ostera. He’s been working on Riot - a project to bring the best of Erlang’s runtime system and philosophy to OCaml. But why OCaml? Is it possible to marry together OCaml’s type system with Erlang’s dynamic dispatch systems? And what is it about the recent release of OCaml5 that makes the whole project easier?–Leandro’s Blog: https://www.abstractmachines.dev/Why Typing Erlang is Hard: https://www.abstractmachines.dev/posts/am012-why-typing-erlang-is-hard/Riot: https://riot.ml/Riot source: https://github.com/riot-ml/riotReasonML: https://reasonml.github.io/ReScript: https://rescript-lang.org/Leandro on Twitter: https://twitter.com/leosteraKris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Twitter: https://twitter.com/krisajenkins--#podcast #softwaredevelopment #erlang #ocaml #softwaredesign
How Apache Pinot Achieves 200,000 Queries per Second (with Tim Berglund)
The likes of LinkedIn and Uber use Pinot to power some astonishingly high-scale queries against realtime data. The numbers alone would make an impressive case-study. But behind the headline lies a fascinating set of architectural decisions and constraints to get there. So how does Pinot work? How does it process queries? How are the various roles split across a cluster? And equally important - what does it *not* try to achieve.Joining me to go through the nuts and bolts of how Pinot handles SQL queries is Tim Berglund, veteran technology explainer of the realtime-data world. He takes us through Pinot step-by-step, covering the roles of brokers, servers, controllers and minions as we build up the picture of a query engine that's interesting in theory and massively performant in practice.–Apache Pinot: https://pinot.apache.org/Apache Pinot Docs: https://docs.pinot.apache.org/StarTree: https://startree.ai/Event Driven Design episode with Bobby Calderwood: https://youtu.be/V7vhSHqMxusTim on Twitter: https://twitter.com/tlberglundKris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Twitter: https://twitter.com/krisajenkins–#podcast #softwaredevelopment #apachepinot #database #dataengineering #sql
Neovim: Creating, Curating and Customising your Ideal Editor (with TJ DeVries)
TJ DeVries is a core contributor to Neovim and several of its most interesting sub-projects, and he joins us this week to go in depth into how Neovim got started, how it’s structured, and what a truly programmable editor has to offer programmers who want the perfect environment.Along the way we look at what we can learn from Neovim’s successful fork of the 30-year old codebase from Vim, how it still collaborates with the original project, and what putting Lua at the heart of the system has done for casual tinkerers and hardcore plugin writers alike.Not everyone will come away from this discussion wanting to switch editors, but I’m sure you’ll get a newfound appreciation for digging deeper into the developer tools you use everyday.–Neovim: https://neovim.io/Neovim Kickstarter: https://github.com/nvim-lua/kickstart.nvimKickstarter walkthrough video: https://www.youtube.com/watch?v=m8C0Cq9Uv9oA directory of Neovim plugins: https://dotfyle.com/Vimscript’s definition of true and false: https://vimhelp.org/eval.txt.html#BooleanTJ on Twitter: https://twitter.com/teej_dvTJ on Twitch: https://www.twitch.tv/teej_dvTJ on YouTube: https://www.youtube.com/@teej_dvKris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Twitter: https://twitter.com/krisajenkins–#podcast #software #softwareengineering #dx
Creating Hackathons that Work (with Jon Gottfried)
Done right, a Hackathon can be a fantastic place to be a programmer - you get time and space to build and learn, in a room full of like-minded people, with swag and prizes to sweeten the deal. It’s a great way to pick up new ideas and run with them. But done wrong it can be a waste of time. What’s the difference between a good hackathon and a bad one? What do the good ones do right, and what can we learn from that?This week we’re talking about the Joy of Hacks with Major League Hacking Co-Founder Jon Gottfried. He’s got over 10 years of experience building a Hackathon network that provides the right environment for “structured mucking about with computers”, so we’re going to pick his brains.If you’re ever attending a Hackathon, organising one, or looking for a way to build or contribute to your local programming community, Jon can help guide you to events that work.--Major League Hacking: https://mlh.io/Major League Hacking’s 2024 Event Calendar: https://mlh.io/seasons/2024/eventsGames Week: https://events.mlh.io/events/10848 Jon on Mastodon: https://hachyderm.io/@jonmarkgoJon on LinkedIn: https://www.linkedin.com/in/jonmarkgoJon on Twitter: https://twitter.com/jonmarkgoKris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Twitter: https://twitter.com/krisajenkinsBonus link - The Great American Baking Show 2023: https://www.youtube.com/watch?v=IlWLSAKEedk--#software #podcast #programming #hackathon
Automate Your Way to Better Code: Advanced Property Testing (with Oskar Wickström)
One of the most promising techniques for software reliability is property testing. The idea that, instead of writing unit tests we describe some property of our code that ought to always be true, then have the computer figure out thousands of unit tests that try to break that rule.For example, you might say, “No matter which page you visit on my website, there should always be a login button or a logout button.” Then the test’s job is to try to break that rule, but clicking around until it finds some combination of clicks fails that assertion. Like, maybe it finds the 404 page, and you realise it was missing the website’s normal header.At its best, property testing takes far less work than unit testing, but is far more thorough, because it lets us write the rules and has the computer write the examples. The downside is, it often seems theoretical. It can be hard to apply property testing to real-world cases. Let’s fix that.We’re joined by Oskar Wickstrom, who’s been building all kinds of different systems and bringing property testing with him wherever he goes. We discuss the basics of property testing, then he goes into the advanced and cunning techniques that go beyond the ordinary into testing databases, webpages and more. With a bit of thought, he can help us test a ten times as much code with a tenth of the effort.--Oskar’s book, Property Testing a Screencast Editor [ebook]: https://leanpub.com/property-based-testing-in-a-screencast-editorQuickstrom: https://quickstrom.io/F# for Fun & Profit: Property Testing Series: https://fsharpforfunandprofit.com/series/property-based-testing/Linear Temporal Logic: https://en.wikipedia.org/wiki/Linear_temporal_logicThe Quickstrom Paper: https://arxiv.org/abs/2203.11532TodoMVC (One frontend app, many implementations): https://todomvc.com/Oskar on Twitter: https://twitter.com/owickstromKris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Twitter: https://twitter.com/krisajenkins--#softwaredevelopment #podcast #programming #testdrivendevelopment #propertytesting
Bridging the Gap Between Languages (with Martin Johansen)
If you ever feel overwhelmed by the number of different programming languages, this week’s episode might just offer you some solace, as we talk about an attempt to reunify many of the most popular languages by focussing on the bread & butter things that every language supports.I’m joined by Martin Johansen, who’s been working on a new tool called Progsbase. With it, he’s created a spec based on all the things programming languages can agree on, and is building a library that can cross-compile between them. Write a program in Java, and it can be automatically translated to PHP, Python and a great deal more.But how far can he take that idea? Is there really enough unity between these languages to build something universal? How do you bridge the divide between manual memory management languages like C and garbage-collected ones like Java? And what would it actually feel like to write code this way? Let’s put Martin’s plan under the spotlight and find out…–Martin on Twitter: https://twitter.com/martinfjKris on Twitter: https://twitter.com/krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Progsbase homepage: https://www.progsbase.com/The Spec: https://www.progsbase.com/docs/programs/The Progsbase library repository: https://repo.progsbase.com/The Bug Bounty: https://www.progsbase.com/bug-bounty/–#software #programming #podcast #programminglanguages
If You Want Better Code, Do It For Me (with Jonathan Schneider)
A lot of programming is split into the mechanical work of writing what you know, and the creative work of figuring out what you don’t know. Wouldn’t it be nice to automate the mechanical stuff away?Well the good news is we’re already automating a lot of it. Every time you run a refactoring tool or a pretty-printer, you’re handing boring work off to the computer. But how does that magic work, and how can we do more of it?This week we’re joined by one of the authors of OpenRewrite—Jonathan Schneider—to learn how automated code-rewriting tools really work. From the basic approach to the hairy corner cases, to the reality of keeping developers happy with the subjective side of the results.It takes a lot of work to automate work away - this week we’ll learn how the work gets done for us too.–OpenRewrite: https://docs.openrewrite.org/Supported Languages: https://docs.openrewrite.org/recipesModerne: https://www.moderne.io/Gradle Lint: https://github.com/nebula-plugins/gradle-lint-pluginChicory (Native JVM WASM): https://github.com/dylibso/chicoryCall Java from Haskell: https://github.com/tweag/inline-java#readmeCall Haskell from Java: https://github.com/nh2/call-haskell-from-anythingKris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Twitter: https://twitter.com/krisajenkins–#podcast #software #programming #softwareengineering #refactoring #parsers
Implementing Hardware-Friendly Databases (with DuckDB co-creator, Hannes Mühleisen)
SQLite could do with a little competition, so when I invited the co-creator of DuckDB in to talk, I thought we'd be discussing the perils of trying to build a new in-process database engine. I quickly realised things went much deeper than just a tech refresh.Hannes Mühleisen joins me this week to blend his academic credentials as a database researcher with his vehement need to make that research practical. And so we dive into what modern database literature has to say on making queries faster, more parallelizable, and closer to the metal, and how it all comes together in a user-friendly package that’s found its way into my day-to-day workload, and might well help out yours.If you’re curious about the gory details of database queries, how they can take advantage of modern hardware, or how all that research actually turns into a useful tool, Hannes has some great answers.--DuckDB: https://duckdb.org/Database Systems Book: http://infolab.stanford.edu/~ullman/dscb.htmlKris’ first computer: https://en.wikipedia.org/wiki/File:ZX_Spectrum_Plus2_(retouched).jpgVolcano Query Evaluation System [pdf]: https://paperhub.s3.amazonaws.com/dace52a42c07f7f8348b08dc2b186061.pdfMorsel Query Engine [pdf]: https://cs.brown.edu/~kayhan/papers/morsel_cp.pdfUnnesting Arbitrary Queries [pdf]: https://cs.emis.de/LNI/Proceedings/Proceedings241/383.pdfPapers Hannes' team have published: https://duckdb.org/why_duckdb#peer-reviewed-papers-and-thesis-worksDuckDB on Mastodon: https://mastodon.social/@duckdbKris on Twitter: https://twitter.com/krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Mastodon: https://mastodon.social/@krisajenkins--#softwaredevelopment #podcast #programming #database #duckdb #sql #sqlite
Verse, Haskell & Core Language Design (with Simon Peyton Jones)
This week we talk to Simon Peyton Jones, a veteran language designer and researcher, and key figure in the development of Haskell. Haskell. Simon has made countless contributions to advancement of functional programming, and computer programming in general, and is currently working at Epic Games, working on the foundations of their new programming language, Verse.We discuss how programming languages are made, focussing on a big design idea from both Haskell and Verse: building a large language from a small, tightly designed core. Then we move into Simon's current work exploring Functional Logic Programming, the big new idea that underpins Verse. It's an idea that blends the fundamentals FP with the core ideas of logic languages like Prolog in an entirely new way. Not even Simon knows exactly where the idea will lead, but it's a fascinating idea that could potentially bring constraint-solving and deduction right into the heart of modern software.Additionally, Simon discusses his involvement in reshaping the way we teach computing in England. He's been working hard to give computing education the same importance as the teaching of mathematics and sciences - something we should all have a fundamental understanding of.Simon's one of the smartest, nicest people in programming. Come as hear his brilliant brain at work. :-D–Verse: https://github.com/UnrealVerseGuru/VerseProgrammingLanguageThe Verse Language Reference: https://dev.epicgames.com/documentation/en-us/uefn/verse-language-referenceThe Verse Calculus [pdf]: https://simon.peytonjones.org/assets/pdfs/verse-March23.pdfhttps://en.wikipedia.org/wiki/Simon_Peyton_JonesThe LogicT monad: https://hackage.haskell.org/package/logictCan programming be liberated from the von Neumann style?: https://dl.acm.org/doi/10.1145/359576.359579CAS - Computing At School: https://www.computingatschool.org.uk/Computer Science Teachers Association: https://csteachers.org/Kris on Twitter: https://twitter.com/krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/
Shouldn't Data Connections Be Easier? (with Ashley Jeffs)
Benthos wants to be part of your Data Engineering toolkit - it’s there as a quick and easy way to set up data pipelines and start streaming data out of A and into B. In contrast to a lot of the tools we’ve talked about on Developer Voices, Benthos seems focussed on cutting development time down to a minimum, so you can quickly configure a new pipeline and test it out, without making a whole sprint of the task. As quick as a quick-and-dirty shell script, without the dirt. 😉So this week we’re talking to the creator of Benthos, Ashley Jeffs, to hear why he created Benthos, what it can do for you, and what its strengths and weaknesses are. And Jeff’s refreshingly candid about when you should and shouldn’t use it. If you ever need to get data from an HTTP connection into S3, or S3 into Kafka, or Kafka into a flat file, Benthos might just save you a few hours of development.–Benthos: https://www.benthos.dev/A list of supported inputs, processors & outputs: https://www.benthos.dev/docs/about#componentsAll their cute blobfish logos: https://www.benthos.dev/blobfish/IDML: https://en.wikipedia.org/wiki/International_Development_Markup_LanguageKris on Twitter: https://twitter.com/krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/–#software #podcast #dataengineering #datascience
What can game programming teach us about databases? (with Tyler Cloutier)
The world of game programming might seem a million miles away from regular programming. But they still have to deal with the same kinds of data, scale and concurrency problems that we’re all familiar with in the software world. What makes games interesting, is that under the hood they’re solving those same problems, often with some novel ideas about the solutions.So this week we’re off to the massive open world that is game development, to see what we can learn that might make our programming lives easier in the non-gaming space. Joining us for that is Tyler Cloutier, the founder of Clockwork Labs. They’re building SpaceTimeDB, a curiously-distributed database to be the platform underlying their new MMORPG BitCraft. In digging down into the architecture of SpaceTimeDB we pick Tyler’s brain for nuggets of information on event sourcing, request/response vs. subscriptions, transactions, security and much more. All in an effort to make the programmers and data scientists’ lives easier.--SpaceTimeDB: https://spacetimedb.com/BitCraft: https://bitcraftonline.com/“4X games” defined: https://en.wikipedia.org/wiki/4XPlan 9 O.S.: https://en.wikipedia.org/wiki/Plan_9_from_Bell_LabsTyler on LinkedIn: https://www.linkedin.com/in/tylercloutier/Kris on Twitter: https://twitter.com/krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Mastodon: https://mastodon.social/@krisajenkins
Is Odin, "programming done right"? (with 'Ginger' Bill Hall)
Odin’s creator, Bill Hall, makes some bold claims about the language, including that it’s “programming done right”. Before that starts a war on the internet, we’d best ask him to explain what that means, and how Odin tries to achieve it. And while we get deep into the details, overall his answer seems to be, “By gathering masses of feedback and then refining C until it feels joyous again.Of all the C-like languages we’ve looked at on Developer Voices, Odin seems to be the most at-ease with its progenitor. It’s not trying to be a revolutionary new way of thinking about systems programming; it’s just trying to rethink C for modern conventions. If Bill’s hit his goals, it might be the most comfortable way to get a language that’s C, but C done better…–Odin: https://odin-lang.org/Odin Packages: https://pkg.odin-lang.org/Newsqueak [pdf]: https://swtch.com/~rsc/thread/newsqueak.pdfEmberGen: https://jangafx.com/software/embergen/Raylib: https://www.raylib.com/RayLib bindings for Odin: https://github.com/odin-lang/Odin/tree/master/vendor/raylibVerse language: https://dev.epicgames.com/documentation/en-us/uefn/verse-language-referenceAlgorithms + Data Structures = Programs: https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures_%3D_ProgramsBill on Twitter: https://twitter.com/TheGingerBillKris on Twitter: https://twitter.com/krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/--#podcast #software #softwareprogramming #programming #odin #odinlang