Python Bytes is a weekly podcast hosted by Michael Kennedy and Brian Okken. The show is a short discussion on the headlines and noteworthy news in the Python, developer, and data science space.
#336 We found one of your batteries
Watch on YouTube About the show Sponsored by InfluxDB from Influxdata. Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Brian #1: Python's Missing Batteries: Essential Libraries You're Missing Out On Martin Heinz Fun collection of a bunch of libraries you may not know about (or forgot about), with code examples. Utilities boltons : iterate through json and dates, quickly grab data out of nested structures, and convert nested data with jsonutils, timeutils, and iterutils sh : conveniently call shell funcitons Data Validation validators : validate email addresses, credit cars, IP addresses, and more. the fuzz : fuzzy string comparisons Debugging stackprinter : nice stack traces with exception messages higlighted Testing freezegun : stop time, change dates, … dirty_equals : comparing things that are kinda equal CLI tqdm : add a progress bar to command line apps Michael #2: awesome-polars A curated list of Polars talks, tools, examples & articles. Mostly articles and tutorials however. Brian #3: Running Headless Selenium in Python (2023) Siddiqi First off, if you are doing automated testing with Selenium, I hope you already know about headless. It’s awesome and speeds up testing. Next, there’s changes to how you code headless, as of Selenium 4.8.0 (Jan. 2023). Old: options.headless` `**=**` `True New: options.add_argument('--headless=new') for Chrome options.add_argument('--headless') for Firefox Reasons: Read Headless is Going Away! post on Selenium blog. Subtitle: “Now that we got your attention, headless is not actually going away, just the convenience method to set it in Selenium” Michael #4: Gracy Gracy helps you handle failures, logging, retries, throttling, and tracking for all your HTTP interactions. Has support for Parsing per status code Throttling Retries Custom validation Record/replay for testing A bit non-pythonic but perhaps inspriation for some out there Extras Michael: Mobile apps are finally out Take the git course for free for a limited time. Michael's blog post announcing the apps Joke: It’s practice
#335 Should you get your mojo on?
Watch on YouTube About the show Sponsored by InfluxDB from Influxdata. Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Michael #1: Introducing 'Trusted Publishers’ PyPI package maintainers can adopt a new, more secure publishing method that does not require long-lived passwords or API tokens to be shared with external systems. Our term for using the OpenID Connect (OIDC) standard to exchange short-lived identity tokens between a trusted third-party service and PyPI. Instead, PyPI maintainers can configure PyPI to trust an identity provided by a given OpenID Connect Identity Provider (IdP). These API tokens never need to be stored or shared rotate automatically by expiring quickly provide a verifiable link between a published package and its source Additional security hardening is available Brian #2: Mojo : a new programming language for all AI developers. Mojo may be the biggest programming language advance in decades - fast.ai blog Suggested by many listeners “Mojo combines the usability of Python with the performance of C, unlocking unparalleled programmability of AI hardware and extensibility of AI models.” A programming language compatible with Python, with performance similar to C++/Rust. “Mojo is designed to become a superset of Python over time by preserving Python’s dynamic features while adding new primitives for systems programming.” - emphasis from Brian It’s not there yet, but still super cool Built on a MLIR, not LLVM “How compatible is Mojo with Python really? Mojo already supports many core features of Python including async/await, error handling, variadics, etc, but… it is still very early and missing many features - so today it isn’t very compatible. Mojo doesn’t even support classes yet!” Michael #3: django-prose Wonderful rich-text editing for your Django project. Rendering rich-text in templates Small rich-text content (as model fields) Django Prose is using Bleach to only allow certain tags and attributes See the website for a screenshot of it in action Brian #4: pylyzer is a static code analyzer / language server for Python, written in Rust. Shunsuke Shibayama Suggested by Owen Features fast detailed analysis type checking plus things like out-of-bounds accesses to lists, and non-existent key references to dicts more readable reports and a VS Code extension pylyzer vs ruff “Ruff, like pylyzer, is a static code analysis tool for Python written in Rust, but Ruff is a linter and pylyzer is a type checker & language server. pylyzer does not perform linting, and Ruff does not perform type checking.” Some limitations and incomplete “todo list”. See README for more details. Joke: Escape Room
#334 Packaging Organizations
Watch on YouTube About the show Sponsored by us! Support our work through: Our courses at Talk Python Training Test & Code Podcast Patreon Supporters Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Brian #1: rye - Python workflow tool Armin Ronacher Rust built tool, currently only for Linux and MacOS Project workflow commands, like init - new project add - add a dependency (including optional) remove - remove a dependency build - build wheel lock - update lock file virtualenv commands add —dev - install in environment sync - install/update dependencies in env based on pyprojec.toml run - run command within environment Install Python fetch - Fetches a Python interpreter for the local machine Register existing Python toolchain Helper utility to manage Python toolchains Kinda like pipx install - Installs a package as global tool uninstall - Uninstalls a global tool I didn’t see that it added anything to my PATH, so this addition made it work: Bonus Everything lives under ~/.rye So it’s easy to stop using, and doesn’t muck up see also Simon Willison’s A few notes on Rye Python Bytes #332, where we talked about huak Michael #2: PyPI Organizations The first step in our plan to build financial support and long-term sustainability of the Python Packaging Index (PyPI) Small fee for organizations rather than individual users Like Github orgs Brian #3: 5 tips to learn any new Python library faster Bob Belderbos The tiips RTFM - at lest the getting started docs Install it Explore the library - play. Bob recommends Jupyter notebook for this. Apply it to a real world problem - deliberate practice Build something with it (bonus) Teach it - blog, TIL, video tutorial, etc. Michael #4: Python gets down to (the) Metal Extras Brian: frogmouth - Markdown viewer / browser for your terminal, built with Textual. Michael: Was going to talk about Serenade, but seems to have gone silent. Packaging follow up discussion. Joke: It’s the progress that counts
#333 Live From PyCon
Watch on YouTube Sponsored by us! Support our work through: Our courses at Talk Python Training Test & Code Podcast Patreon Supporters Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.orgx Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Michael #1: Introducing Microsoft Security Copilot Security Copilot combines this advanced large language model (LLM) with a security-specific model from Microsoft. When Security Copilot receives a prompt from a security professional, it uses the full power of the security-specific model to deploy skills and queries that maximize the value of the latest large language model capabilities. Your data and stays within your control. It is not used to train the foundation AI models, and in fact, it is protected by the most comprehensive enterprise compliance and security controls. Brian #2: PEP 695 – Type Parameter Syntax “This PEP specifies an improved syntax for specifying type parameters within a generic class, function, or type alias. It also introduces a new statement for declaring type aliases.” To get a feel for this, jump to the examples One example Here is an example of a generic function today. from typing import TypeVar _T = TypeVar("_T") def func(a: _T, b: _T) -> _T: ... - And the new syntax. def func[T](a: T, b: T) -> T: ... Michael #3: Auto-GPT An experimental open-source attempt to make GPT-4 fully autonomous. This program, driven by GPT-4, chains together LLM "thoughts", to autonomously achieve whatever goal you set. Features 🌐 Internet access for searches and information gathering 💾 Long-term and short-term memory management 🧠 GPT-4 instances for text generation 🔗 Access to popular websites and platforms 🗃️ File storage and summarization with GPT-3.5 Brian #4: Astral : Ruff is now a company Charlie Marsh announces Astral, starting off with a healthy $4m in seed money. Not a bad start Astral will continue building high-performance developer tools for the Python ecosystem — to keep building Ruff, and to build more Ruff-like things. “to make the Python ecosystem more productive by building high-performance developer tools.” Undoubtedly Rust-ifying more of the Python tool chain. Related: In a discussion at PyCon, someone asked me if Ruff was a replacement for Black. My answer, “not really, more of a flake8 replacement and a few other tools, but a partial overlap with Black. Real answer is at the FAQ Is Ruff compatible with Black? Yes. Ruff is compatible with Black out-of-the-box, as long as the line-length setting is consistent between the two. As a project, Ruff is designed to be used alongside Black and, as such, will defer implementing stylistic lint rules that are obviated by autoformatting.” Extras Brian: Registration open for SciPy 2023 Michael: Android Mobile App is out. Please give us feedback Joke: Releasing to prod
#332 A Python, a Slurpee, and Some Chaos
Watch on YouTube About the show Sponsored by InfluxDB from Influxdata. Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Brian #1: huak - A Python package manager written in Rust. Inspired by Cargo Suggested by Owen Tons of workflows activate - activate a virtual environment add add a dependency to a project pip install it into your virtual environment, and add it to the dependency list in pyproject.toml test - run pytest update update dependencies lint - run ruff, installing it first if necessary fix - autofix fixable lint conflicts build - build wheel in isolated virtual environment using hatchling Honestly I was considering building my own workflow tool, but this is darned close to what I want. Even though it’s still “in an experimental state”. There are rough edges (ruff edges, get it), but still, way cool. I just don’t know how to pronounce it. Is it like “walk”, or more like “whack”? Michael #2: PSF expresses concerns about a proposed EU law that may make it impossible to continue providing Python and PyPI to the European public After reviewing the proposed Cyber Resilience Act and Product Liability Act, the PSF has found issues that put the mission of our organization and the health of the open-source software community at risk. As currently written, the authors of open-source components might bear legal and financial responsibility for the way their components are applied in someone else’s commercial product. The risk of huge potential costs would make it impossible in practice for us to continue to provide Python and PyPI to the European public. Brian #3: ChaosToolkit Suggested by the maintainer, Sylvain Hellegouarch Declare and store your Chaos Engineering experiments as JSON/YAML files so you can collaborate and orchestrate them as any other piece of code. Extensible through an Open API Can be automated in CI/CD pipeline Michael #4: PEP 711 – PyBI: a standard format for distributing Python Binaries “Like wheels, but instead of a pre-built python package, it’s a pre-built python interpreter” Joke: It’s the effort that counts
#331 Python From the Future
Watch on YouTube About the show Sponsored by InfluxDB from Influxdata. Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Michael #1: makeapp via Felix Ingram Simplifies Python application rollout and publishing. Link to its mention on Talk Python. Simplifies Python application rollout and publishing: Make a skeleton for your new application with one console command Automatically create a VCS repository for your application. Automatically check whether the chosen application name is not already in use. Customize new application layouts with skeleton templates. Put some skeleton default settings into a configuration file not to mess with command line switches anymore. Easily add entries to your changelog. Publish your application to remotes (VCS, PyPI) with single command. Brian #2: Looking forward to Python 3.12 We’re on 3.12.0a7 now, the last alpha, final is scheduled for October schedule So far, in 3.12.0a7 What’s new in Python 3.12 page has some examples of the Improved Error Messages Recent addition, PEP 684 - A Per-Interpreter GIL was approved recently “… sufficient isolation would facilitate true multi-core parallelism …” seems like a good thing. But also, “… this is an advanced feature meant for a narrow set of users of the C-API. “, so not really sure how this will affect us. Still, seems cool. Michael #3: Python 3.11.3 is out Fixes a HIGH level CVE in OpenSSL (so patch it) Lots of changes in Core and Builtins Brian #4: How to Make a Great Conference Talk Sebastian Witowski Lots of great advice for tech conf talks. Don’t skip the last half of this, getting your talk accepted is really when the work starts. Good sections to make sure you don’t miss Live demos “First of all - do you really need a demo? …” Rehearsing Don’t skip this. Do this. A lot. Out loud. With a timer. While standing. Memorize the first few minutes, and the last few. Know how you’re going to open and close. Night before get enough sleep Day of eat well. Don’t drink too much liquids. Be comfortable. Sebastian was honest in saying this stuff works for him, but do what works for you. From Brian: I deviate from Sebastian in quite a few places, but still don’t disagree with his advice. I can’t give a talk without slides, as I use them for prompts to know what I’m talking about next. My talks usually have a lot of code snippets. Obviously, that would be difficult without slides. I write my talk and my slides in Markdown. Sebastian writes in something else, then builds slides as visual aids. That’s cool. Do what works for you. Bonus tool from the article: demo-magic - If I’m ever tempted to live code again, I think I’ll try this instead. Extras Michael: NOW the CDN course is out. Django 4.2 released. Joke: Using A.I. for Efficiency
#330 Your data, validated 5x-50x faster, coming soon
Watch on YouTube About the show Sponsored by Influxdb Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Brian #1: Pydantic V2 Pre Release Terrence Dorsey & Samuel Colvin Alpha release available to everyone: pip install --pre -U "pydantic>=2.0a1" Headlines: pydantic-core - all validation logic rewritten in Rust and moved to separate package, pytest-core 5-50x faster separation will aid safety and maintainability Lots ready for experimentation BaseModel, Dataclasses, Serialization, … Much still under construction Docs, BaseSettings→ pydantic-settings, … Michael #2: microdot The impossibly small web framework for Python and MicroPython Microdot is a minimalistic Python web framework inspired by Flask, and designed to run on systems with limited resources such as microcontrollers. It runs on standard Python and on MicroPython. Support for async, websockets, tls, even ASGI servers. Less mem usage by a big margin. Brian #3: GitHub Actions Tools: watchgha, build and inspect, and pytest annotate failures watchgha Ned Batchelder Watch GH Actions progress on the command line build-and-inspect-python-package Hynek Test the build of wheels, check contents, lint README print sdist contents, wheel contents, and metadata pytest-github-actions-annotate-failures utgwkk Nice traceback annotations for pytest Michael #4: PEP 709 – Inlined comprehensions by Carl Meyer Comprehensions are currently compiled as nested functions, which provides isolation of the comprehension’s iteration variable, but is inefficient at runtime. This PEP proposes to inline list, dictionary, and set comprehensions into the code where they are defined, and provide the expected isolation by pushing/popping clashing locals on the stack. This change makes comprehensions much faster: up to 2x faster for a microbenchmark of a comprehension alone. Extras Michael: Python Web Apps that Fly with CDNs Course Joke: Can’t watch movies
#329 Creating very old Python code
Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Michael #1: Prefix-cache via Brendan Hannigan You can set an environment variable or use it as a command line argument and then instead of creating tons of __pycache__ folders to store your *.pyc files right next to the source code, it puts them in some specified folder. Introduced in python 3.8. Brian #2: NiceGUI Suggested by several listeners Browser based GUI “NiceGUI is an easy-to-use, Python-based UI framework, which shows up in your web browser. You can create buttons, dialogs, Markdown, 3D scenes, plots and much more. It is great for micro web apps, dashboards, robotics projects, smart home solutions and similar use cases. You can also use it in development, for example when tweaking/configuring a machine learning algorithm or tuning motor controllers.” - from the README Michael #3: flask-ngrok A simple way to demo Flask apps from your machine. Makes your Flask apps running on localhost available over the internet via ngrok. Great for testing API consumers too. app = Flask(__name__) run_with_ngrok(app) # Start ngrok when app is run # Endpoints ... if __name__ == '__main__': app.run() Brian #4: No-async async with Python Will McGugan Allowing async while not requiring async Await me (maybe) borrowed from Simon Willison’s The “await me maybe” pattern for Python asyncio Optionally awaitable Providing API methods that can be called by both async and non-async code. The called method really is async, but if a caller doesn’t want to know when the code is done, it can ignore the return value and not await. MK: I had to solve a similar problem in fastapi-chameleon MK: Syncify async functions. Extras: Brian: PyPI has a blog Docker no longer sunsetting free team plan Jokes: Long-lived software Mysteries make life more interesting last paragraph, discussing the cov fixture of pytest-cov
#328 We are going to need some context here
Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Brian #1: zipapp Part of standard library since 3.5 Yet another thing I learned recently from Brett Cannon “This module provides tools to manage the creation of zip files containing Python code, which can be executed directly by the Python interpreter. The module provides both a Command-Line Interface and a Python API.” Including: Creating Standalone Applications with zipapp Michael #2: Reverse engineering the Apple News app with #python and #nerd power As we navigate the digital world, we often come across articles we don't have time to read but still want to save for later. One way to accomplish this is by using the Read Later feature in Apple News. But what if you want to access those articles outside the Apple News app, such as on a different device or with someone who doesn't use Apple News? Or what if you want to automatically post links to those articles on your blog? That's where the nerd powers come in. The linked article shows how to use Python to solve your own problem Leading to Rhet Turnbull’s CLI: apple-news-to-sqlite Brian #3: What is a context manager? Trey Hunner Also look at all the cool goodies in contextlib from standard library @contextmanager closing suppress redirect_stdout, redirect_stderr chdir Michael #4: nox-poetry: Use Poetry inside Nox sessions via 2 people: John Hagen and Marc Prewitt This package provides a drop-in replacement for the nox.session decorator, and for the nox.Session object passed to user-defined session functions. Comes from Claudio Jolowicz's hypermodern python cookiecutter Covered this on Talk Python: talkpython.fm/episodes/show/362/hypermodern-python-projects This session performs the following steps: Build a wheel from the local package. Install the wheel as well as the pytest package. Invoke pytest to run the test suite against the installation. Consider what would happen in this session if we had imported @session from nox instead of nox_poetry: Package dependencies would only be constrained by the wheel metadata, not by the lock file. In other words, their versions would not be pinned. The pytest dependency would not be constrained at all. Poetry would be installed as a build backend every time. Extras Brian: Sharing is Caring: Sharing pytest fixtures talk availabe at about 2:40:58 on Day 2 video of PyCascades 2023. Also full Day 1 and Day 2 Michael: Wired connection to remote mesh router == wow! Using the Linksys Atlas Max 6E Joke: UnsafeWarnings
#327 Untangling XML with Pydantic
Watch on YouTube About the show Sponsored by Compiler Podcast from Red Hat. Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Michael #1: pydantic-xml extension via Ilan Recall untangle. How about some pydantic in the mix? pydantic-xml is a pydantic extension providing model fields xml binding and xml serialization / deserialization. It is closely integrated with pydantic which means it supports most of its features. Brian #2: How virtual environments work Brett Cannon This should be required reading for anyone learning Python. Maybe right after “Hello World” and right before “My first pytest test”, approximately. Some history of environments Back in the day, there was global and your directory. How environments work structure: bin, include, and lib pyvenv.cfg configuration file How Python uses virtual environments What activation does, and that it’s optional. Yes, activation is optional. A new project called microvenv that helps VS Code. Mostly to fix the “Debian doesn’t ship python3 with venv” problem. It doesn’t include script activation stuff It’s super small, less than 100 lines of code, in one file. Michael #3: DbDeclare Declarative layer for your database. https://raaidarshad.github.io/dbdeclare/guide/controller/#example Sent in by creator raaid DbDeclare is a Python package that helps you create and manage entities in your database cluster, like databases, roles, access control, and (eventually) more. It aims to fill the gap between SQLAlchemy (SQLA) and infrastructure as code (IaC). You can: Declare desired state in Python Avoid maintaining raw SQL Tightly integrate your databases, roles, access control, and more with your tables Migrations like alembic coming too. Brian #4: Testing multiple Python versions with nox and pyenv Seth Michael Larson This is a cool “what to do first” with nox. Specifically, how to use it to run pytest against your project on multiple versions of Python. Example noxfile.py is super small import nox @nox.session(python=["3.8", "3.9", "3.10", "3.11", "3.12", "pypy3"]) def test(session): session.install(".") session.install("-rdev-requirements.txt") session.run("pytest", "tests/") How to run everything, nox or nox -s test. How to run single sessions, nox -s test-311 for just Python 3.11 Also how to get this to work with pyenv. pyenv global 3.8 3.9 3.10 3.11 3.12-dev This reminds me that I keep meaning to write a workflow comparison post about nox and tox. Extras Michael: GitHub makes 2FA mandatory next week for active developers New adventure bike [image 1, image 2]. Who’s got good ideas for where to ride in the PNW? Wondering why I got it, here’s a fun video. Joke: Case of the Mondays
#326 Let's Go for a PyGWalk
Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Brian #1: Data Classification : Does Python still have a need for class without @dataclass? Glyph dataclasses have been in the the language since 3.7 That’s pretty much all modern Python, right? “…, is there any point to having non-@dataclass classes any more? Is there any remaining justification for writing them in new code?” Options: class just becomes a dataclass if you have typehinted members in it. data instead of class, to avoid decorators Michael #2: PyGWalker Turn your pandas dataframe into a Tableau-style User Interface for visual analysis. Works with pandas and polars Open-source alternative to Tableau It allows data scientists to analyze data and visualize patterns with simple drag-and-drop operations. Brian #3: An opinionated Python boilerplate Duarte O.Carmo Tools and processes for new projects pip-tools - Pip-tools strikes the right balance between simplicity, effectiveness, and speed. especially for generating pinned requirements.txt files, if necessary pyproject.toml - for configuration. packaging, but also any tool that supports it. ruff black, isort no pre-commit hooks, just run it in CI Michael #4: Front Matter VS Code via Mark Little If you have content that supports frontmatter and is markdown-based, check this out. Stay in your editor and easily create, manage, and publish content. Don’t make front matter mistakes When was it published? What is the timezone text formatting again? Learn new features of your existing static site (e.g. article image) Manage images and more. Extras Brian: VSCode improves IntelliSense support for pytest in Feb release Michael: AI search wars get weird Proton Drive is Out of Beta, Available for Everyone Joke: Is your computer on? Is it on fire?
#325 It's called a merge conflict
Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Michael #1: Python Parquet and Arrow: Using PyArrow With Pandas Parquet is an efficient, compressed, column-oriented storage format for arrays and tables of data. Less wrangle-able than Pandas, but way faster and lower memory Questions answered Can we use Pandas DataFrames and Arrow tables together, and if so, how is this done? (It turns out the answer is yes, and it’s quite simple, as we’ll see). In what ways are Arrow tables “better” than Pandas DataFrames? In other words, for which tasks are Arrow tables better suited? Conversely, what tasks are possible or easy in Pandas that are difficult or impossible in Arrow? As an on-disk format, how does Parquet compare to popular alternatives such as feather, orc, CSV, etc.? Brian #2: FastAPI-Filter Arthur Rio Add query string filters to your api endpoints and show them in the swagger UI. The supported backends are SQLAlchemy and MongoEngine. FastAPI-Filter documentation The philosophy of fastapi_filter is to be very declarative. You define the fields you want to be able to filter on as well as the type of operator, then tie your filter to a specific model. default filters: neq, gt, gte, in, isnull, lt, lte, not/ne, not_in, nin, like/ilike The swagger support is actually quite cool. Michael #3: 12 Python Decorators to Take Your Code to the Next Level Decorators are awesome This is mostly home-grown decorators, but some standard ones too Notable ones: @warps @lru_cache @repeat @timeit @retry ← no please use tenacity @countcall @rate_limited @dataclass @register @property @singledispatch Brian #4: PyHamcrest Contributed by Txels PyHamcrest is a framework for writing matcher objects, allowing you to declaratively define “match” rules. PyHamcrest tutorial Having a tool that allows you to pick out precisely the aspect under test and describe the values it should have, to a controlled level of precision, helps greatly in writing tests that are “just right.” From Brian: I’ve been reluctant to try matcher style assertion helper libraries, as, with pytest, assert works just fine. However, I can see cases where PyHamcrest assertions could help test readability, and that’s always a win. Examples: equality: assert_that(theBiscuit, equal_to(myBiscuit)) exceptions: assert_that(calling(parse, bad_data), raises(ValueError)) async: assert_that(``**await** resolved(future), future_raising(ValueError)) boolean: assert_that(theBiscuit.isCooked()) There’s predefined matchers for objects, numbers, text, logical checks, dequences, dictionaries Extras Brian: pytest tips and tricks - recent post, and discussion on upcoming Talk Python episode sharing pytest fixtures - placeholder page where I’ll share slides and code after my talk. Michael: Python runtime updates Django 4.2 beta 1 released Joke: A group of developers is called …
#324 JSON in My DB?
Watch on YouTube About the show Sponsored by Compiler Podcast from Red Hat. Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Special guest, Erin Mullaney: @erinrachel@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Brian #1: Use TOML for .env files? Brett Cannon .env files are used to store default settings that can be overridden by environmental variables. Possibly brought on by twelve-factor app design. Supported by python-dotenv, which is also used by pydantic, pipenv, and others. One issue is that it’s not a defined standard. from python-dotenv docs “The format is not formally specified and still improves over time. That being said, .env files should mostly look like Bash files.” Adafruit decided that an upcoming CircuitPython will use TOML as the format for settings.toml files, which are to be used mostly how .env files are being used. Brett notices this may fix things for Python for VS Code, and other people as well. So… Is this a good idea? I think so. Michael #2: Pydantic gets serious funding via Mark Little (was on episode 285) Sequoia backs open source data-validation framework Pydantic to commercialize with cloud services. Pydantic Services Inc. emerges from stealth today with $4.7 million in seed funding. Pydantic’s new commercial entity will incorporate a swath of new tools and services that are both “powered-by and inspired-by the Pydantic library” Pydantic will start with an initial team of six, with the first three engineers based in Montana, Chicago and Berlin. “With $4.7 million in the bank, Colvin said that they’re continuing to rewrite parts of Pydantic in Rust, with a view toward making it more efficient via a ten-fold performance improvement.” Erin #3: JSON Fields for performance (Denormalization) David Stokes Using JSON fields when you design your databases is a good way to improve database query performance. Brian #4: f-strings with pandas and Jupyter keyboard shortcuts Kevin Markham After a couple year break from blogging, friend of the show Kevin Markham has a couple great, short, useful posts. How to use Python's f-strings with pandas My favorite bit is the part about using f-strings for dictionary keys Fly through Jupyter with keyboard shortcuts 🚀 I’m a sucker for a rocket emoji Not an overwhelming list. Just the essentials for even the casual Jupyter user. Examples Esc and Enter for command mode/edit mode a and b for creating a new cell above or below current cell. m and y for changing the cell type to Markdown or code. Shift+m to merge cells so many more - Michael #5: BioGPT “GPT” for biomedical text generation and mining As motivation, let’s see what ChatGPT can do with arrow anti-patterns in Python. Smaller models and “Large” models Used via an API rather than chat style. BioGPT has also been integrated into the Hugging Face transformers library too Play with it here. Erin #6: Code Mentorship and Communicating with Newer Devs Sheena O’Connell Sheena O’Connell gave a talk at DjangoCon about her work at Umuzi, training unemployed young people in underserved communities in Africa and also was on Django Chat Podcast. Dmitriy Chukhin Caktus Group is trying a new mentorship program for folks who don’t have the necessary training. Extras: Michael: News is, these are no loner news: Security Researchers Uncover 700+ Malicious Open-Source Packages in npm and PyPI Git security vulnerabilities announced, again git ignores https://github.com/github/gitignore https://gitignore.io Erin: DjangoCon is in October in Durham, NC this year (Oct 15-20) Joke: Remember your pointers?
#323 AI search wars have begun
Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org - may be a minute or two late. Show: @pythonbytes@fosstodon.org Special guest: Pamela Fox - @pamelafox@fosstodon.org Join us on YouTube at pythonbytes.fm/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Michael #1: camply A tool to find campsites at sold out campgrounds through sites like recreation.gov and Yellowstone Finding reservations at sold out campgrounds can be tough. Searches the APIs of booking services like recreation.gov (which indexes thousands of campgrounds across the USA) to continuously check for cancellations and availabilities to pop up. Once a campsite becomes available, camply sends you a notification to book your spot! Want to camp in a tower in California? camply campgrounds --search "Fire Lookout Towers" --state CA Brian #2: hatch-fancy-pypi-readme Your ✨Fancy✨ Project Deserves a ✨Fancy✨ PyPI Readme! 🧐 Hynek Schlawack Include lots of extras in a README.md text fragments files, like AUTHORS.md or Changelog.md, with custom start, stop, pattern includes, etc. regular expression substitutions Several projects with examples, including black. Pamela #3: Pyodide dev branch now supports 3.11 Python 3.11 PR Benchmark Py3.11 and Py3.10 pyodide console TODO list for 0.23.0 alpha release Dis-this: specializing adaptive interpreter Recursion visualizer Michael #4: EU hates open source? via Pamphile Roy The Cyber Resilience Act (CRA) is an interesting and important proposal for a European law that aims to drive the safety and integrity of software The proposal includes a requirement for self-certification by suppliers of software to attest conformity with the requirements of the CRA including security, privacy and the absence of Critical Vulnerability Events (CVEs). We recognize that the European Commission has framed an exception in recital 10 attempting to ensure these provisions do not accidentally impact Open Source software. However, drawing on more than two decades of experience, we at the Open Source Initiative can clearly see that the current text will cause extensive problems for Open Source software. Since the goal is to avoid harming Open Source software this goal should be stated at the start of the paragraph as the rationale, replacing the introductory wording about avoiding harm to "research and innovation" to avoid over-narrowing the exception. The reference to "non-commercial" as a qualifier should be substituted. The term “commercial” has always led to legal uncertainty for software and is a term which should not be applied in the context of open source OSI recommends further work on the Open Source exception to the requirements within the body of the Act to exclude all activities prior to commercial deployment of the software and to clearly ensure that responsibility for CE marks does not rest with any actor who is not a direct commercial beneficiary of deployment. Brian #5: So, Single (‘) or Double (“) Quotes in Python? Marcin Kozak PEP8 doesn’t recommend anything. REPL uses single quotes. >>> x = "one" >>> x 'one' Black sides with “double quotes”, due to the apostrophe in the string problem. 'Don\'t be so sad.' vs “Don’t be sad.” You get to pick, and don’t be bullied by black-fanatics. There’s always blue, which is just like black, but defaults to single-quotes line length defaults to 79, not black’s 88. preserves whitespace before hash marks for right hanging comments (so multiple lines can line up). Pamela #6: Frozen-Flask Pamela’s PR for moving to Frozen Flask Stepping down as a maintainer Extras Brian: What does everyone think of GitHub pricing? Michael: Much much better transcripts, for example, this episode. Means our search works way better too The AI search wars have begun - Google Panics Over ChatGPT [The AI Wars Have Begun] video Microsoft Bing rockets to the top of the App Store after announcing ChatGPT integration Google shares lose $100 billion after company's AI chatbot makes an error during demo Free PyCharm for all the Talk Python customers Thanks for the help with finding a good Flutter dev. Important Talk Python episode: Fusion Ignition Breakthrough and Python Pamela: Github pyproject.toml support. Python Package Template Jokes: $McTitle Worst input fields
#322 Python Packages, Let Me Count The Ways
Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Connect with the hosts Michael: @mkennedy@fosstodon.org Brian: @brianokken@fosstodon.org Show: @pythonbytes@fosstodon.org Special guest: @calvinhp@fosstodon.org Join us on YouTube at pythonbytes.fm/stream/live to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too. Brian #1: Packaging Python Projects Tutorial from PyPA This is a really good starting point to understand how to share Python code through packaging. Includes discussion of directory layout creating package files, LICENSE, pyproject.toml, README.md, tests and src dir how to fill out build-system section of pyproject.toml using either hatchling, setuptools, flit, or pdm as backends metadata using build to generate wheels and tarballs uploading with twine However For small-ish pure Python projects, I still prefer flit flit init creates pyproject.toml and LICENSE will probably still need to hand tweak pyproject.toml flit build replaces build flit publish replaces twine The process can be confusing, even for seasoned professionals. Further discussion later in the show Michael #2: untangle xml Convert XML to Python objects Children can be accessed with parent.child, attributes with element['attribute']. Call the parse() method with a filename, an URL or an XML string. Given this XML: [HTML_REMOVED] [HTML_REMOVED] [HTML_REMOVED] [HTML_REMOVED] Access the document: obj.root.child['name'] # u'child1' A little cleaner that ElementTree perhaps. Calvin #3: Mypy 1.0 Released Mypy is a static type checker for Python, basically a Python linter on steroids Started in 2012 and developed by a team at Dropbox lead by https://github.com/JukkaL What’s New? New Release Numbering Scheme not using symver Significant backward incompatible changes will be announced in the blog post for the previous feature release feature flags will allow users to upgrade and turn on the new behavior Mypy 1.0 is 40% faster than 0.991 against the Dropbox internal codebase 20 optimizations included in this release Mypy now warns about errors used before definition or possibly undefined variables for example if a variable is used outside of a block of code that may not execute Mypy now supports the new Self type introduced in PEP 673 and Python 3.11 Support ParamSpec in Type Aliases Also, ParamSpec and Generic Self types are no loner experimental Lots of Miscellaneous New Features Fixes to crashes Support for compiling Python match statements introduced in Python 3.10 Brian #4: Thoughts on the Python packaging ecosystem Pradyun Gedam Some great background on the internal tension around packaging. Brian’s note: in the meantime people are struggling to share Python code the “best practice” answer seems to shift regularly this might be healthy to arrive at better tooling in the long term, but in the short term, it’s hurting us. From the article: The Python packaging ecosystem unintentionally became the type of competitive space that it is today. The community needs to make an explicit decision if it should continue operating under the model that led to status quo. Pick from N different tools that do N different things is a good model. Pick from N ~equivalent choices is a really bad user experience. Picking a default doesn’t make other approaches illegal. Communication about the Python packaging ecosystem is fragmented, and we should improve that. Pradyun: “Many of the users who write Python code are not primarily full-time software engineers or “developers”.” from Thea: “The reason there are so many tools for managing Python dependencies is because Python is not a monoculture and different folks need different things.” opening up the build backend through pyproject.toml-based builds was good but the fracturing of multiple “workflow” tools seems bad. “I am certain that it is not possible to create a single “workflow” tool for Python software. What we have today, an ecosystem of tooling where each makes different design choices and technical trade-offs, is a part of why Python is as widespread as it is today. This flexibility and availability of choice is, however, both a blessing and a curse.” On building a default workflow tool around pip interesting idea There’s tension between “we need a default workflow tool” and “unix philosophy: many focused tools that can work together”. Michael #5: Top PyPI Packages A monthly dump of the 5,000 most-downloaded packages from PyPI. Also, a full copy of PyPI info too: github.com/orf/pypi-data Calvin #6: SQLAlchemy 2.0 Released #57 on the Top PyPI Packages 😸 Will be giving a SQLAlchemy tutorial at Python Web Conf What’s New? Significant API change from 1.4 You’ll want to follow the migration guide and see also the what’s new in 2.0 guide Fully takes advantage of Python 3 features such as dataclasses, enums and inline annotations Typing support in Core and ORM, but still should be considered beta all SQLAlchemy stubs packages must be uninstalled all SQLAlchemy stubs packages must be uninstalled for typing to work Mypy Plugin is considered deprecated now Major speed increase in the all new fully ORM-integrated bulk INSERTs sorry if you are on MySQL, they don’t support INSERT RETURNING yet but MariaDB does support this All new bulk optimized schema reflection architecture Currently enabled for PostgreSQL and Oracle 250% perf increase for Postgres 900% per increase for Oracle Native extensions ported to Cython C extensions have been replaced by Cython Benchmarks as fast or sometimes faster than the previous C extensions Removes some risk of memory or stability issues introduced by C SQLAlchemy is now pep-517 enabled and has a pyproject.toml at the root means that local source building with pip can auto install the Cython dependancy Extras Brian: Nothing to share yet, but I’m building a new alternative Python build backend. which if course will be followed with a new workflow tool that follows “my workflow”. Michael: “Create shortcut: New window” tip: In the dock/task bar Running as an app Speaking of Proton, started using simplelogin.io What’s all this banning chips about? Great documentary Talk Python is hiring! Calvin: 5th Annual Python Web Conf 2023