Python Bytes is a weekly podcast hosted by Michael Kennedy and Brian Okken. The show is a short discussion on the headlines and noteworthy news in the Python, developer, and data science space.
#308 Conference season is heating up
Watch the live stream: Watch on YouTube About the show Sponsored by Complier Podcast from RedHat Michael #0: New livestream time - 11am PT on Tuesdays. Also, subscribe to the youtube channel and “hit the bell” to get notified of all the live streams. Brian #1: It’s PyCon US 2023 CFP time Will be held in Salt Lake City, Salt Palace Convention Center Talks are Friday - Sunday, April 19-23 PyCon US 2023 launch announcement PyCon 2023 site features images taken from past PyCon artwork Call for proposals open until Dec 9, but please don’t wait that long. Michael #2: Any.io AnyIO is an asynchronous networking and concurrency library that works on top of either asyncio or trio. It implements trio-like structured concurrency (SC) on top of asyncio. Cool interpretability between native threads and asyncio Using subprocesses: AnyIO allows you to run arbitrary executables in subprocesses, either as a one-shot call or by opening a process handle for you that gives you more control over the subprocess. Async file I/O: AnyIO provides asynchronous wrappers for blocking file operations. These wrappers run blocking operations in worker threads. Cool synchronization primitives too. Catch the Talk Python episode with Alex: talkpython.fm/385 Brian #3: How to propose a winning conference talk Reuven Lerner Some nice tips and advice Build a list of topics If you train, teach, mentor, lead, or coach already: what questions to people always ask you? what knowledge would help people to have? where do people seem to just “not get it”? If you don’t train or teach, then maybe hit up Stack Overflow… From Brian: I think you can imagine yourself a year or two ago and think about stuff you know now you wish you knew then and could learn faster. Build an outline with times This part often seems scary, but Reuven’s example is 10 bullets with (x min) notes. Write up a summary. One short, one longer. Indicate who will benefit, what they will come out knowing, and how it will help them. Propose to multiple conferences. Why not? Practice (from Brian: Even if you get rejected, you’ve gained. Turn it into a youTube video or blog post or both.) - Michael #4: Sanic release adds background workers via Felix In v22.9 (go cal-ver!), the main new feature is the worker process management - the main Sanic process handles a pool of workers. They are normally used for handling requests but you can also use them to handle background jobs and similar things. You could probably use it for a lot of the reasons people turn to something like Celery. The lead developer (Adam Hopkins) has written a blog post about this feature. MK: Sanic has been flying a bit under my radar. Maybe time to dive into it a bit more. Extras Brian: Create Presentation from Jupyter Notebook Cool walkthrough of how to use the built in slideshow features of Jupyter Notebooks. pytest 7.2.0 is out No longer depends on the py library. So if you do, you need to add it to your dependencies. nose officially deprecated, which includes setup() and teardown(). Really glad I dropped the “x unit” section on the 2nd edition of the pytest book. testpaths now supports shell-style wildcards Lots of other improvements. check out the change log Michael: Rich on pyscript (via Matt Kramer) Python 3.11 in 100 seconds video from Michael Joke: Deep questions & Relationship advice from geeks
#307 Your Python just got faster (3.11 is out!)
Watch the live stream: Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Michael #1: Python 3.11 is released Live stream of the actual release procedure Talk Python episode coming next week (live stream on Friday) Major new features of the 3.11 series, compared to 3.10 General changes PEP 657 -- Include Fine-Grained Error Locations in Tracebacks PEP 654 -- Exception Groups and except* PEP 680 -- tomllib: Support for Parsing TOML in the Standard Library gh-90908 -- Introduce task groups to asyncio gh-34627 -- Atomic grouping ((?>...)) and possessive quantifiers (*+, ++, ?+, {m,n}+) are now supported in regular expressions. The Faster CPython Project is already yielding some exciting results. Python 3.11 is up to 10-60% faster than Python 3.10. On average, we measured a 1.22x speedup on the standard benchmark suite. See Faster CPython for details. Typing and typing language changes PEP 673 -- Self Type PEP 646 -- Variadic Generics PEP 675 -- Arbitrary Literal String Type PEP 655 -- Marking individual TypedDict items as required or potentially-missing PEP 681 -- Data Class Transforms Brian #2: Installing Python 3.11 on Mac or Windows pythontest.com I wrote this up because there are lots tutorials with weird instructions. For most people, I think the right answer is to use the python.org installer. It just works. One change, for Windows: Use “Advanced Options” and select “Add Python to environment variables”. The default process: allows multiple versions, like 3.7, 3.10, 3.11, to all live side by side. allows tox to use all of these allows you to specify which one if you want python3.10, for example, on mac py -3.10 on windows allows you to update versions in place. Say 3.10.7 to 3.10.8, or 3.11.0 to 3.11.1 when it comes out. Please, blog writers, stop recommending pyenv to novices. It’s a cool project, but it is not a project for casual users. And homebrew lovers, go for it, you are not going to read my article anyway. Michael #4: Bossie 2022 Awards Notable winners Wasmtime: Similar to what Node.js does for the JavaScript runtime, Wasmtime allows developers to leverage all of the advantages that WebAssembly provides inside the browser-including safe sandboxed execution, near-native performance, and support across multiple programming languages and platforms -outside the browser. (Python’s integration) PyScript: One of the long-gestating promises of WebAssembly is enabling the use of languages other than JavaScript in the web browser. PyScript delivers a full Python runtime in the browser, allowing you to use Python in webpages as a full-blown scripting language. Sentry: Alongside security, error and performance tracing are among the most frustratingly inevitable requirements for many apps. Cue a sigh of relief. Sentry offers an entire ecosystem of open source tools for monitoring the health of applications, services, and APIs, from the server-side API for collecting data, to a dashboard for making it manageable, to a comprehensive slew of application-side integrations. nbdev: One of the dirty secrets of notebook programming, using environments like Jupyter or Google Colab, is that it produces some of the worst spaghetti code you've ever seen, with data scientists hopping from cell to cell and creating an unmaintainable mess. Some even go so far as to say that notebook programming might be as harmful as GOTO was back in the day. Brian #5: Textual 0.2.0 All the cool things Will has been showing off on Twitter recently are part of the css branch. This has been merged, and released as 0.2.0 They also held off this release until they were happy with the documentation, which includes: A new tutorial that walks through a stopwatch application and everything that goes into it. An in depth reference guide with fully working examples, all of which are also in github, so you can play with it directly without retyping everything. Extras Michael: Video I created: 17x Faster Page Load in 30 minutes using Python, PyCharm, and MongoDB Pandas Markets Calendar (by Ryan Sheftel) Beanie adds a sync API DuckDuckGo browser, exciting and disappointing int() isn’t done yet via Will Shanks Ubuntu has decided to patch out the int limit and preserve the previous behavior on the basis that the risk factor is low and not worth breaking compatibility for. Joke: i heard you like getters
#306 Some Fun pytesting Tools
Watch the live stream: Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Brian #1: Awesome pytest speedup Neyts Zupan A checklist of best practices to speed up your pytest suite. as a talk at Plone NAMUR 2022 Measure first Then make sure (all items have explanations) Hardware is fast use a faster computer also try a self-hosted runner seriously, a dedicated computer (or a few) for making test runs faster might be worth it. CI resources are usually slower in cloud than local, and even expensive VM farms are often slower. Try local Collection is fast utilize norecursedirs and specifying the location of the tests, either on the command line or with testpaths PYTHONDONTWRITEBYTECODE=1 is set might help Built-in pytest plugins are disabled try -p no:pastebin -p no:nose -p no:doctest Only a subset of tests are executed Especially when developing or debugging, run a subset and skip the slow tests. Network access is disabled pytest-socket can make sure of that Disk access is disabled interesting idea Database access is optimized great discussion here, including using truncate and rollback. Tests run in parallel pytest-xdist or similar Then keep them fast monitor test speed Michael #2: Strive to travel without a laptop Prompt from Panic for SSH on iThings github.dev for an editor on iPad Push to branch for continuous deployment BTW, Apple could just make M1 iPads boot to macOS rather than chase silly multi windowing systems (stage manager, etc, etc) Brian #3: Some fun tools from the previous testing article hyperfine for timing the whole suite pytest --``durations 10 for finding test times of slowest 10 tests leave the 10 off to find times of everything, sorted pyinstrument for profiling with nice tree structures and how to use it with pytest pytest-socket disables network calls with --disable-socket, helping to find tests that use network calls. pyfakefs, a fake file system that mocks the Python file system modules. “Using pyfakefs, your tests operate on a fake file system in memory without touching the real disk.” BlueRacer.io Michael #4: Refurb A tool for refurbishing and modernizing Python codebases Think of it as suggesting the pythonic line of code. A little sampling of what I got on Talk Python Training file.py:186:25 [FURB106]: Replace x.replace("\t", " ") with x.expandtabs(1) file.py:128:17 [FURB131]: Replace del x[y] with x.pop(y) file.py:103:17 [FURB131]: Replace del x[y] with x.pop(y) file.py:112:39 [FURB109]: Replace not in [x, y, z] with not in (x, y, z) file.py:45:5 [FURB131]: Replace del x[y] with x.pop(y) file.py:81:21 [FURB131]: Replace del x[y] with x.pop(y) file.py:143:9 [FURB131]: Replace del x[y] with x.pop(y) file.py:8:50 [FURB123]: Replace list(x) with x.copy() You don’t always want the change, can suppress the recommendation with either a CLI flag or comment. Extras Michael: Back on episode 54 in 2017 we discussed python apps in systemd daemons. Multiprocessing allows for a cool way to save on server memory Do the scheduled work a multiprocessing.Process Here’s an example from Talk Python Training Completely rewrote search UI for Talk Python courses Google analytics is now illegal? Fleet is finally in public preview I’ll be on a JetBrains/PyCharm webcast Thursday. Joke: Tests pass
#305 Decorators need love too
Watch the live stream: Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Michael #0: Scorpions concert Check out this short video if you're interested. Michael #1: Pyscript 2022.09.1 is out Beware of the breaking changes Brian #2: Decorator shortcuts by Ned Batchelder Michael #3: Panel (of Holoviz) on Pyscript via Marc Skov Madsen Panel can now run on pyodide and pyscript with no backend server. Brian #4: auto-walrus Marco Gorelli Also, a reminder about pyupgrade, mentioned on the show back in 2018 Especially, look at the --py38-plus through --py39-plus flags. There are flags for 3.10 and 3.11, but limited testing for them. Joke: I need a new shirt
#304 Build your own text adventure language in Python
Watch the live stream: Watch on YouTube About the show Sponsored by Compiler Podcast from RedHat Brian #1: Ten tasty ingredients for a delicious pull request on wagtail blog, from LB Great tips for helping out with open source projects. But even for closed source, there’s good stuff there to know for people transitioning to single person projects to working on a team. The tips Read the [development] instructions. The contributing guide, etc. Read the issue and comments Create a fresh branch for your contribution Keep the changes focused Write unit tests also, extending tests for untested or under-tested features are a great way to contributes Give your pull request a name with context Reference the issue being fixed or resolved in the pull request Review & fix the CI failures Push to the same branch with fixes and do not open a new pull request Be patient. (Article lists this as “Eagerness balanced with patience”). Michael #2: textX via Rhet Turnbull textX is a meta-language for building Domain-Specific Languages (DSLs) in Python. It is inspired by Xtext. In a nutshell, textX will help you build your textual language in an easy way. You can invent your own language or build a support for already existing textual language or file format. From a single language description (grammar), textX will build a parser and a meta-model (a.k.a. abstract syntax) for the language Brian #3: Reasoning about asyncio.Semaphore Guido van Rossum Article uses a fast food restaurant analogy to reason about concurrency, asyncio, locks, and semaphores. A lock is like a single table restaurant with buzzers handed out to people waiting. A semaphore is like the same thing but with more than one table. Great discussion of the complexities of the Semaphore implementation. Also of concurrency. But almost as important, it’s an excellent example of utilizing a fairly easy to visualize analogy to reason about a complex problem. It also hits parts of the problem difficult to fit into the analogy, and pragmatically abandons the analogy when necessary. Michael #4: Turnstile A user-friendly, privacy-preserving alternative to CAPTCHA I created a Python library based on Pydantic to validate the forms: turnstile.py (should I make this proper package and GitHub repo?) Extras Brian: Choosing a place to host your Python application. aka Heroku Alternatives for Python-based Applications Kicking around the idea of starting cohort based pytest training, possibly starting mid Dec. Please let me know if you think this is a good idea and you might be interested. message me @brianokken Michael: Cppfront project aims to modernize C++ New pyscript (will cover in more depth soon) NextDNS follow up https://nextdns.io/ Yes, it’s basically PiHole in the Cloud with tons of options Currently outdoing my NordVPN protections Try with https://adblock-tester.com Had to manually set DNS-over-HTTPS in Vivaldi to work with VPN Add additional blocking lists Got a Feather ESP32-S2 and put CircuitPython on it Joke: Getting help with code
#303 This title is required or is it optional?
Watch the live stream: Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Michael #1: Human regular expressions revisited via Mikael Honkala We mentioned of Al Sweigart's humre in Python Bytes… Mikael went on a little search and compiled my findings into this repo. A lot of people feel that re needs some help. At least 3 of the "serious" packages I found came out in the last few months. Since a package like this is not overly complex to make, all the ways to approach the problem are clearly being explored. Unfortunately these seem to be mostly single-person efforts, and many have fallen to the wayside before long. Hopefully there's some consolidation on the horizon, to share some of the maintenance effort and establish some of the packages as here for the long haul. The list could be useful to you if you are: Looking for a tool: Check the list to get a quick idea of the "look and feel" of each package. Thinking about building a tool: Check the list for alternative approaches, and maybe consider if contributing to an existing package might be a better way to get what you need. Building a tool, or already have one: Use the list to clarify and communicate what the main differences and strengths of your solution are. Brian #2: Implicit Optional Types Will Be Disabled by Default … in a future mypy feature release (possibly the next one after 0.98x) … Thanks Adam Johnson for spotting this and letting us know Stop doing this: s: str = None Do one of these: s: str | None = None s: Union[str, None] = None s: Optional[str] = None ← but this has problems Optional != optional From python docs: ”Optional[X] is equivalent to X | None (or Union[X, None]).” “Note that this is not the same concept as an optional argument, which is one that has a default. An optional argument with a default does not require the Optional qualifier on its type annotation just because it is optional. “ Best described in FastAPI docs, Python Types Intro, starting at “Possibly None" Recommendation is to use: s: str | None = None for Python 3.10+ s: Union[str, None] = None for Python 3.9+ For 3.7, 3.8, you still have Optional as an option, I think. Why haven’t you upgraded to 3.9? We’re almost to 3.11, what’s the problem?! Michael #3: cython-lint by Marco Gorelli A tool (and pre-commit hook) to lint Cython files, similar to how flake8 lints Python files, and works by parsing Cython's own AST (abstract syntax tree). Found quite a few nice clean-ups which could be applied on: pandas numpy scikit-learn cupy Brian #4: difftastic - structural diff “Difftastic is a structural diff tool that understands syntax.” “Difftastic detects the language, parses the code, and then compares the syntax trees.” Interesting story about building difftastic For one off git diff replacement use GIT_EXTERNAL_DIFF=difft git diff or GIT_EXTERNAL_DIFF="difft --syntax-highlight=off" git diff To always use difft with git, see https://difftastic.wilfred.me.uk/git.html Extras Brian: Oh My Git! - An open source game about learning Git! Python 3.11.0 is up to rc2 Michael: NextDNS Joke: I mean, who’s wrong?
#302 The Blue Shirt Episode
Watch the live stream: Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Brian #1: Can Amazon’s CodeWhisperer write better Python than you? Brian Tarbox “Despite the clickbait-y title, whether CW’s code is better or worse than mine is at the margins and not really important. What is significant is that it has the potential to save me a ton of time and mental space to focus on improving, refactoring and testing. It’s making me a better programmer by taking on some of the undifferentiated heavy lifting.” Some decent code generation, starting with Amazon API examples. The generated dataclass method was neat, but really, the comment “prompt” probably took as much time to write as the code would have. The generated test case is workable, but I would not consider that a good test. Perhaps don’t lump together construction, attribute access, and tests for all methods in one test function. That said, I’ve seen way worse test methods in my career. So, decent starting point. Related and worth listening to: Changelog #506: Stable Diffusion breaks the internet w/ Simon Willison Mostly an episode about AI generated art. There is a bit of a tie in to AI code generation, the ethics around it, and making sure you walk up the value chain. I’m planning on playing with GitHub CoPilot. I’ve been reluctant in the past, but Simon’s interview is compelling to combine experienced engineering skill with AI code generation to possibly improve productivity. Simon does warn against possible abuse by Junior devs and the “just believe the code” problem that we also see with “copy from StackOverflow” situations. Michael #2: Apache Superset Apache Superset is a modern data exploration and visualization platform An intuitive interface for visualizing datasets and crafting interactive dashboards A wide array of beautiful visualizations to showcase your data Code-free visualization builder to extract and present datasets A world-class SQL IDE for preparing data for visualization, including a rich metadata browser A lightweight semantic layer which empowers data analysts to quickly define custom dimensions and metrics Out-of-the-box support for most SQL-speaking databases Seamless, in-memory asynchronous caching and queries An extensible security model that allows configuration of very intricate rules on who can access which product features and datasets. Integration with major authentication backends (database, OpenID, LDAP, OAuth, REMOTE_USER, etc) The ability to add custom visualization plugins An API for programmatic customization Brian #3: Recipes from Python SQLite docs Redowan Delowar Expanding on sqlite3 Python docs with more examples, including Executing individual and batch statements Applying user-defined callbacks: scalar and aggregate scalar example shows using a sha256 function to hash passwords as their inserted into the database Enabling tracebacks when callbacks raise an error Transforming types between SQLite and Python Implementing authorization control … much more … This is great for not only learning SQLite, but also, since these kinds of topics exist in other databases, learning about databases. AND a great example of learning a subsystem by creating little code snippets to check your understanding of something. One mod I would do in practice is to write these examples as pytest functions, because I can then run them individually while keeping a bunch in the same file. 🙂 Michael #4: -ffast-math and indirect changes Brendan Dolan-Gavitt downloaded 4 TB of Python packages containing native x86-64 libraries and see how many of them use -ffast-math, potentially altering floating point behavior in any program unlucky enough to load them! Python packages built with an appealing-sounding but dangerous compiler option, -ffast-math, could end up causing any program that uses them to compute incorrect numerical results. When -ffast-math is enabled, the compiler will link in a constructor that sets the FTZ/DAZ flags whenever the library is loaded — even on shared libraries, which means that any application that loads that library will have its floating point behavior changed for the whole process. A total of 2,514 packages eventually depend on a package that uses -ffast-math. Because of highly connected nature of the modern software supply chain, even though a mere 49 packages were actually built with -ffast-math, thousands of other packages, with a total of at least 9.7 million downloads over the past 30 days, are affected. Extras Brian: Thinking about changelogs Focusing on helping teams with specific goals Working on an experiment in consulting with some lead engineers before the training to altering the content of a pytest course so the examples better match what the team will need. Sharing packages through internal system, as that’s usually different than the PyPI process. Altering the database and API example of the TalkPython pytest course content to match a teams external resources and responsibility scope. It takes extra time and thought, but in the end might increase engagement and excitement about testing and keeping up on Python’s evolving common practices. Michael: New course: Python Data Visualization pytest course going strong Joke: They all use it State of emergency
#301 PyTorch Grows Up and Moves Out
Watch the live stream: Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Michael #1: PythonAnywhere: Our Commitment to Providing Free Accounts via Matthew Kramer In light of Heroku’s cancelling their free tiers… They believe free tiers are important for beginners Two part solution: Limit outbound internet access for free accounts “Proof of life” to keep running - 3 months for apps, 1 yr for accounts BTW, they were acquired by Anaconda Inc. Brian #2: ruff: An extremely fast Python linter, written in Rust. Announcement article: Python tooling could be much, much faster Charlie Marsh Quite the star history, as it’s a new repo as of Aug 30. Now at 1.8k. It is extremely fast. I installed it and tried it on a small project. It ran so fast I thought it didn’t do anything. I went and added some errors to convince myself it was running. $ time flake8 src tests ... flake8 src tests 0.29s user 0.02s system 98% cpu 0.311 total $ time ruff src/ tests/ ... ruff src/ tests/ 0.01s user 0.01s system 162% cpu 0.011 total Michael #3: Meta spins off PyTorch Foundation to make AI framework vendor neutral PyTorch, which powers Tesla Autopilot and 150K other projects, will join the Linux Foundation. Its governing board includes representatives from Nvidia, Meta, Google, Microsoft, Amazon, and AMD. The PyTorch Foundation will strive to adhere to four principles, Remaining open Maintaining neutral branding Staying fair Forging a strong technical identity According to Meta, the transition to the PyTorch Foundation will not affect any existing PyTorch code Brian #4: Two string resources Python String Methods to Know Trey Hunner F-Strings Number Formatting Cheat Sheet Brian Allan Extras Brian: In Feb, on episode 271, we talked about Seaborn’s new object interface Well, it’s out now in seaborn 0.12 Interesting discussion about lazy imports. Other than that, I’m good with your extra. 🙂 Michael: pytest course is out! Joke: If a developer had to build a horse…
#300 A Jupyter merge driver for git
Watch the live stream: Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Special guest: Seth Larson Brian #1: Test your packages and wheels I’ve been building some wheels the last couple of weeks with various tools: flit, flit-core, and flit build hatch, hatchling, and hatch build setuptools, build_meta, and python -m build There are a few projects I’ve used to make sure my projects are in good shape wheel-inspect - you can inspect within Python code through inspect_wheel() function that converts to json. Or use on the command line with wheel2json check-wheel-contents - a linter for wheels tox - easily test the building, installation, and running of a package locally I actually start here, then utilize the other two tools Should have been obvious, but it wasn’t to me Projects saved on git (such as gitHub) don’t keep wheels in git. (this was obvious) When installing from git using pip install git+https://path/to/git/repo.git Your local pip will run the packaging backend to build the wheel before installing. Yet another way to test packaging. Michael #2: The Jupyter+git problem is now solved Jupyter notebooks don’t work with git by default (they inherently have meaningless conflicts). With nbdev2, the Jupyter+git problem has been totally solved. Uses a set of hooks which provide clean git diffs, solve most git conflicts automatically, and ensure that any remaining conflicts can be resolved entirely within the standard Jupyter notebook environment. The techniques used to make the merge driver work are quite fascinating Seth #3: Help us test system trust stores in Python Package aiming to replace certifi called “truststore”, use system trust stores for HTTPS instead of a static list of certificates. Problem truststore is solving usually manifests in corporate networks: “unable to get local issuer certificate”. Experimental support added to pip to prove the implementation Users can try out the functionality and report issues. Brian #4: Making plots in your terminal with plotext Bob Belderbos Tutorial on using plotext - that’s one t in the middle With the rise of CLI usage, plots are a nice addition. Bob’s plot is great, but check out the options in the plotext docs lots-o-plots streaming data images subplots so fun Michael #5: jinja2-fragments Carson from HTMX (see podcast and course) wrote about template fragments. My jinja_partials project sorta fulfills this, but not really. I had a nice discussion with Sergi Pons Freixes who uses jinja_partials about this. He created Jinja2 fragments Seth #6: SLSA 3 Generic Builder for GitHub Actions GA Supply chain Levels for Software Artifacts, or SLSA (“salsa”) Tools to attest to and verify “provenance” of artifacts, ie “where it came from” Prove cryptographically that artifacts are built from a specific GitHub repository, commit, tag. Another future defense against stolen PyPI credentials/accounts. Generic builder means you can sign anything, like wheels/sdists Extras Brian: Bring your pytest books to PyBay, if you want them signed. I’m only bringing a small amount. I’ll be presenting "Sharing is Caring - pytest fixture edition” at 3:05 “Experts Panel on Testing in Python” at 7:00 And be a zombie on my 8 am flight back unless I can change my reservation. That’s this weekend, Sat Sept 10, in SF Michael: Heroku announces plans to eliminate free plans Banned paywalls PyPI phisher identified: Actor Phishing PyPI Users Identified and Actors behind PyPI supply chain attack have been active since late 2021 Major Python CVE: CVE-2020-10735: Prevent DoS by large int[HTML_REMOVED]str conversions Seth: Pyxel, retro game engine for Python, v1.8.0 added experimental web support with WASM Joke: Dev just after work
#299 Will McGugan drops by
Watch the live stream: Watch on YouTube About the show Sponsored by the IRL Podcast from Mozilla Michael #1: Careful with that PyPI email via John Hagen There is a widespread Phishing attack going on against PyPI users. The PyPA is currently tweeting about it: https://twitter.com/pypi/status/1562442188285308929 Brian #2: IEEE Top Programming Languages 2022 : Python’s still No. 1, but employers love to see SQL skills by Stephen Cass Related: The Rise of SQL It’s become the second programming language everyone needs to know by Rina Diane Caballar Good to see Python on top for Spectrum and Trending But interesting to see growth and strength in SQL SQL is actually top for Jobs SQL is a skill you can’t ignore. Not only are relational databases just as important for large systems, they’re increasingly more important for small and even local systems, and are ever growing the responsibility of developers, not left to database specialists. Will #3: Using Mypy in production at Spring by Charlie Marsh Michael #4: Django 4.1 Yes, I’m a bit slow to notice this, from August 3, 2022. Big deal for us async folks! Asynchronous ORM interface: QuerySet now provides an asynchronous interface for all data access operations. Asynchronous handlers for class-based views: View subclasses may now define async HTTP method handlers: Also: Validation of Constraints: Check, unique, and exclusion constraints defined in the Meta.constraints option are now checked during model validation. Check out Chris’ Django: Getting Started course at Talk Python. Brian #5: You Should Be Using Python's Walrus Operator - Here’s Why by Martin Heinz A fun look at some places where I’ve never considered using := Examples reusing a value while building a list regular expression match results cleaning up while loops (ok, that I’m using already, but it’s great) accumulating data in place named values in f-strings for multiple formatting. wow, super cool. … Will #6: Humre By Al Sweigart Author of "Automate the Boring Stuff with Python" Human readable regular expressions§ Joke: Password PR
#298 "Unstoppable" Python
Watch the live stream: Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Brian #1: Uncommon Uses of Python in Commonly Used Libraries by Eugene Yan Specifically, Using relative imports Example from sklearn’s base.py from .utils.validation import check_X_y from .utils.validation import check_array “Relative imports ensure we search the current package (and import from it) before searching the rest of the PYTHONPATH. “ For relative imports, we have to use the from .something import thing form. We cannot use import .something since later on in the code .something isn’t valid. There’s a good discussion of relative imports in pep 328 Michael #2: Skyplane Cloud Transfers Skyplane is a tool for blazingly fast bulk data transfers in the cloud. Skyplane manages parallelism, data partitioning, and network paths to optimize data transfers, and can also spin up VM instances to increase transfer throughput. You can use Skyplane to transfer data: Between buckets within a cloud provider Between object stores across multiple cloud providers (experimental) Between local storage and cloud object stores Skyplane takes several steps to ensure the correctness of transfers: Checksums, verify files exist and match sizes. Data transfers in Skyplane are encrypted end-to-end. Security: Encrypted while in transit and over TLS + config options Brian #3: 7 things I've learned building a modern TUI framework by Will McGugan Specifically, DictViews are amazing. They have set operations. Example of using items() to get views, then ^ for symmetric difference (done at the C level): # Get widgets which are new or changed print(render_map.items() ^ new_render_map.items()) Lots of other great topics in the article lru_cache is fast Unicode art in addition to text in doc strings The fractions module and a cool embedded video demo of some of the new css stuff in Textual Python’s object allocator ascii art Michael #4: ‘Unstoppable’ Python Python popularity still soaring: ‘Unstoppable’ Python once again ranked No. 1 in the August updates of both the Tiobe and Pypl indexes of programming language popularity. Python first took the top spot in the index last October, becoming the only language besides Java and C to hold the No. 1 position. “Python seems to be unstoppable,” said the Tiobe commentary accompanying the August index. In the alternative Pypl Popularity of Programming Language index, which assesses language popularity based on Google searches of programming language tutorials, Python is way out front. Extras Brian: Matplotlib stylesheets can make your chart look awesome with one line of code. But it never occurred to me that I could write my own style sheet. Here’s an article discussing creation of custom matplotlib stylesheets The Magic of Matplotlib Stylesheets XKCD Plots Michael: Back on 295 we talked about Flet. We now have a Talk Python episode on it (live and polished versions). Joke: Rakes and AWS
#297 I AM the documentation
Watch the live stream: Watch on YouTube About the show Sponsored by the IRL Podcast from Mozilla Michael #1: SQLCodeGen via Josh Thurston This is a tool that reads the structure of an existing database and generates the appropriate SQLAlchemy model code, using the declarative style if possible. This tool was written as a replacement for sqlautocode, which was suffering from several issues (including, but not limited to, incompatibility with Python 3 and the latest SQLAlchemy version). Features: Supports SQLAlchemy 1.4.x Produces declarative code that almost looks like it was hand written Produces PEP 8 compliant code Accurately determines relationships, including many-to-many, one-to-one Automatically detects joined table inheritance Excellent test coverage Brian #2: The death of setup.py*, long live pyproject.toml for Python-only projects Juan Luis Cano Rodriguez tweet pip install --``editable . now works with setuptools, as of version 64.0.0 To be clear, setup.cfg also not required. So everything can be in pyproject.toml The * part: projects with non-Python bits may still need setup.py See also the newly updated tutorial by the PyPA: Packaging Python Projects Now with absolutely no mention of setup.py or setup.cfg It’s all pyproject.toml Commentary: For Python only projects, is setuptools a decent flit contender??? stay tuned Michael #3: aiocache via Owen Lamont In the same vein as async-cache you might also be interested in aiocache. It has some cool functionality like an optional Redis backend for multi process caching. his library aims for simplicity over specialization. All caches contain the same minimum interface which consists on the following functions: add: Only adds key/value if key does not exist. get: Retrieve value identified by key. set: Sets key/value. multi_get: Retrieves multiple key/values. multi_set: Sets multiple key/values. exists: Returns True if key exists False otherwise. increment: Increment the value stored in the given key. delete: Deletes key and returns number of deleted items. clear: Clears the items stored. raw: Executes the specified command using the underlying client. Brian #4: Hatch : a modern, extensible Python project manager Another flit contender? While reading Packaging Python Projects tutorial update, I noticed some examples for hatchling, as an alternative to setuptools, flit-core, and pdm. Played with it some, but still have some exploring to do. features Standardized build system with reproducible builds by default Robust environment management with support for custom scripts Easy publishing to PyPI or other sources includes --repo flag to be able to publish to alternative indices. Awesome for internal systems. Version management Configurable project generation with sane defaults Responsive CLI, ~2-3x faster than equivalent tools This sounds great. I haven’t verified this Commentary: Good to see more packaging tools and user workflow explorations around packaging. Extras Michael: M1 Support for PyPy Announced (via PyCoders) Joke: I am the docs
#296 pip: Constrain your excitement
Watch the live stream: Watch on YouTube About the show Sponsored by the IRL Podcast from Mozilla Brian #1: Pip constraints files by luminousmen You can put some constraints on your dependencies with a constraints file. “Constraints files are requirements files that only control which version of a requirement is installed, not whether it is installed or not. “ Syntax is a subset of requirements.txt syntax but all the restrictions seem reasonable, considering must have a name can’t be editable can’t specify extras (that one is maybe slightly weird) You can put --constraint constraints.txt right at the top of your requirements.txt file or specify it on command line, pip install --constraint constraints.txt -r requirements.txt Or, my favorite, stick it at the top of requirements.in file. yes. pip-compile correctly handles constraints when generating requirements.txt. Example requirements.in --constraint constraints.txt typer constraints.txt click<8.1.3 Output from pip-compile requirements.in # # This file is autogenerated by pip-compile with python 3.10 # To update, run: # # pip-compile requirements.in # click==8.1.2 # via # -c constraints.txt # typer typer==0.6.1 # via -r requirements.in Michael #2: async-cache A caching solution for asyncio Quite simple but looks effective and flexible too Example: # TTL Cache from cache import AsyncTTL @AsyncTTL(time_to_live=60, maxsize=1024) async def func(*args, **kwargs): """ time_to_live : max time for which a cached result is valid maxsize : max number of results that are cached. if max limit is reached the oldest result is deleted. """ pass Brian #3: Organize Python code like a PRO Guilherme Latrova Yes, this is one author’s opinion. but… lots of great advice nothing too weird no tool recommendations Recommendations of note keep a src dir. A cool and simple reason: it keeps your source code together in alphabetized IDEs. file/module names: plural except for config, main, core, or similar slightly weird tangent that there are no files, there are modules. ok, whatever. Also talking about directories as main modules. odd. but ok. functions/methods should be verbs variables/constants should be nouns Class names should be singular, unless the class really represents a container The __name__ == "__main__" trick for modules. The __main__.py entry point trick for modules/packages so that -m mymodule does something. - Michael #4: keyring via Trent The Python keyring library provides an easy way to access the system keyring service from python. It can be used in any application that needs safe password storage. It’s also helpful in that it allows you to write your own backends. Useful for testing. Basically create a dummy keychain that stores to a pytest temp_path fixture instead of cluttering the real keychain. You could potentially write a backend to interact with any service such as 1Password. Extras Brian: I’m taking a class on FastAPI. The instructor is awesome! Also, editing some pytest course video. Michael: Gitlab, you alright?? Joke: from a dad-jokes repo Q: How do programming pirates pass method parameters? A: Varrrrarrrgs. Q: How do you get the code for the bank vault? A: You checkout their branch. "Unfortunately these jokes only work if you git them." Screw driver
#295 Flutter + Python GUI Apps?
Watch the live stream: Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Michael #1: Faster routing for Flask & Quart Flask and Quart both utilise Werkzeug's HTTP route With the upcoming 2.2 release of Werkzeug this router will be significantly faster, with up to a factor of 5. Better for large sets of routes Micro-benchmarks are unaffected and are unlikely to show a speedup. Started with tree-based radix algorithm Moved to state machine algorithm because of wild cards Brian #2: Quarto: an open-source scientific and technical publishing system built on Pandoc suggested by Paul Mackenzie Power of Pandoc and Jupyter Build documents - html, pdf, word presentations - Revealjs, PowerPoint, Beemer websites books - html, pdf, word, epub journal articles - acm, plos, elsevier, acs, jss Publish GitHub pages, Netlify, … kinda related - Kindle to support ePub Michael #3: Flet UI via Mikael Honkala New and upcoming UI framework by Feodor Fitzner: flet. It has a very interesting stack - a Python client driving a Flutter front-end via a Go server. That sounds complicated, but the developer experience is incredibly simple. Installation is just pip install flet. Here's a quick and stupid but working sample: import time import flet from flet import Column, ElevatedButton, Page, Row, TextField def main(page: Page): text_field = TextField() def clear_field(event): text_field.value = "CLEARING" page.update() time.sleep(1) text_field.value = "" page.update() clear_button = ElevatedButton("Clear the field", on_click=clear_field) page.add(Row([Column([text_field, clear_button])], alignment="center")) page.update() flet.app(target=main) # If you run this, you get a native app window on Mac, Windows or Linux, looking something like this: While the example is simple, it shows the handling of an event, updating the UI, and even doing a little sleeping between the updates, without having to worry about threads and such. What's more important, if you change the last line to: flet.app(target=main, view=WEB_BROWSER) You get the exact same functionality, but as a web application in a browser, with support for multiple users and deep linking to different parts of the app. All without leaving the comfortable Python world, with its access to all Python libraries, and without having to learn 3 extra, completely different languages (yes, HTML, CSS and JavaScript). As this is Flutter, mobile support is in the works, after the basic UI functionality is all there. Check the project front page here: flet.dev Jump directly to the currently available controls: flet.dev/docs/controls Check couple of tutorials here: flet.dev/docs/tutorials Or read the plans for the mobile support here: flet.dev/blog/flet-mobile-strategy Brian #4: Building an authenticated Python CLI Project that uses click, rich, and OAth for using Twitter API Persistent authentication requests secret information from user using getpass and input Client ID, Client Secret, App name fetches bearer token from Twitter API stores token in netrc file I’m not familiar with netrc, so I don’t know if this is a good idea or not. So I figured I’d ask Michael Extras Michael: New course: Django - Getting Started Joke: Light touches kingdom
#294 Specializing Adaptive Interpreters in Full Color
Watch the live stream: Watch on YouTube About the show Sponsored by Microsoft for Startups Founders Hub. Michael #1: Specialist: Python 3.11 perf highlighter via Alex Waygood Visualize CPython 3.11's specializing, adaptive interpreter. 🔥 PEP 659 – Specializing Adaptive Interpreter Specialist uses fine-grained location information to create visual representations of exactly where and how CPython 3.11's new specializing, adaptive interpreter optimizes your code. Dark, rich colors indicate code with many quickened instructions (and, therefore, high specialization potential), while light, pale colors indicate code with relatively few specialization opportunities. Brian #2: tomli “A lil’ TOML parser” Fully compatible with TOML spec 1.0.0 This is the library that tomllib from Python 3.11 is based on, so great to use for Python 3.7-3.10 applications. We discussed Python 3.11 and PEP 680 on episode 273 Real Python has a great introduction for TOML in Python: Python and TOML: New Best Friends TOML as a config format, key-value pairs, data types using both tomli and tomllib Loading TOML documents into Python And like, writing, and updating toml docs programatically, which, although cool, I think the bulk of users can kinda skip over. But the the first 3 sections are an excellent reference. Tables are cool, with [name] and [name.subsection] syntax, as well as arrays of tables with [[name]] syntax. I didn’t know how to do that before this article. Michael #3: Pydantic V2 Plan via Douglas Nichols and John Thagen A very detailed plan Goal to have all this done by the end of October, definitely by the end of the year. Samuel Colvin take a sabbatical to work on this (sound familiar?) Some details highlighted by John: Moving the core logic to Rust to drastically increase performance (17x) https://pydantic-docs.helpmanual.io/blog/pydantic-v2/#performance Strict mode (something I've wanted for a long time): https://pydantic-docs.helpmanual.io/blog/pydantic-v2/#strict-mode Cleaning up required vs nullable: https://pydantic-docs.helpmanual.io/blog/pydantic-v2/#required-vs-nullable-cleanup Naming cleanup: https://pydantic-docs.helpmanual.io/blog/pydantic-v2/#model-namespace-cleanup This is a huge change, but overall it looks very promising for the community. It will likely require refactors by downstream users, so pinning pydantic using Poetry/pip-tools etc like always is a good idea. Many things have Pydantic at the core, so this matters, including: FastAPI Beanie SQLModel Pydastic … Brian #4: pikepdf a Python library for reading and writing PDF files Based on QPDF, which is written in C++. Features: Supports password protected PDFs Creates linearized ("fast web view") PDFs Integrates with Jupyter and IPython notebooks for rapid development Some cool uses copy pates from one PDF into another split and merge PDFs extract content replace content, such as replacing images, without altering the rest of the file. Documentation mentions that if you only want to write PDFs, consider other libs, such as reportlab. Extras Brian: pytest-check I’ve set up 2fa for my account, so now I have no excuse for not looking into feature requests and merge requests for pytest-check, other than like all the other things I’m doing. I don’t have data for the top 3,500 for the last 6 months, but there is a list of the top 5,000 for last 30 days. pytest-check is #1677 in the last 30 days. pytest is #72 on the same list. pydantic is #117 There are 57 pytest plugins that show up in the top 3,500 python packages. (packages that start with “pytest-”) pytest-check is #20 of those. I guess it’s time to do another top plugins episode of Test & Code. Joke: Error, OK, I’ll check the logs