Created by three guys who love BSD, we cover the latest news and have an extensive series of tutorials, as well as interviews with various people from all areas of the BSD community. It also serves as a platform for support and questions. We love and advocate FreeBSD, OpenBSD, NetBSD, DragonFlyBSD and TrueOS. Our show aims to be helpful and informative for new users that want to learn about them, but still be entertaining for the people who are already pros. The show airs on Wednesdays at 2:00PM (US Eastern time) and the edited version is usually up the following day.
Similar Podcasts
Elixir Outlaws
Elixir Outlaws is an informal discussion about interesting things happening in Elixir. Our goal is to capture the spirit of a conference hallway discussion in a podcast.
The Cynical Developer
A UK based Technology and Software Developer Podcast that helps you to improve your development knowledge and career,
through explaining the latest and greatest in development technology and providing you with what you need to succeed as a developer.
Programming Throwdown
Programming Throwdown educates Computer Scientists and Software Engineers on a cavalcade of programming and tech topics. Every show will cover a new programming language, so listeners will be able to speak intelligently about any programming language.
314: Swap that Space
Unix virtual memory when you have no swap space, Dsynth details on Dragonfly, Instant Workstation on FreeBSD, new servers new tech, Experimenting with streaming setups on NetBSD, NetBSD’s progress towards Steam support thanks to GSoC, and more. Headlines What has to happen with Unix virtual memory when you have no swap space (https://utcc.utoronto.ca/~cks/space/blog/unix/NoSwapConsequence) Recently, Artem S. Tashkinov wrote on the Linux kernel mailing list about a Linux problem under memory pressure (via, and threaded here). The specific reproduction instructions involved having low RAM, turning off swap space, and then putting the system under load, and when that happened (emphasis mine): Once you hit a situation when opening a new tab requires more RAM than is currently available, the system will stall hard. You will barely be able to move the mouse pointer. Your disk LED will be flashing incessantly (I'm not entirely sure why). [...] I'm afraid I have bad news for the people snickering at Linux here; if you're running without swap space, you can probably get any Unix to behave this way under memory pressure. If you can't on your particular Unix, I'd actually say that your Unix is probably not letting you get full use out of your RAM. To simplify a bit, we can divide pages of user memory up into anonymous pages and file-backed pages. File-backed pages are what they sound like; they come from some specific file on the filesystem that they can be written out to (if they're dirty) or read back in from. Anonymous pages are not backed by a file, so the only place they can be written out to and read back in from is swap space. Anonymous pages mostly come from dynamic memory allocations and from modifying the program's global variables and data; file backed pages come mostly from mapping files into memory with mmap() and also, crucially, from the code and read-only data of the program. See link for the rest of the article Dsynth details on Dragonfly (https://www.dragonflydigest.com/2019/08/27/23398.html) First, history: DragonFly has had binaries of dports available for download for quite some time. These were originally built using poudriere, and then using the synth tool put together by John Marino. Synth worked both to build all software in dports, and as a way to test DragonFly’s SMP capability under extreme load. Matthew Dillon is working on a new version, called dsynth. It is available now but not yet part of the build. He’s been working quickly on it and there’s plenty more commits than what I have linked here. It’s already led to finding more high-load fixes. dsynth DSynth is basically synth written in C, from scratch. It is designed to give us a bulk builder in base and be friendly to porting and jails down the line (for now its uses chroot's). The original synth was written by John R. Marino and its basic flow was used in writing this program, but as it was written in ada no code was directly copied. The intent is to make dsynth compatible with synth's configuration files and directory structure. This is a work in progress and not yet ready for prime-time. Pushing so we can get some more eyeballs. Most of the directives do not yet work (everything, and build works, and 'cleanup' can be used to clean up any dangling mounts). dsynth code (https://gitweb.dragonflybsd.org/dragonfly.git/blob/HEAD:/usr.bin/dsynth/dsynth.1) News Roundup Instant Workstation (https://euroquis.nl/freebsd/2019/08/12/instant-workstation.html) Some considerable time ago I wrote up instructions on how to set up a FreeBSD machine with the latest KDE Plasma Desktop. Those instructions, while fairly short (set up X, install the KDE meta-port, .. and that’s it) are a bit fiddly. So – prompted slightly by a Twitter exchange recently – I’ve started a mini-sub-project to script the installation of a desktop environment and the bits needed to support it. To give it at least a modicum of UI, dialog(1) is used to ask for an environment to install and a display manager. The tricky bits – pointed out to me after I started – are hardware support, although a best-effort is better than having nothing, I think. In any case, in a VBox host it’s now down to running a single script and picking Plasma and SDDM to get a usable system for me. Other combinations have not been tested, nor has system-hardware-setup. I’ll probably maintain it for a while and if I have time and energy it’ll be tried with nVidia (those work quite well on FreeBSD) and AMD (not so much, in my experience) graphics cards when I shuffle some machines around. Here is the script in my GitHub repository with notes-for-myself. (https://raw.githubusercontent.com/adriaandegroot/FreeBSDTools/master/bin/instant-workstation) New Servers, new Tech (https://www.dragonflydigest.com/2019/08/26/23396.html) Following up on an earlier post, the new servers for DragonFly are in place. The old 40-core machine used for bulk build, monster, is being retired. The power efficiency of the new machines is startling. Incidentally, this is where donations go – infrastructure. New servers in the colo, monster is being retired (http://lists.dragonflybsd.org/pipermail/users/2019-August/358271.html) We have three new servers in the colo now that will be taking most/all bulk package building duties from monster and the two blades (muscles and pkgbox64) that previously did the work. Monster will be retired. The new servers are a dual-socket Xeon (sting) and two 3900X based systems (thor and loki) which all together burn only around half the wattage that monster burned (500W vs 1000W) and 3 times the performance. That's at least a 6:1 improvement in performance efficiency. With SSD prices down significantly the new machines have all-SSDs. These new machines allow us to build dports binary packages for release, master, and staged at the same time and reduces the full-on bulk build times for getting all three done down from 2 weeks to 2 days. It will allow us to more promptly synchronize updates to ports with dports and get binary packages up sooner. Monster, our venerable 48-core quad-socket opteron is being retired. This was a wonderful dev machine for working on DragonFly's SMP algorithms over the last 6+ years precisely because its inter-core and inter-socket latencies were quite high. If a SMP algorithm wasn't spot-on, you could feel it. Over the years DragonFly's performance on monster in doing things like bulk builds increased radically as the SMP algorithms got better and the cores became more and more localized. This kept monster relevant far longer than I thought it would be. But we are at a point now where improvements in efficiency are just too good to ignore. Monster's quad-socket opteron (4 x 12 core 6168's) pulls 1000W under full load while a single Ryzen 3900X (12 core / 24 thread) in a server configuration pulls only 150W, and is slightly faster on the same workload to boot. I would like to thank everyone's generous donations over the last few years! We burned a few thousand on the new machines (as well as the major SSD upgrades we did to the blades) and made very good use of the money, particularly this year as prices for all major components (RAM, SSDs, CPUs, Mobos, etc) have dropped significantly. Experimenting with streaming setups on NetBSD (https://dressupgeekout.blogspot.com/2019/08/experimenting-with-streaming-setups-on.html?m=1) Ever since OBS was successfully ported to NetBSD, I’ve been trying it out, seeing what works and what doesn’t. I’ve only just gotten started, and there’ll definitely be a lot of tweaking going forward. Capturing a specific application’s windows seems to work okay. Capturing an entire display works, too. I actually haven’t tried streaming to Twitch or YouTube yet, but in a previous experiment a few weeks ago, I was able to run a FFmpeg command line and that could stream to Twitch mostly OK. My laptop combined with my external monitor allows me to have a dual-monitor setup wherein the smaller laptop screen can be my “broadcasting station” while the bigger screen is where all the action takes place. I can make OBS visible on all Xfce workspaces, but keep it tucked away on that display only. Altogether, the setup should let me use the big screen for the fun stuff but I can still monitor everything in the small screen. NetBSD Made Progress Thanks To GSoC In Its March Towards Steam Support (https://www.phoronix.com/scan.php?page=news_item&px=NetBSD-Linux-DRM-Ioctl-GSoC2019) Ultimately the goal is to get Valve's Steam client running on NetBSD using their Linux compatibility layer while the focus the past few months with Google Summer of Code 2019 were supporting the necessary DRM ioctls for allowing Linux software running on NetBSD to be able to tap accelerated graphics support. Student developer Surya P spent the summer working on compat_netbsd32 DRM interfaces to allow Direct Rendering Manager using applications running under their Linux compatibility layer. These interfaces have been tested and working as well as updating the "suse131" packages in NetBSD to make use of those interfaces. So the necessary interfaces are now in place for Linux software running on NetBSD to be able to use accelerated graphics though Steam itself isn't yet running on NetBSD with this layer. Those curious about this DRM ioctl GSoC project can learn more from the NetBSD blog (https://blog.netbsd.org/tnf/entry/gsoc_2019_report_implementation_of). NetBSD has also been seeing work this summer on Wayland support and better Wine support to ultimately make this BSD a better desktop operating system and potentially a comparable gaming platform to Linux. Beastie Bits FreeBSD in Wellington? (https://twitter.com/MengTangmu/status/1163265206660694016) FreeBSD on GFE (https://twitter.com/onewilshire/status/1163792878642114560) Clarification (https://twitter.com/onewilshire/status/1166323112620826624) Distrotest.net now with BSDs (https://distrotest.net/) Lecture: Anykernels meet fuzzing NetBSD (https://fahrplan.events.ccc.de/camp/2019/Fahrplan/events/10334.html) Sun Microsystems business plan from 1982 [pdf] (https://www.khoslaventures.com/wp-content/uploads/SunMicrosystem_bus_plan.pdf) Feedback/Questions Alan - Questions (http://dpaste.com/1Z8EGTW) Rodriguez - Feedback and a question (http://dpaste.com/2PZFP4X#wrap) Jeff - OpenZFS follow-up, FreeBSD Adventures (http://dpaste.com/02ZM6YE#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
313: In-Kernel TLS
OpenBSD on 7th gen Thinkpad X1 Carbon, how to install FreeBSD on a MacBook, Kernel portion of in-kernel TLS (KTLS), Boot Environments on DragonflyBSD, Project Trident Updates, vBSDcon schedule, and more. Headlines OpenBSD on the Thinkpad X1 Carbon 7th Gen (https://jcs.org/2019/08/14/x1c7) Another year, another ThinkPad X1 Carbon, this time with a Dolby Atmos sound system and a smaller battery. The seventh generation X1 Carbon isn't much different than the fifth and sixth generations. I opted for the non-vPro Core i5-8265U, 16Gb of RAM, a 512Gb NVMe SSD, and a matte non-touch WQHD display at ~300 nits. A brighter 500-nit 4k display is available, though early reports indicated it severely impacts battery life. Gone are the microSD card slot on the back and 1mm of overall thickness (from 15.95mm to 14.95mm), but also 6Whr of battery (down to 51Whr) and a little bit of travel in the keyboard and TrackPoint buttons. I still very much like the feel of both of them, so kudos to Lenovo for not going too far down the Apple route of sacrificing performance and usability just for a thinner profile. On my fifth generation X1 Carbon, I used a vinyl plotter to cut out stickers to cover the webcam, "X1 Carbon" branding from the bottom of the display, the power button LED, and the "ThinkPad" branding from the lower part of the keyboard deck. See link for the rest of the article How To Install FreeBSD On A MacBook 1,1 or 2,1 (http://lexploit.com/freebsdmacbook1-1-2-1/) FreeBSD Setup For MacBook 1,1 and 2,1 FreeBSD with some additional setup can be installed on a MacBook 1,1 or 2,1. This article covers how to do so with FreeBSD 10-12. Installing FreeBSD can be installed as the only OS on your MacBook if desired. What you should have is: A Mac OS X 10.4.6-10.7.5 installer. Unofficial versions modified for these MacBooks such as 10.8 also work. A blank CD or DVD to burn the FreeBSD image to. Discs simply work best with these older MacBooks. An ISO file of FreeBSD for x86. The AMD64 ISO does not boot due to the 32 bit EFI of these MacBooks. Burn the ISO file to the blank CD or DVD. Once done, make sure it's in your MacBook and then power off the MacBook. Turn it on, and hold down the c key until the FreeBSD disc boots. See link for the rest of the guide News Roundup Patch for review: Kernel portion of in-kernel TLS (KTLS) (https://svnweb.freebsd.org/base?view=revision&revision=351522) One of the projects I have been working on for the past several months in conjunction with several other folks is upstreaming work from Netflix to handle some aspects of Transport Layer Security (TLS) in the kernel. In particular, this lets a web server use sendfile() to send static content on HTTPS connections. There is a lot more detail in the review itself, so I will spare pasting a big wall of text here. However, I have posted the patch to add the kernel-side of KTLS for review at the URL below. KTLS also requires other patches to OpenSSL and nginx, but this review is only for the kernel bits. Patches and reviews for the other bits will follow later. https://reviews.freebsd.org/D21277 DragonFly Boot Enviroments (https://github.com/newnix/dfbeadm) This is a tool inspired by the beadm utility for FreeBSD/Illumos systems that creates and manages ZFS boot environments. This utility in contrast is written from the ground up in C, this should provide better performance, integration, and extensibility than the POSIX sh and awk script it was inspired by. During the time this project has been worked on, beadm has been superseded by bectl on FreeBSD. After hammering out some of the outstanding internal logic issues, I might look at providing a similar interface to the command as bectl. See link for the rest of the details Project Trident Updates 19.08 Available (https://project-trident.org/post/2019-08-15_19.08_available/) This is a general package update to the CURRENT release repository based upon TrueOS 19.08. Legacy boot ISO functional again This update includes the FreeBSD fixes for the “vesa” graphics driver for legacy-boot systems. The system can once again be installed on legacy-boot systems. PACKAGE CHANGES FROM 19.07-U1 New Packages: 154 Deleted Packages: 394 Updated Packages: 4926 12-U3 Available (https://project-trident.org/post/2019-08-22_stable12-u3_available/) This is the third general package update to the STABLE release repository based upon TrueOS 12-Stable. PACKAGE CHANGES FROM STABLE 12-U2 New Packages: 105 Deleted Packages: 386 Updated Packages: 1046 vBSDcon (https://www.vbsdcon.com/schedule/) vBSDcon 2019 will return to the Hyatt Regency in Reston, VA on September 5-7 2019. *** Beastie Bits The next NYCBUG meeting will be Sept 4 @ 18:45 (https://www.nycbug.org/index?action=view&id=10671) Feedback/Questions Tom - Questions (http://dpaste.com/1AXXK7G#wrap) Michael - dfbeadm (http://dpaste.com/0PNEDYT#wrap) Bostjan - Questions (http://dpaste.com/1N7T7BR#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
312: Why Package Managers
The UNIX Philosophy in 2019, why use package managers, touchpad interrupted, Porting wine to amd64 on NetBSD second evaluation report, Enhancing Syzkaller Support for NetBSD, all about the Pinebook Pro, killing a process and all of its descendants, fast software the best software, and more. Headlines The UNIX Philosophy in 2019 (https://triosdevelopers.com/jason.eckert/blog/Entries/2019/6/1_Entry_1.html) Today, Linux and open source rules the world, and the UNIX philosophy is widely considered compulsory. Organizations are striving to build small, focused applications that work collaboratively in a cloud and microservices environment. We rely on the network, as well as HTTP (text) APIs for storing and referencing data. Moreover, nearly all configuration is stored and communicated using text (e.g. YAML, JSON or XML). And while the UNIX philosophy has changed dramatically over the past 5 decades, it hasn’t strayed too far from Ken Thompson’s original definition in 1973: We write programs that do one thing and do it well We write programs to work together And we write programs that handle text streams, because that is a universal interface Why Use Package Managers? (https://uwm.edu/hpc/software-management/) Valuable research is often hindered or outright prevented by the inability to install software. This need not be the case. Since I began supporting research computing in 1999, I’ve frequently seen researchers struggle for days or weeks trying to install a single open source application. In most cases, they ultimately failed. In many cases, they could have easily installed the software in seconds with one simple command, using a package manager such as Debian packages, FreeBSD ports, MacPorts, or Pkgsrc, just to name a few. Developer websites often contain poorly written instructions for doing “caveman installs”; manually downloading, unpacking, patching, and building the software. The same laborious process must often be followed for other software packages on which it depends, which can sometimes number in the dozens. Many researchers are simply unaware that there are easier ways to install the software they need. Caveman installs are a colossal waste of man-hours. If 1000 people around the globe spend an average of 20 hours each trying to install the same program that could have been installed with a package manager (this is not uncommon), then 20,000 man-hours have been lost that could have gone toward science. How many important discoveries are delayed by this? The elite research institutions have ample funding and dozens of IT staff dedicated to research computing. They can churn out publications even if their operation is inefficient. Most institutions, however, have few or no IT staff dedicated to research, and cannot afford to squander precious man-hours on temporary, one-off software installs. The wise approach for those of us in that situation is to collaborate on making software deployment easier for everyone. If we do so, then even the smallest research groups can leverage that work to be more productive and make more frequent contributions to science. Fortunately, the vast majority of open source software installs can be made trivial for anyone to do for themselves. Modern package managers perform all the same steps as a caveman install, but automatically. Package managers also install dependencies for us automatically. News Roundup Touchpad, Interrupted (https://jcs.org/2019/07/28/ihidev) For two years I've been driving myself crazy trying to figure out the source of a driver problem on OpenBSD: interrupts never arrived for certain touchpad devices. A couple weeks ago, I put out a public plea asking for help in case any non-OpenBSD developers recognized the problem, but while debugging an unrelated issue over the weekend, I finally solved it. It's been a long journey and it's a technical tale, but here it is. Porting wine to amd64 on NetBSD, second evaluation report (https://blog.netbsd.org/tnf/entry/porting_wine_to_amd64_on2) Summary Presently, Wine on amd64 is in test phase. It seems to work fine with caveats like LDLIBRARYPATH which has to be set as 32-bit Xorg libs don't have ${PREFIX}/emul/netbsd32/lib in its rpath section. The latter is due to us extracting 32-bit libs from tarballs in lieu of building 32-bit Xorg on amd64. As previously stated, pkgsrc doesn't search for pkgconfig files in ${PREFIX}/emul/netbsd32/lib which might have inadvertent effects that I am unaware of as of now. I shall be working on these issues during the final coding period. I would like to thank @leot, @maya and @christos for saving me from shooting myself in the foot many a time. I, admittedly, have had times when multiple approaches, which all seemed right at that time, perplexed me. I believe those are times when having a mentor counts, and I have been lucky enough to have really good ones. Once again, thanks to Google for this wonderful opportunity. Enhancing Syzkaller Support for NetBSD, Part 2 (https://blog.netbsd.org/tnf/entry/enchancing_syzkaller_support_for_netbsd) As a part of Google Summer of Code’19, I am working on improving the support for Syzkaller kernel fuzzer. Syzkaller is an unsupervised coverage-guided kernel fuzzer, that supports a variety of operating systems including NetBSD. This report details the work done during the second coding period. You can also take a look at the first report to learn more about the initial support that we added. : https://blog.netbsd.org/tnf/entry/enhancingsyzkallersupportfornetbsd July Update: All about the Pinebook Pro (https://www.pine64.org/2019/07/05/july-update-all-about-the-pinebook-pro/) "So I said I won’t be talking about the BSDs, but I feel like I should at the very least give you a general overview of the RK3399 *BSD functionality. I’ll make it quick. I’ve spoken to *BSD devs whom worked on the RockPro64 and from what I’ve gathered (despite the different *BSDs having varying degree of support for the RK3399 SOC) many of the core features are already supported, which bodes well for *BSD on the Pro. That said, some of the things you’d require on a functional laptop – such as the LCD (using eDP) for instance – will not work on the Pinebook Pro using *BSD as of today. So clearly a degree of work is yet needed for a BSD to run on the device. However, keep in mind that *BSD developers will be receiving their units soon and by the time you receive yours some basic functionality may be available." Killing a process and all of its descendants (http://morningcoffee.io/killing-a-process-and-all-of-its-descendants.html) Killing processes in a Unix-like system can be trickier than expected. Last week I was debugging an odd issue related to job stopping on Semaphore. More specifically, an issue related to the killing of a running process in a job. Here are the highlights of what I learned: Unix-like operating systems have sophisticated process relationships. Parent-child, process groups, sessions, and session leaders. However, the details are not uniform across operating systems like Linux and macOS. POSIX compliant operating systems support sending signals to process groups with a negative PID number. Sending signals to all processes in a session is not trivial with syscalls. Child processes started with exec inherit their parent signal configuration. If the parent process is ignoring the SIGHUP signal, for example, this configuration is propagated to the children. The answer to the “What happens with orphaned process groups” question is not trivial. Fast Software, the Best Software (https://craigmod.com/essays/fast_software/) I love fast software. That is, software speedy both in function and interface. Software with minimal to no lag between wanting to activate or manipulate something and the thing happening. Lightness. Software that’s speedy usually means it’s focused. Like a good tool, it often means that it’s simple, but that’s not necessarily true. Speed in software is probably the most valuable, least valued asset. To me, speedy software is the difference between an application smoothly integrating into your life, and one called upon with great reluctance. Fastness in software is like great margins in a book — makes you smile without necessarily knowing why. But why is slow bad? Fast software is not always good software, but slow software is rarely able to rise to greatness. Fast software gives the user a chance to “meld” with its toolset. That is, not break flow. When the nerds upon Nerd Hill fight to the death over Vi and Emacs, it’s partly because they have such a strong affinity for the flow of the application and its meldiness. They have invested. The Tool Is Good, so they feel. Not breaking flow is an axiom of great tools. A typewriter is an excellent tool because, even though it’s slow in a relative sense, every aspect of the machine itself operates as quickly as the user can move. It is focused. There are no delays when making a new line or slamming a key into the paper. Yes, you have to put a new sheet of paper into the machine at the end of a page, but that action becomes part of the flow of using the machine, and the accumulation of paper a visual indication of work completed. It is not wasted work. There are no fundamental mechanical delays in using the machine. The best software inches ever closer to the physical directness of something like a typewriter. (The machine may break down, of course, ribbons need to be changed — but this is maintenance and separate from the use of the tool. I’d be delighted to “maintain” Photoshop if it would lighten it up.) Beastie Bits Register for vBSDCon 2019, Sept 5-7 in Reston VA (https://vbsdcon.com/registration) Register for EuroBSDCon 2019, Sept 19-22 in Lillehammer, Norway (https://2019.eurobsdcon.org/registration/) Feedback/Questions Paulo - FreeNAS Question (http://dpaste.com/2GDG7WR#wrap) Marc - Changing VT without function keys? (http://dpaste.com/1AKC7A1#wrap) Caleb - Patch, update, and upgrade management (http://dpaste.com/2D6J482#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
311: Conference Gear Breakdown
NetBSD 9.0 release process has started, xargs, a tale of two spellcheckers, Adapting TriforceAFL for NetBSD, Exploiting a no-name freebsd kernel vulnerability, and more. Headlines NetBSD 9.0 release process has started (https://mail-index.netbsd.org/netbsd-announce/2019/07/31/msg000301.html) If you have been following source-changes, you may have noticed the creation of the netbsd-9 branch! It has some really exciting items that we worked on: + New AArch64 architecture support: + Symmetric and asymmetrical multiprocessing support (aka big.LITTLE) + Support for running 32-bit binaries + UEFI and ACPI support + Support for SBSA/SBBR (server-class) hardware. + The FDT-ization of many ARM boards: + the 32-bit GENERIC kernel lists 129 different DTS configurations + the 64-bit GENERIC64 kernel lists 74 different DTS configurations + All supported by a single kernel, without requiring per-board configuration. + Graphics driver update, matching Linux 4.4, adding support for up to Kaby Lake based Intel graphics devices. + ZFS has been updated to a modern version and seen many bugfixes. + New hardware-accelerated virtualization via NVMM. + NPF performance improvements and bug fixes. A new lookup algorithm, thmap, is now the default. + NVMe performance improvements + Optional kernel ASLR support, and partial kernel ASLR for the default configuration. + Kernel sanitizers: + KLEAK, detecting memory leaks + KASAN, detecting memory overruns + KUBSAN, detecting undefined behaviour + These have been used together with continuous fuzzing via the syzkaller project to find many bugs that were fixed. + The removal of outdated networking components such as ISDN and all of its drivers + The installer is now capable of performing GPT UEFI installations. + Dramatically improved support for userland sanitizers, as well as the option to build all of NetBSD's userland using them for bug-finding. + Update to graphics userland: Mesa was updated to 18.3.4, and llvmpipe is now available for several architectures, providing 3D graphics even in the absence of a supported GPU. We try to test NetBSD as best as we can, but your testing can help NetBSD 9.0 a great release. Please test it and let us know of any bugs you find. + Binaries are available at https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/latest/ xargs wtf (https://medium.com/@aarontharris/xargs-wtf-34d2618286b7) xargs is probably one of the more difficult to understand of the unix command arsenal and of course that just means it’s one of the most useful too. I discovered a handy trick that I thought was worth a share. Please note there are probably other (better) ways to do this but I did my stackoverflow research and found nothing better. xargs — at least how I’ve most utilized it — is handy for taking some number of lines as input and doing some work per line. It’s hard to be more specific than that as it does so much else. It literally took me an hour of piecing together random man pages + tips from 11 year olds on stack overflow, but eventually I produced this gem: This is an example of how to find files matching a certain pattern and rename each of them. It sounds so trivial (and it is) but it demonstrates some cool tricks in an easy concept. News Roundup PkgSrc: A Tale of Two Spellcheckers (https://bentsukun.ch/posts/pkgsrccon-2019/) This is a transcript of the talk I gave at pkgsrcCon 2019 in Cambridge, UK. It is about spellcheckers, but there are much more general software engineering lessons that we can learn from this case study. The reason I got into this subject at all was my paternal leave last year, when I finally had some more time to spend working on pkgsrc. It was a tiny item in the enormous TODO file at the top of the source tree (“update enchant to version 2.2”) that made me go into this rabbit hole. Adapting TriforceAFL for NetBSD, Part 2 (https://blog.netbsd.org/tnf/entry/adapting_triforceafl_for_netbsd_part1) I have been working on adapting TriforceAFL for NetBSD kernel syscall fuzzing. This blog post summarizes the work done until the second evaluation. For work done during the first coding period, check out this post. Summary > So far, the TriforceNetBSDSyscallFuzzer has been made available in the form of a pkgsrc package with the ability to fuzz most of NetBSD syscalls. In the final coding period of GSoC. I plan to analyse the crashes that were found until now. Integrate sanitizers, try and find more bugs and finally wrap up neatly with detailed documentation. > Last but not least, I would like to thank my mentor, Kamil Rytarowski for helping me through the process and guiding me. It has been a wonderful learning experience so far! Exploiting a no-name freebsd kernel vulnerability (https://www.synacktiv.com/posts/exploit/exploiting-a-no-name-freebsd-kernel-vulnerability.html) A new patch has been recently shipped in FreeBSD kernels to fix a vulnerability (cve-2019-5602) present in the cdrom device. In this post, we will introduce the bug and discuss its exploitation on pre/post-SMEP FreeBSD revisions. > A closer look at the commit 6bcf6e3 shows that when invoking the CDIOCREADSUBCHANNEL_SYSSPACE ioctl, data are copied with bcopy instead of the copyout primitive. This endows a local attacker belonging to the operator group with an arbitrary write primitive in the kernel memory. [Allan and Benedicts Conference Gear Breakdown] Benedict’s Gear: GlocalMe G3 Mobile Travel HotSpot and Powerbank (https://www.glocalme.com/CA/en-US/cloudsim/g3) Mogics Power Bagel (http://www.mogics.com/3824-2) Charby Sense Power Cable (https://charbycharge.com/charby-sense-worlds-smartest-auto-cutoff-cable/) Allan’s Gear: Huawei E5770s-320 4G LTE 150 Mbps Mobile WiFi Pro (https://smile.amazon.com/gp/product/B013CEGGKI/) AOW Global Data SIM Card for On-Demand 4G LTE Mobile Data in Over 90 Countries (https://smile.amazon.com/dp/B071HJFX27/) All my devices charge from USB-C, so that is great More USB thumb drives than strictly necessary My Lenovo X270 laptop running FreeBSD 13-current My 2016 Macbook Pro (a prize from the raffle at vBSDCon 2017) that I use for email and video conferencing to preserve battery on my FreeBSD machine for work Beastie Bits Replacing the Unix tradition (Warning may be rage inducing) (https://www.youtube.com/watch?v=L9v4Mg8wi4U&feature=youtu.be) Installing OpenBSD over remote serial on the AtomicPI (https://www.thanassis.space/remoteserial.html#remoteserial) Zen 2 and DragonFly (https://www.dragonflydigest.com/2019/08/05/23294.html) Improve Docking on FreeBSD (https://blog.yukiisbo.red/posts/2019/05/improve-docking-on-freebsd/) Register for vBSDCon 2019, Sept 5-7 in Reston VA. Early bird ends August 15th. (https://vbsdcon.com/registration) Register for EuroBSDCon 2019, Sept 19-22 in Lillehammer, Norway (https://2019.eurobsdcon.org/registration/) Feedback/Questions JT - Congrats (http://dpaste.com/0D7Y31E#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
310: My New Free NAS
OPNsense 19.7.1 is out, ZFS on Linux still has annoying issues with ARC size, Hammer2 is now default, NetBSD audio – an application perspective, new FreeNAS Mini, and more. Headlines OPNsense 19.7.1 (https://opnsense.org/opnsense-19-7-1-released/) We do not wish to keep you from enjoying your summer time, but this is a recommended security update enriched with reliability fixes for the new 19.7 series. Of special note are performance improvements as well as a fix for a longstanding NAT before IPsec limitation. Full patch notes: system: do not create automatic copies of existing gateways system: do not translate empty tunables descriptions system: remove unwanted form action tags system: do not include Syslog-ng in rc.freebsd handler system: fix manual system log stop/start/restart system: scoped IPv6 "%" could confuse mwexecf(), use plain mwexec() instead system: allow curl-based downloads to use both trusted and local authorities system: fix group privilege print and correctly redirect after edit system: use cached address list in referrer check system: fix Syslog-ng search stats firewall: HTML-escape dynamic entries to display aliases firewall: display correct IP version in automatic rules firewall: fix a warning while reading empty outbound rules configuration firewall: skip illegal log lines in live log interfaces: performance improvements for configurations with hundreds of interfaces reporting: performance improvements for Python 3 NetFlow aggregator rewrite dhcp: move advanced router advertisement options to correct config section ipsec: replace global array access with function to ensure side-effect free boot ipsec: change DPD action on start to "dpdaction = restart" ipsec: remove already default "dpdaction = none" if not set ipsec: use interface IP address in local ID when doing NAT before IPsec web proxy: fix database reset for Squid 4 by replacing use of sslcrtd with securityfile_certgen plugins: os-acme-client 1.24[1] plugins: os-bind 1.6[2] plugins: os-dnscrypt-proxy 1.5[3] plugins: os-frr now restricts characters BGP prefix-list and route-maps[4] plugins: os-google-cloud-sdk 1.0[5] ports: curl 7.65.3[6] ports: monit 5.26.0[7] ports: openssh 8.0p1[8] ports: php 7.2.20[9] ports: python 3.7.4[10] ports: sqlite 3.29.0[11] ports: squid 4.8[12] Stay safe and hydrated, Your OPNsense team ZFS on Linux still has annoying issues with ARC size (https://utcc.utoronto.ca/~cks/space/blog/linux/ZFSOnLinuxARCShrinkage) One of the frustrating things about operating ZFS on Linux is that the ARC size is critical but ZFS's auto-tuning of it is opaque and apparently prone to malfunctions, where your ARC will mysteriously shrink drastically and then stick there. Linux's regular filesystem disk cache is very predictable; if you do disk IO, the cache will relentlessly grow to use all of your free memory. This sometimes disconcerts people when free reports that there's very little memory actually free, but at least you're getting value from your RAM. This is so reliable and regular that we generally don't think about 'is my system going to use all of my RAM as a disk cache', because the answer is always 'yes'. (The general filesystem cache is also called the page cache.) This is unfortunately not the case with the ZFS ARC in ZFS on Linux (and it wasn't necessarily the case even on Solaris). ZFS has both a current size and a 'target size' for the ARC (called 'c' in ZFS statistics). When your system boots this target size starts out as the maximum allowed size for the ARC, but various events afterward can cause it to be reduced (which obviously limits the size of your ARC, since that's its purpose). In practice, this reduction in the target size is both pretty sticky and rather mysterious (as ZFS on Linux doesn't currently expose enough statistics to tell why your ARC target size shrunk in any particular case). The net effect is that the ZFS ARC is not infrequently quite shy and hesitant about using memory, in stark contrast to Linux's normal filesystem cache. The default maximum ARC size starts out as only half of your RAM (unlike the regular filesystem cache, which will use all of it), and then it shrinks from there, sometimes very significantly, and once shrunk it only recovers slowly (if at all). News Roundup Hammer2 is now default (http://lists.dragonflybsd.org/pipermail/commits/2019-June/718989.html) ``` commit a49112761c919d42d405ec10252eb0553662c824 Author: Matthew Dillon Date: Mon Jun 10 17:53:46 2019 -0700 installer - Default to HAMMER2 * Change the installer default from HAMMER1 to HAMMER2. * Adjust the nrelease build to print the location of the image files when it finishes. Summary of changes: nrelease/Makefile | 2 +- usr.sbin/installer/dfuibe_installer/flow.c | 20 ++++++++++---------- 2 files changed, 11 insertions(+), 11 deletions(-) http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/a49112761c919d42d405ec10252eb0553662c824 ``` NetBSD audio – an application perspective (https://netbsd.org/gallery/presentations/nia/netbsd-audio/) NetBSD audio – an application perspective ... or, "doing it natively, because we can" audio options for NetBSD in pkgsrc Use NetBSD native audio (sun audio/audioio.h) Or OSS emulation layer: Basically a wrapper around sun audio in the kernel. Incomplete and old version, but works for simple stuff Many many abstraction layers available: OpenAL-Soft alsa-lib (config file required) libao, GStreamer (plugins!) PortAudio, SDL PulseAudio, JACK ... lots more!? some obsolete stuff (esd, nas?) Advantages of using NetBSD audio directly Low latency, low CPU usage: Abstraction layers differ in latency (SDL2 vs ALSA/OpenAL) Query device information: Is /dev/audio1 a USB microphone or another sound card? Avoid bugs from excessive layering Nice API, well documented: [nia note: I had no idea how to write audio code. I read a man page and now I do.] Your code might work on illumos too [nia note: SDL2 seems very sensitive to the blk_ms sysctl being high or low, with other implementations there seems to be a less noticable difference. I don't know why.] New FreeNAS Mini (https://www.ixsystems.com/blog/new-freenas-mini-models-release-pr/) Two new FreeNAS Mini systems join the very popular FreeNAS Mini and Mini XL: FreeNAS Mini XL+: This powerful 10 Bay platform (8x 3.5” and 1x 2.5” hot-swap, 1x 2.5” internal) includes the latest, compact server technology and provides dual 10GbE ports, 8 CPU cores and 32 GB RAM for high performance workgroups. The Mini XL+ scales beyond 100TB and is ideal for very demanding applications, including hosting virtual machines and multimedia editing. Starting at $1499, the Mini XL+ configured with cache SSD and 80 TB capacity is $4299, and consumes about 100 Watts. FreeNAS Mini E: This cost-effective 4 Bay platform provides the resources required for SOHO use with quad GbE ports and 8 GB of RAM. The Mini E is ideal for file sharing, streaming and transcoding video at 1080p. Starting at $749, the Mini E configured with 8 TB capacity is $999, and consumes about 36 Watts. Beastie Bits Welcome to NetBSD 9.99.1! (https://mail-index.netbsd.org/source-changes/2019/07/30/msg107671.html) Berkeley smorgasbord — part II (http://blog.snailtext.com/posts/berkeley-smorgasbord-part-2.html) dtracing postgres (https://www.youtube.com/watch?v=Brt41xnMZqo&list=PLuJmmKtsV1dOTmlImlD9U5j1P1rLxS2V8&index=20&t=0s) Project Trident 19.07-U1 now available (https://project-trident.org/post/2019-07-30_19.07-u1_available/) Need a Secure Operating System? Take a Look at OpenBSD (https://www.devprojournal.com/technology-trends/operating-systems/need-a-secure-operating-system-take-a-look-at-openbsd/) Feedback/Questions Jeff - OpenZFS Port Testing Feedback (http://dpaste.com/2AT7JGP#wrap) Malcolm - Best Practices for Custom Ports (http://dpaste.com/1R170D7) Michael - Little Correction (http://dpaste.com/0CERP6R) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
Episode 309: Get Your Telnet Fix
DragonFlyBSD Project Update - colo upgrade, future trends, resuming ZFS send, realtime bandwidth terminal graph visualization, fixing telnet fixes, a chapter from the FBI’s history with OpenBSD and an OpenSSH vuln, and more. Headlines DragonFlyBSD Project Update - colo upgrade, future trends (http://lists.dragonflybsd.org/pipermail/users/2019-July/358226.html) For the last week I've been testing out a replacement for Monster, our 48-core opteron server. The project will be removing Monster from the colo in a week or two and replacing it with three machines which together will use half the power that Monster did alone. The goal is to clear out a little power budget in the colo and to really beef-up our package-building capabilities to reduce the turn-around time needed to test ports syncs and updates to the binary package system. Currently we use two blades to do most of the building, plus monster sometimes. The blades take almost a week (120 hours+) to do a full synth run and monster takes around 27.5 hours. But we need to do three bulk builds more or less at the same time... one for the release branch, one for the development branch, and one for staging updates. It just takes too long and its been gnawing at me for a little while. Well, Zen 2 to the rescue! These new CPUs can take ECC, there's actually an IPMI mobo available, and they are fast as hell and cheap for what we get. The new machines will be two 3900X based servers, plus a dual-xeon system that I already had at home. The 3900X's can each do a full synth run in 24.5 hours and the Xeon can do it in around 31 hours. Monster will be retired. And the crazy thing about this? Monster burns 1000W going full bore. Each of the 3900X servers burns 160W and the Xeon burns 200W. In otherwords, we are replacing 1000W with only 520W and getting roughly 6x the performance efficiency in the upgrade. This tell you just how much more power-efficient machines have become in the last 9 years or so. > This upgrade will allow us to do full builds for both release and dev in roughly one day instead of seven days, and do it without interfering with staging work that might be happening at the same time. Future trends - DragonFlyBSD has reached a bit of a cross-roads. With most of the SMP work now essentially complete across the entire system the main project focus is now on supplying reliable binary ports for release and developer branches, DRM (GPU) support and other UI elements to keep DragonFlyBSD relevant on workstations, and continuing Filesystem work on HAMMER2 to get multi-device and clustering going. Resuming ZFS send (https://www.oshogbo.vexillium.org/blog/66/) One of the amazing functionalities of ZFS is the possibility of sending a whole dataset from one place to another. This mechanism is amazing to create backups of your ZFS based machines. Although, there were some issues with this functionality for a long time when a user sent a big chunk of data. What if you would do that over the network and your connection has disappeared? What if your machine was rebooted as you are sending a snapshot? For a very long time, you didn't have any options - you had to send a snapshot from the beginning. Now, this limitation was already bad enough. However, another downside of this approach was that all the data which you already send was thrown away. Therefore, ZFS had to go over all this data and remove them from the dataset. Imagine the terabytes of data which you sent via the network was thrown away because as you were sending the last few bytes, the network went off. In this short post, I don't want to go over the whole ZFS snapshot infrastructure (if you think that such a post would be useful, please leave a comment). Now, to get back to the point, this infrastructure is used to clone the datasets. Some time ago a new feature called “Resuming ZFS send” was introduced. That means that if there was some problem with transmitting the dataset from one point to another you could resume it or throw them away. But the point is, that yes, you finally have a choice. News Roundup Realtime bandwidth terminal graph visualization (https://dataswamp.org/~solene/2019-07-19-ttyplot-netstat-openbsd.html) If for some reasons you want to visualize your bandwidth traffic on an interface (in or out) in a terminal with a nice graph, here is a small script to do so, involving ttyplot, a nice software making graphics in a terminal. The following will works on OpenBSD. You can install ttyplot by pkg_add ttyplot as root, ttyplot package appeared since OpenBSD 6.5. fixing telnet fixes (https://flak.tedunangst.com/post/fixing-telnet-fixes) There’s a FreeBSD commit to telnet. fix a couple of snprintf() buffer overflows. It’s received a bit of attention for various reasons, telnet in 2019?, etc. I thought I’d take a look. Here’s a few random observations. The first line is indented with spaces while the others use tabs. The correct type for string length is size_t not unsigned int. sizeof(char) is always one. There’s no need to multiply by it. If you do need to multiply by a size, this is an unsafe pattern. Use calloc or something similar. (OpenBSD provides reallocarray to avoid zeroing cost of calloc.) Return value of malloc doesn’t need to be cast. In fact, should not be, lest you disguise a warning. Return value of malloc is not checked for NULL. No reason to cast cp to char * when passing to snprintf. It already is that type. And if it weren’t, what are you doing? The whole operation could be simplified by using asprintf. Although unlikely (probably impossible here, but more generally), adding the two source lengths together can overflow, resulting in truncation with an unchecked snprintf call. asprintf avoids this failure case. A Chapter from the FBI’s History with OpenBSD and an OpenSSH Vuln (https://twitter.com/RooneyMcNibNug/status/1152327783055601664) Earlier this year I FOIAed the FBI for details on allegations of backdoor installed in the IPSEC stack in 2010, originally discussed by OpenBSD devs (https://marc.info/?l=openbsd-tech&m=129236621626462 …) Today, I got an interesting but unexpected responsive record: Freedom of Information Act: FBI: OpenBSD (https://www.muckrock.com/foi/united-states-of-america-10/foia-fbi-openbsd-70084/) GitHub Repo (https://github.com/RooneyMcNibNug/FOIA/blob/master/Responsive%20Docs/OpenBSD/FBI_OpenBSD_response_OCRd.pdf) Beastie Bits “Sudo Mastery, 2nd Edition” open for tech review (https://mwl.io/archives/4378) FreeBSD Journal: FreeBSD for Makers (https://www.freebsdnews.com/2019/07/12/freebsd-journal-freebsd-for-makers/) OpenBSD and NetBSD machines at Open Source Conference 2019 Nagoya (http://mail-index.netbsd.org/netbsd-advocacy/2019/07/19/msg000808.html) FreeBSD 12.0: WINE Gaming (https://www.youtube.com/watch?v=zuj9pRNR2oM) Introduction to the Structure and Interpretation of TNF (The NetBSD Foundation) (https://www.netbsd.org/gallery/presentations/wiz/pkgsrccon2019/index.html#/) vBSDcon speakers announced (https://www.vbsdcon.com/) Feedback/Questions Pat - NYCBug Aug 7th (http://dpaste.com/21Y1PRM) Tyler - SSH keys vs password (http://dpaste.com/3JEVVEF#wrap) Lars - Tor-Talk (http://dpaste.com/0RAFMXZ) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
308: Mumbling with OpenBSD
Replacing a (silently) failing disk in a ZFS pool, OPNsense 19.7 RC1 released, implementing DRM ioctl support for NetBSD, High quality/low latency VOIP server with umurmur/Mumble on OpenBSD, the PDP-7 where Unix began, LLDB watchpoints, and more. Headlines Replacing a (silently) failing disk in a ZFS pool (https://imil.net/blog/2019/07/02/Replacing-a-silently-failing-disk-in-a-ZFS-pool/) Maybe I can’t read, but I have the feeling that official documentations explain every single corner case for a given tool, except the one you will actually need. My today’s struggle: replacing a disk within a FreeBSD ZFS pool. What? there’s a shitton of docs on this topic! Are you stupid? I don’t know, maybe. Yet none covered the process in a simple, straight and complete manner. OPNsense 19.7 RC1 released (https://opnsense.org/opnsense-19-7-rc1-released/) Hi there, For four and a half years now, OPNsense is driving innovation through modularising and hardening the open source firewall, with simple and reliable firmware upgrades, multi-language support, HardenedBSD security, fast adoption of upstream software updates as well as clear and stable 2-Clause BSD licensing. We thank all of you for helping test, shape and contribute to the project! We know it would not be the same without you. Download links, an installation guide[1] and the checksums for the images can be found below as well. News Roundup Implementation of DRM ioctl Support for NetBSD kernel (https://blog.netbsd.org/tnf/entry/implementation_of_drm_ioctl_support) What is DRM ioctl ? Ioctls are input/output control system calls and DRM stands for direct rendering manager The DRM layer provides several services to graphics drivers, many of them driven by the application interfaces it provides through libdrm, the library that wraps most of the DRM ioctls. These include vblank event handling, memory management, output management, framebuffer management, command submission & fencing, suspend/resume support, and DMA services. Native DRM ioctl calls NetBSD was able to make native DRM ioctl calls with hardware rendering once xorg and proper mesa packages where installed. We used the glxinfo and glxgears applications to test this out. High quality / low latency VOIP server with umurmur/Mumble on OpenBSD (https://dataswamp.org/~solene/2019-07-04-umurmur.html) Discord users keep telling about their so called discord server, which is not dedicated to them at all. And Discord has a very bad quality and a lot of voice distorsion. Why not run your very own mumble server with high voice quality and low latency and privacy respect? This is very easy to setup on OpenBSD! Mumble is an open source voip client, it has a client named Mumble (available on various operating system) and at least Android, the server part is murmur but there is a lightweight server named umurmur. People authentication is done through certificate generated locally and automatically accepted on a server, and the certificate get associated with a nickname. Nobody can pick the same nickname as another person if it’s not the same certificate. TMWL June’19 — JS Fetch API, scheduling in Spring, thoughts on Unix (https://blog.softwaremill.com/tmwl-june19-js-fetch-api-scheduling-in-spring-thoughts-on-unix-fd54f50ecd64) Unix — going back to the roots From time to time, I like to review my knowledge in a certain area, even when I feel like I know a lot about it already. I go back to the basics and read tutorials, manuals, books or watch interesting videos. I’ve been using macOS for a couple of years now, previously being a linux user for some (relatively short) time. Both these operating systems have a common ancestor — Unix. While I’m definitely not an expert, I feel quite comfortable using linux & macOS — I understand the concepts behind the system architecture, know a lot of command line tools & navigate through the shell without a hassle. So-called unix philosophy is also close to my heart. I always feel like there’s more I could squeeze out of it. Recently, I found that book titled “Unix for dummies, 5th edition” which was published back in… 2004. Feels literally like AGES in the computer-related world. However, it was a great shot — the book starts with the basics, providing some brief history of Unix and how it came to life. It talks a lot about the structure of the system and where certain pieces fit (eg. “standard” set of tools), and how to understand permissions and work with files & directories. There’s even a whole chapter about shell-based text editors like Vi and Emacs! Despite the fact that I am familiar with most of these, I could still find some interesting pieces & tools that I either knew existed (but never had a chance to use), or even haven’t ever heard of. And almost all of these are still valid in the modern “incarnations” of Unix’s descendants: Linux and macOS. The book also talks about networking, surfing the web & working with email. It’s cute to see pictures of those old browsers rendering “ancient” Internet websites, but hey — this is how it looked like no more than fifteen years ago! I can really recommend this book to anyone working on modern macOS or Linux — you will certainly find some interesting pieces. Especially if you like to go back to the roots from time to time as I do! ThePDP-7 Where Unix Began (https://bsdimp.blogspot.com/2019/07/the-pdp-7-where-unix-began.html) In preparation for a talk on Seventh Edition Unix this fall, I stumbled upon a service list from DEC for all known PDP-7 machines. From that list, and other sources, I believe that PDP-7 serial number 34 was the original Unix machine. V0 Unix could run on only one of the PDP-7s. Of the 99 PDP-7s produced, only two had disks. Serial number 14 had an RA01 listed, presumably a disk, though of a different type. In addition to the PDP-7 being obsolete in 1970, no other PDP-7 could run Unix, limiting its appeal outside of Bell Labs. By porting Unix to the PDP-11 in 1970, the group ensured Unix would live on into the future. The PDP-9 and PDP-15 were both upgrades of the PDP-7, so to be fair, PDP-7 Unix did have a natural upgrade path (the PDP-11 out sold the 18 bit systems though ~600,000 to ~1000). Ken Thompson reports in a private email that there were 2 PDP-9s and 1 PDP-15 at Bell Labs that could run a version of the PDP-7 Unix, though those machines were viewed as born obsolete. LLDB: watchpoints, XSTATE in ptrace() and core dumps (https://blog.netbsd.org/tnf/entry/lldb_watchpoints_xstate_in_ptrace) Upstream describes LLDB as a next generation, high-performance debugger. It is built on top of LLVM/Clang toolchain, and features great integration with it. At the moment, it primarily supports debugging C, C++ and ObjC code, and there is interest in extending it to more languages. In February, I have started working on LLDB, as contracted by the NetBSD Foundation. So far I've been working on reenabling continuous integration, squashing bugs, improving NetBSD core file support and lately extending NetBSD's ptrace interface to cover more register types and fix compat32 issues. You can read more about that in my May 2019 report. In June, I have finally finished the remaining ptrace() work for xstate and got it merged both on NetBSD and LLDB end (meaning it's going to make it into NetBSD 9). I have also worked on debug register support in LLDB, effectively fixing watchpoint support. Once again I had to fight some upstream regressions. Beastie Bits Project Trident 19.07 Available (https://project-trident.org/post/2019-07-12_19.07_available/) A list of names from "Cold Blood" -- Any familiar? (https://www.montanalinux.org/cold-blood-list-of-numbers-201907.html) fern: a curses-based mastodon client modeled off usenet news readers & pine, with an emphasis on getting to 'timeline zero' (https://github.com/enkiv2/fern) OpenBSD Community goes Platinum for 2019! (https://undeadly.org/cgi?action=article;sid=20190707065226) tcp keepalive and dports on DragonFly (https://www.dragonflydigest.com/2019/07/15/23199.html) Feedback/Questions Patrick - OpenZFS/ZoL Module from Ports (http://dpaste.com/1W2HJ04) Brad - Services not starting (http://dpaste.com/345VM9Y#wrap) Simon - Feedback (http://dpaste.com/1B4ZKC8#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
307: Twitching with OpenBSD
FreeBSD 11.3 has been released, OpenBSD workstation, write your own fuzzer for the NetBSD kernel, Exploiting FreeBSD-SA-19:02.fd, streaming to twitch using OpenBSD, 3 different ways of dumping hex contents of a file, and more. Headlines FreeBSD 11.3-RELEASE Announcement (https://www.freebsd.org/releases/11.3R/announce.html) The FreeBSD Release Engineering Team is pleased to announce the availability of FreeBSD 11.3-RELEASE. This is the fourth release of the stable/11 branch. Some of the highlights: The clang, llvm, lld, lldb, and compiler-rt utilities as well as libc++ have been updated to upstream version 8.0.0. The ELF Tool Chain has been updated to version r3614. OpenSSL has been updated to version 1.0.2s. The ZFS filesystem has been updated to implement parallel mounting. The loader(8) has been updated to extend geli(8) support to all architectures. The pkg(8) utility has been updated to version 1.10.5. The KDE desktop environment has been updated to version 5.15.3. The GNOME desktop environment has been updated to version 3.28. The kernel will now log the jail(8) ID when logging a process exit. Several feature additions and updates to userland applications. Several network driver firmware updates. Warnings for features deprecated in future releases will now be printed on all FreeBSD versions. Warnings have been added for IPSec algorithms deprecated in RFC 8221. Deprecation warnings have been added for weaker algorithms when creating geli(8) providers. And more... OpenBSD Is Now My Workstation (https://sogubsys.com/openbsd-is-now-my-workstation-operating-system/) Why OpenBSD? Simply because it is the best tool for the job for me for my new-to-me Lenovo Thinkpad T420. Additionally, I do care about security and non-bloat in my personal operating systems (business needs can have different priorities, to be clear). I will try to detail what my reasons are for going with OpenBSD (instead of GNU/Linux, NetBSD, or FreeBSD of which I’m comfortable using without issue), challenges and frustrations I’ve encountered, and what my opinions are along the way. Disclaimer: in this post, I’m speaking about what is my opinion, and I’m not trying to convince you to use OpenBSD or anything else. I don’t truly care, but wanted to share in case it could be useful to you. I do hope you give OpenBSD a shot as your workstation, especially if it has been a while. A Bit About Me and OpenBSD I’m not new to OpenBSD, to be clear. I’ve been using it off and on for over 20 years. The biggest time in my life was the early 2000s (I was even the Python port maintainer for a bit), where I not only used it for my workstation, but also for production servers and network devices. I just haven’t used it as a workstation (outside of a virtual machine) in over 10 years, but have used it for servers. Workstation needs, especially for a primary workstation, are greatly different and the small things end up mattering most. News Roundup Write your own fuzzer for NetBSD kernel! [Part 1] (https://blog.netbsd.org/tnf/entry/write_your_own_fuzzer_for) How Fuzzing works? The dummy Fuzzer. The easy way to describe fuzzing is to compare it to the process of unit testing a program, but with different input. This input can be random, or it can be generated in some way that makes it unexpected form standard execution perspective. The simplest 'fuzzer' can be written in few lines of bash, by getting N bytes from /dev/rand, and putting them to the program as a parameter. Coverage and Fuzzing What can be done to make fuzzing more effective? If we think about fuzzing as a process, where we place data into the input of the program (which is a black box), and we can only interact via input, not much more can be done. However, programs usually process different inputs at different speeds, which can give us some insight into the program's behavior. During fuzzing, we are trying to crash the program, thus we need additional probes to observe the program's behaviour. Additional knowledge about program state can be exploited as a feedback loop for generating new input vectors. Knowledge about the program itself and the structure of input data can also be considered. As an example, if the input data is in the form of HTML, changing characters inside the body will probably cause less problems for the parser than experimenting with headers and HTML tags. For open source programs, we can read the source code to know what input takes which execution path. Nonetheless, this might be very time consuming, and it would be much more helpful if this can be automated. As it turns out, this process can be improved by tracing coverage of the execution vBSDcon - CFP - Call for Papers ends July 19th (https://vbsdcon.com/) You can submit your proposal at https://easychair.org/conferences/?conf=vbsdcon2019 The talks will have a very strong technical content bias. Proposals of a business development or marketing nature are not appropriate for this venue. If you are doing something interesting with a BSD operating system, please submit a proposal. Whether you are developing a very complex system using BSD as the foundation, or helping others and have a story to tell about how BSD played a role, we want to hear about your experience. People using BSD as a platform for research are also encouraged to submit a proposal. Possible topics include: How we manage a giant installation with respect to handling spam, snd/or sysadmin, and/or networking, Cool new stuff in BSD, Tell us about your project which runs on BSD. Both users and developers are encouraged to share their experiences. Exploiting FreeBSD-SA-19:02.fd (https://secfault-security.com/blog/FreeBSD-SA-1902.fd.html) In February 2019 the FreeBSD project issued an advisory about a possible vulnerability in the handling of file descriptors. UNIX-like systems such as FreeBSD allow to send file descriptors to other processes via UNIX-domain sockets. This can for example be used to pass file access privileges to the receiving process. Inside the kernel, file descriptors are used to indirectly reference a C struct which stores the relevant information about the file object. This could for instance include a reference to a vnode which describes the file for the file system, the file type, or the access privileges. What really happens if a UNIX-domain socket is used to send a file descriptor to another process is that for the receiving process, inside the kernel a reference to this struct is created. As the new file descriptor is a reference to the same file object, all information is inherited. For instance, this can allow to give another process write access to a file on the drive even if the process owner is normally not able to open the file writable. The advisory describes that FreeBSD 12.0 introduced a bug in this mechanism. As the file descriptor information is sent via a socket, the sender and the receiver have to allocate buffers for the procedure. If the receiving buffer is not large enough, the FreeBSD kernel attempts to close the received file descriptors to prevent a leak of these to the sender. However, while the responsible function closes the file descriptor, it fails to release the reference from the file descriptor to the file object. This could cause the reference counter to wrap. The advisory further states that the impact of this bug is possibly a local privilege escalation to gain root privileges or a jail escape. However, no proof-of-concept was provided by the advisory authors. In the next section, the bug itself is analyzed to make a statement about the bug class and a guess about a possible exploitation primitive. After that, the bug trigger is addressed. It follows a discussion of three imaginable exploitation strategies - including a discussion of why two of these approaches failed. In the section before last, the working exploit primitive is discussed. It introduces a (at least to the author’s knowledge) new exploitation technique for these kind of vulnerabilities in FreeBSD. The stabilization of the exploit is addressed, too. The last section wraps everything up in a conclusion and points out further steps and challenges. The privilege escalation is now a piece of cake thanks to a technique used by kingcope, who published a FreeBSD root exploit in 2005, which writes to the file /etc/libmap.conf. This configuration file can be used to hook the loading of dynamic libraries if a program is started. The exploit therefore creates a dynamic library, which copies /bin/sh to another file and sets the suid-bit for the copy. The hooked library is libutil, which is for instance called by su. Therefore, a call to su by the user will afterwards result in a suid copy of /bin/sh. Streaming to Twitch using OpenBSD (https://dataswamp.org/~solene/2019-07-06-twitch.html) Introduction If you ever wanted to make a twitch stream from your OpenBSD system, this is now possible, thanks to OpenBSD developer thfr@ who made a wrapper named fauxstream using ffmpeg with relevant parameters. The setup is quite easy, it only requires a few steps and searching on Twitch website two informations, hopefully, to ease the process, I found the links for you. You will need to make an account on twitch, get your api key (a long string of characters) which should stay secret because it allow anyone having it to stream on your account. These same techniques should work for Twitch, YouTube Live, Periscope, Facebook, etc, including the live streaming service ScaleEngine provides free to BSD user groups. There is also an open source application called ‘OBS’ or Open Broadcaster Studio. It is in FreeBSD ports and should work on all of the other BSDs as well. It has a GUI and supports compositing and green screening. We use it heavily at ScaleEngine and it is also used at JupiterBroadcasting in place of WireCast, a $1000-per-copy commercial application. Beastie Bits Portland BSD Pizza Night - 2019-07-25 19:00 - Rudy's Gourmet Pizza (http://calagator.org/events/1250475868) KnoxBUG - Michael W. Lucas : Twenty Years in Jail (http://knoxbug.org/2019-07-29) Ohio Linuxfest - CFP - Closes August 17th (https://ohiolinux.org/call-for-presentations/) My college (NYU Tandon) is moving their CS department and I saw this on a shelf being moved (https://old.reddit.com/r/freebsd/comments/cdx8fp/my_college_nyu_tandon_is_moving_their_cs/) 3 different ways of dumping hex contents of a file (https://moopost.blogspot.com/2019/07/3-different-ways-of-dumping-hex.html) Feedback/Questions Sebastian - ZFS setup toward ESXi (http://dpaste.com/0DRKFH6#wrap) Christopher - Questions (http://dpaste.com/2YNN1SH) Ser - Bhyve and Microsoft SQL (http://dpaste.com/1F5TMT0#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
306: Comparing Hammers
Am5x86 based retro UNIX build log, setting up services in a FreeNAS Jail, first taste of DragonflyBSD, streaming Netflix on NetBSD, NetBSD on the last G4 Mac mini, Hammer vs Hammer2, and more. Headlines Polprog's Am5x86 based retro UNIX build log (https://polprog.net/blog/486/) I have recently acquired an Am5x86 computer, in a surprisingly good condition. This is an ongoing project, check this page often for updates! I began by connecting a front panel. The panel came from a different chassis and is slightly too wide, so I had to attach it with a couple of zip-ties. However, that makes it stick out from the PC front at an angle, allowing easy access when the computer sits at the floor - and thats where it is most of the time. It's not that bad, to be honest, and its way easier to access than it would be, if mounted vertically There is a mains switch on the front panel because the computer uses an older style power supply. Those power supplies instead of relying on a PSON signal, like modern ATX supplies, run a 4 wire cable to a mains switch. The cable carries live and neutral both ways, and the switch keys in or out the power. The system powers on as soon as the switch is enabled. Originally there was no graphics card in it. Since a PC will not boot with out a GPU, I had to find one. The mainboard only has PCI and ISA slots, and all the GPUs I had were AGP. Fortunately, I bought a PCI GPU hoping it would solve my issue... However the GPU turned out to be faulty. It took me some time to repair it. I had to repair a broken trace leading to one of the EEPROM pins, and replace a contact in the EEPROM's socket. Then I replaced all the electrolytic capacitors on it, and that fixed it for good. Having used up only one of the three PCI slots, I populated the remaining pair with two ethernet cards. I still have a bunch of ISA slots available, but I have nothing to install there. Yet. See the article for the rest of the writeup Setting up services in a FreeNAS Jail (https://www.ixsystems.com/blog/services-in-freenas-jail/) This piece demonstrates the setup of a server service in a FreeNAS jail and how to share files with a jail using Apache 2.4 as an example. Jails are powerful, self-contained FreeBSD environments with separate network settings, package management, and access to thousands of FreeBSD application packages. Popular packages such as Apache, NGINX, LigHTTPD, MySQL, and PHP can be found and installed with the pkg search and pkg install commands. This example shows creating a jail, installing an Apache web server, and setting up a simple web page. NOTE: Do not directly attach FreeNAS to an external network (WAN). Use port forwarding, proper firewalls and DDoS protections when using FreeNAS for external web sites. This example demonstrates expanding the functionality of FreeNAS in an isolated LAN environment. News Roundup First taste of DragonflyBSD (https://nanxiao.me/en/first-taste-of-dragonfly-bsd/) Last week, I needed to pick a BSD Operating System which supports NUMA to do some testing, so I decided to give Dragonfly BSD a shot. Dragonfly BSDonly can run on X86_64 architecture, which reminds me of Arch Linux, and after some tweaking, I feel Dragonfly BSD may be a “developer-friendly” Operating System, at least for me. I mainly use Dragonfly BSD as a server, so I don’t care whether GUI is fancy or not. But I have high requirements of developer tools, i.e., compiler and debugger. The default compiler of Dragonfly BSD is gcc 8.3, and I can also install clang 8.0.0 from package. This means I can test state-of-the-art features of compilers, and it is really important for me. gdb‘s version is 7.6.1, a little lag behind, but still OK. Furthermore, the upgradation of Dragonfly BSD is pretty simple and straightforward. I followed document to upgrade my Operating System to 5.6.0 this morning, just copied and pasted, no single error, booted successfully. Streaming Netflix on NetBSD (https://www.unitedbsd.com/d/68-streaming-netflix-on-netbsd) Here's a step-by-step guide that allows streaming Netflix media on NetBSD using a intel-haxm accelerated QEMU vm. Heads-up! Sound doesn't work, but everything else is fine. Please read the rest of this thread for a solution to this!! “Sudo Mastery 2nd Edition” cover art reveal (https://mwl.io/archives/4320) I’m about halfway through the new edition of Sudo Mastery. Assuming nothing terrible happens, should have a complete first draft in four to six weeks. Enough stuff has changed in sudo that I need to carefully double-check every single feature. (I’m also horrified by the painfully obsolete versions of sudo shipped in the latest versions of CentOS and Debian, but people running those operating systems are already accustomed to their creaky obsolescence.) But the reason for this blog post? I have Eddie Sharam’s glorious cover art. My Patronizers saw it last month, so now the rest of you get a turn. NetBSD on the last G4 Mac mini (https://tenfourfox.blogspot.com/2019/06/and-now-for-something-completely.html) I'm a big fan of NetBSD. I've run it since 2000 on a Mac IIci (of course it's still running it) and I ran it for several years on a Power Mac 7300 with a G3 card which was the second incarnation of the Floodgap gopher server. Today I also still run it on a MIPS-based Cobalt RaQ 2 and an HP Jornada 690. I think NetBSD is a better match for smaller or underpowered systems than current-day Linux, and is fairly easy to harden and keep secure even though none of these systems are exposed to the outside world. Recently I had a need to set up a bridge system that would be fast enough to connect two networks and I happened to have two of the "secret" last-of-the-line 1.5GHz G4 Mac minis sitting on the shelf doing nothing. Yes, they're probably outclassed by later Raspberry Pi models, but I don't have to buy anything and I like putting old hardware to good use. Hammer vs Hammer2 (https://phoronix.com/scan.php?page=news_item&px=DragonFlyBSD-5.6-HAMMER2-Perf) With the newly released DragonFlyBSD 5.6 there are improvements to its original HAMMER2 file-system to the extent that it's now selected by its installer as the default file-system choice for new installations. Curious how the performance now compares between HAMMER and HAMMER2, here are some initial benchmarks on an NVMe solid-state drive using DragonFlyBSD 5.6.0. With a 120GB Toshiba NVMe SSD on an Intel Core i7 8700K system, I ran some benchmarks of DragonFlyBSD 5.6.0 freshly installed with HAMMER2 and then again when returning to the original HAMMER file-system that remains available via its installer. No other changes were made to the setup during testing. And then for the more synthetic workloads it was just a mix. But overall HAMMER2 was performing well during the initial testing and great to see it continuing to offer noticeable leads in real-world workloads compared to the aging HAMMER file-system. HAMMER2 also offers better clustering, online deduplication, snapshots, compression, encryption, and many other modern file-system features. Beastie Bits Unix CLI relational database (https://spin.atomicobject.com/2019/06/16/unix-cli-relational-database/) The TTY demystified (https://www.linusakesson.net/programming/tty/index.php) Ranger, a console file manager with VI keybindings (https://ranger.github.io/) Some Unix Humor (https://www.reddit.com/r/unix/comments/c6o5ze/some_unix_humor/) OpenBSD -import vulkan-loader for Vulkan API support (https://marc.info/?l=openbsd-ports-cvs&m=156121732625604&w=2) FreeBSD ZFS without drives (https://savagedlight.me/2019/06/09/freebsd-zfs-without-drives/) Feedback/Questions Moritz - ARM Builds (http://dpaste.com/175RRAZ) Dave - Videos (http://dpaste.com/2DYK85B) Chris - Raspberry Pi4 (http://dpaste.com/1B16QVN) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
305: Changing face of Unix
Website protection with OPNsense, FreeBSD Support Pull Request for ZFS-on-Linux, How much has Unix changed, Porting Wine to amd64 on NetBSD, FreeBSD Enterprise 1 PB Storage, the death watch for X11 has started, and more. Headlines Website protection with OPNsense (https://medium.com/@jccwbb/website-protection-with-opnsense-3586a529d487) with nginx plugin OPNsense become a strong full featured Web Application Firewall (WAF) The OPNsense security platform can help you to protect your network and your webservers with the nginx plugin addition. In old days, install an open source firewall was a very trick task, but today it can be done with few clicks (or key strokes). In this article I'll not describe the detailed OPNsense installation process, but you can watch this video that was extracted from my OPNsense course available in Udemy. The video is in portuguese language, but with the translation CC Youtube feature you may be able to follow it without problems (if you don't are a portuguese speaker ofcourse) :-) + See the article for the rest of the writeup FreeBSD Support Pull Request against the ZFS-on-Linux repo (https://github.com/zfsonlinux/zfs/pull/8987) This pull request integrates the sysutils/openzfs port’s sources into the upstream ZoL repo > Adding FreeBSD support to ZoL will make it easier to move changes back and forth between FreeBSD and Linux > Refactor tree to separate out Linux and FreeBSD specific code > import FreeBSD's SPL > add ifdefs in common code where it made more sense to do so than duplicate the code in separate files > Adapted ZFS Test Suite to run on FreeBSD and all tests that pass on ZoL passing on ZoF The plan to officially rename the common repo from ZFSonLinux to OpenZFS was announced at the ZFS Leadership Meeting on June 25th Video of Leadership Meeting (https://www.youtube.com/watch?v=TJwykiJmH0M) Meeting Agenda and Notes (https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit) This will allow improvements made on one OS to be made available more easily (and more quickly) to the other platforms For example, mav@’s recent work: Add wakeupany(), cheaper version of wakeupone() for taskqueue(9) (https://svnweb.freebsd.org/base?view=revision&revision=349220) > As result, on 72-core Xeon v4 machine sequential ZFS write to 12 ZVOLs with 16KB block size spend 34% less time in wakeupany() and descendants then it was spending in wakeupone(), and total write throughput increased by ~10% with the same as before CPU usage. News Roundup Episode 5 Notes - How much has UNIX changed? (http://adventofcomputing.libsyn.com/episode-5-notes-how-much-has-unix-changed) UNIX-like systems have dominated computing for decades, and with the rise of the internet and mobile devices their reach has become even larger. True, most systems now use more modern OSs like Linux, but how much has the UNIX-like landscape changed since the early days? So, my question was this: how close is a modern *NIX userland to some of the earliest UNIX releases? To do this I'm going to compare a few key points of a modern Linux system with the earliest UNIX documentation I can get my hands on. The doc I am going to be covering(https://www.tuhs.org/Archive/Distributions/Research/Dennisv1/UNIXProgrammersManual_Nov71.pdf) is from November 1971, predating v1 of the system. I think the best place to start this comparison is to look at one of the highest-profile parts of the OS, that being the file system. Under the hood modern EXT file systems are completely different from the early UNIX file systems. However, they are still presented in basically the same way, as a heirerarchicat structure of directories with device files. So paths still look identical, and navigating the file system still functions the same. Often used commands like ls, cp, mv, du, and df function the same. So are mount and umount. But, there are some key differences. For instance, cd didn't exist, yet instead chdir filled its place. Also, chmod is somewhat different. Instead of the usual 3-digit octal codes for permissions, this older version only uses 2 digits. Really, that difference is due to the underlying file system using a different permission set than modern systems. For the most part, all the file handling is actually pretty close to a Linux system from 2019. See the article for the rest of the writeup Porting Wine to amd64 on NetBSD (https://blog.netbsd.org/tnf/entry/porting_wine_to_amd64_on) I have been working on porting Wine to amd64 on NetBSD as a GSoC 2019 project. Wine is a compatibility layer which allows running Microsoft Windows applications on POSIX-complaint operating systems. This report provides an overview of the progress of the project during the first coding period. Initially, when I started working on getting Wine-4.4 to build and run on NetBSD i386 the primary issue that I faced was Wine displaying black windows instead of UI, and this applied to any graphical program I tried running with Wine. I suspected it , as it is related to graphics, to be an issue with the graphics driver or Xorg. Subsequently, I tried building modular Xorg, and I tried running Wine on it only to realize that Xorg being modular didn't affect it in the least. After having tried a couple of configurations, I realized that trying to hazard out every other probability is going to take an awful lot of time that I didn't have. This motivated me to bisect the repo using git, and find the first version of Wine which failed on NetBSD. + See the article for the rest of the writeup FreeBSD Enterprise 1 PB Storage (https://vermaden.wordpress.com/2019/06/19/freebsd-enterprise-1-pb-storage/?utm_source=discoverbsd) Today FreeBSD operating system turns 26 years old. 19 June is an International FreeBSD Day. This is why I got something special today :). How about using FreeBSD as an Enterprise Storage solution on real hardware? This where FreeBSD shines with all its storage features ZFS included. Today I will show you how I have built so called Enterprise Storage based on FreeBSD system along with more then 1 PB (Petabyte) of raw capacity. This project is different. How much storage space can you squeeze from a single 4U system? It turns out a lot! Definitely more then 1 PB (1024 TB) of raw storage space. See the article for the rest of the writeup The death watch for the X Window System (aka X11) has probably started (https://utcc.utoronto.ca/~cks/space/blog/unix/XDeathwatchStarts) Once we are done with this we expect X.org to go into hard maintenance mode fairly quickly. The reality is that X.org is basically maintained by us and thus once we stop paying attention to it there is unlikely to be any major new releases coming out and there might even be some bitrot setting in over time. We will keep an eye on it as we will want to ensure X.org stays supportable until the end of the RHEL8 lifecycle at a minimum, but let this be a friendly notice for everyone who rely the work we do maintaining the Linux graphics stack, get onto Wayland, that is where the future is. I have no idea how true this is about X.org X server maintenance, either now or in the future, but I definitely think it's a sign that developers have started saying this. If Gnome developers feel that X.org is going to be in hard maintenance mode almost immediately, they're probably pretty likely to also put the Gnome code that deals with X into hard maintenance mode. And public Gnome statements about this (and public action or lack of it) provide implicit support for KDE and any other desktop to move in this direction if they want to (and probably create some pressure to do so). I've known that Wayland was the future for some time, but I would still like it to not arrive any time soon. Beastie Bits Porting NetBSD to Risc-V -- Video (https://www.youtube.com/watch?v=2vQXGomKoxA) FreeBSD 11.3RC3 Available (https://www.freebsd.org/news/newsflash.html#event20190628:01) Open Source Could Be a Casualty of the Trade War (https://www.bunniestudios.com/blog/?p=5590) Celebrate UNIX50 and SDF32 (https://sdf.org/sdf32/) doas environmental security (https://undeadly.org/cgi?action=article;sid=20190621104048) Feedback/Questions Matt - BSD or Older Hardware (http://dpaste.com/1RP09F0#wrap) MJRodriguez - Some Playstation news (http://dpaste.com/046SPPB#wrap) Moritz - bhyve VT-x passthrough (http://dpaste.com/1H4PJXW) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
304: Prospering with Vulkan
DragonflyBSD 5.6 is out, OpenBSD Vulkan Support, bad utmp implementations in glibc and FreeBSD, OpenSSH protects itself against Side Channel attacks, ZFS vs OpenZFS, and more. Headlines DragonflyBSD 5.6 is out (https://www.dragonflybsd.org/release56) Version 5.6.0 released 17 June 2019 Version 5.6.1 released 19 June 2019 (https://www.dragonflydigest.com/2019/06/19/23091.html) Big-ticket items Improved VM Informal test results showing the changes from 5.4 to 5.6 are available. Reduce stalls in the kernel vmpagealloc() code (vmpagelist_find()). Improve page allocation algorithm to avoid re-iterating the same queues as the search is widened. Add a vmpagehash*() API that allows the kernel to do heuristical lockless lookups of VM pages. Change vmhold() and vmunhold() semantics to not require any spin-locks. Change vmpagewakeup() to not require any spin-locks. Change wiring vm_page's no longer manipulates the queue the page is on, saving a lot of overhead. Instead, the page will be removed from its queue only if the pageout demon encounters it. This allows pages to enter and leave the buffer cache quickly. Refactor the handling of fictitious pages. Remove m->md.pvlist entirely. VM pages in mappings no longer allocate pventry's, saving an enormous amount of memory when multiple processes utilize large shared memory maps (e.g. postgres database cache). Refactor vmobject shadowing, disconnecting the backing linkages from the vmobject itself and instead organizing the linkages in a new structure called vmmapbacking which hangs off the vmmapentry. pmap operations now iterate vmmapbacking structures (rather than spin-locked page lists based on the vmpage and pventry's), and will test/match operations against the PTE found in the pmap at the requisite location. This doubles VM fault performance on shared pages and reduces the locking overhead for fault and pmap operations. Simplify the collapse code, removing most of the original code and replacing it with simpler per-vmmapentry optimizations to limit the shadow depth. DRM Major updates to the radeon and ttm (amd support code) drivers. We have not quite gotten the AMD support up to the more modern cards or Ryzen APUs yet, however. Improve UEFI framebuffer support. A major deadlock has been fixed in the radeon/ttm code. Refactor the startup delay designed to avoid conflicts between the i915 driver initialization and X startup. Add DRMIOCTLGET_PCIINFO to improve mesa/libdrm support. Fix excessive wired memory build-ups. Fix Linux/DragonFly PAGE_MASK confusion in the DRM code. Fix idr_*() API bugs. HAMMER2 The filesystem sync code has been rewritten to significantly improve performance. Sequential write performance also improved. Add simple dependency tracking to prevent directory/file splits during create/rename/remove operations, for better consistency after a crash. Refactor the snapshot code to reduce flush latency and to ensure a consistent snapshot. Attempt to pipeline the flush code against the frontend, improving flush vs frontend write concurrency. Improve umount operation. Fix an allocator race that could lead to corruption. Numerous other bugs fixed. Improve verbosity of CHECK (CRC error) console messages. OpenBSD Vulkan Support (https://www.phoronix.com/scan.php?page=news_item&px=OpenBSD-Vulkan-Support) Somewhat surprisingly, OpenBSD has added the Vulkan library and ICD loader support as their newest port. This new graphics/vulkan-loader port provides the generic Vulkan library and ICD support that is the common code for Vulkan implementations on the system. This doesn't enable any Vulkan hardware drivers or provide something new not available elsewhere, but is rare seeing Vulkan work among the BSDs. There is also in ports the related components like the SPIR-V headers and tools, glsllang, and the Vulkan tools and validation layers. This is of limited usefulness, at least for the time being considering OpenBSD like the other BSDs lag behind in their DRM kernel driver support that is ported over from the mainline Linux kernel tree but generally years behind the kernel upstream. Particularly with Vulkan, newer kernel releases are needed for some Vulkan features as well as achieving decent performance. The Vulkan drivers of relevance are the open-source Intel ANV Vulkan driver and Radeon RADV drivers, both of which are in Mesa though we haven't seen any testing results to know how well they would work if at all currently on OpenBSD, but they're at least in Mesa and obviously open-source. + A note: The BSDs are no longer that far behind. + FreeBSD 12.0 uses DRM from Linux 4.16 (April 2018), and the drm-devel port is based on Linux 5.0 (March 2019) + OpenBSD -current as of April 2019 uses DRM from Linux 4.19.34 News Roundup Bad utmp implementations in glibc and freebsd (https://davmac.wordpress.com/2019/05/04/bad-utmp-implementations-in-glibc-and-freebsd/) I recently released another version – 0.5.0 – of Dinit, the service manager / init system. There were a number of minor improvements, including to the build system (just running “make” or “gmake” should be enough on any of the systems which have a pre-defined configuration, no need to edit mconfig by hand), but the main features of the release were S6-compatible readiness notification, and support for updating the utmp database. In other words, utmp is a record of who is currently logged in to the system (another file, “wtmp”, records all logins and logouts, as well as, potentially, certain system events such as reboots and time updates). This is a hint at the main motivation for having utmp support in Dinit – I wanted the “who” command to correctly report current logins (and I wanted boot time to be correctly recorded in the wtmp file). I wondered: If the files consist of fixed-sized records, and are readable by regular users, how is consistency maintained? That is – how can a process ensure that, when it updates the database, it doesn’t conflict with another process also attempting to update the database at the same time? Similarly, how can a process reading an entry from the database be sure that it receives a consistent, full record and not a record which has been partially updated? (after all, POSIX allows that a write(2) call can return without having written all the requested bytes, and I’m not aware of Linux or any of the *BSDs documenting that this cannot happen for regular files). Clearly, some kind of locking is needed; a process that wants to write to or read from the database locks it first, performs its operation, and then unlocks the database. Once again, this happens under the hood, in the implementation of the getutent/pututline functions or their equivalents. Then I wondered: if a user process is able to lock the utmp file, and this prevents updates, what’s to stop a user process from manually acquiring and then holding such a lock for a long – even practically infinite – duration? This would prevent the database from being updated, and would perhaps even prevent logins/logouts from completing. Unfortunately, the answer is – nothing; and yes, it is possible on different systems to prevent the database from being correctly updated or even to prevent all other users – including root – from logging in to the system. + A good find + On FreeBSD, even though write(2) can be asynchronous, once the write syscall returns, the data is in the buffer cache (or ARC), and any future read(2) will see that new data even if it has not yet been written to disk. OpenSSH gets an update to protect against Side Channel attacks (https://securityboulevard.com/2019/06/openssh-code-gets-an-update-to-protect-against-side-channel-attacks/) Last week, Damien Miller, a Google security researcher, and one of the popular OpenSSH and OpenBSD developers announced an update to the existing OpenSSH code that can help protect against the side-channel attacks that leak sensitive data from computer’s memory. This protection, Miller says, will protect the private keys residing in the RAM against Spectre, Meltdown, Rowhammer, and the latest RAMBleed attack. SSH private keys can be used by malicious threat actors to connect to remote servers without the need of a password. According to CSO, “The approach used by OpenSSH could be copied by other software projects to protect their own keys and secrets in memory”. However, if the attacker is successful in extracting the data from a computer or server’s RAM, they will only obtain an encrypted version of an SSH private key, rather than the cleartext version. In an email to OpenBSD, Miller writes, “this change encrypts private keys when they are not in use with a symmetric key that is derived from a relatively large ‘prekey’ consisting of random data (currently 16KB).” ZFS vs OpenZFS (https://www.ixsystems.com/blog/zfs-vs-openzfs/) You’ve probably heard us say a mix of “ZFS” and “OpenZFS” and an explanation is long-overdue. From its inception, “ZFS” has referred to the “Zettabyte File System” developed at Sun Microsystems and published under the CDDL Open Source license in 2005 as part of the OpenSolaris operating system. ZFS was revolutionary for completely decoupling the file system from specialized storage hardware and even a specific computer platform. The portable nature and advanced features of ZFS led FreeBSD, Linux, and even Apple developers to start porting ZFS to their operating systems and by 2008, FreeBSD shipped with ZFS in the 7.0 release. For the first time, ZFS empowered users of any budget with enterprise-class scalability and data integrity and management features like checksumming, compression and snapshotting, and those features remain unrivaled at any price to this day. On any ZFS platform, administrators use the zpool and zfs utilities to configure and manage their storage devices and file systems respectively. Both commands employ a user-friendly syntax such as‘zfs create mypool/mydataset’ and I welcome you to watch the appropriately-titled webinar “Why we love ZFS & you should too” or try a completely-graphical ZFS experience with FreeNAS. Oracle has steadily continued to develop its own proprietary branch of ZFS and Matt Ahrens points out that over 50% of the original OpenSolaris ZFS code has been replaced in OpenZFS with community contributions. This means that there are, sadly, two politically and technologically-incompatible branches of “ZFS” but fortunately, OpenZFS is orders of magnitude more popular thanks to its open nature. The two projects should be referred to as “Oracle ZFS” and “OpenZFS” to distinguish them as development efforts, but the user still types the ‘zfs’ command, which on FreeBSD relies on the ‘zfs.ko’ kernel module. My impression is that the terms of the CDDL license under which the OpenZFS branch of ZFS is published protects its users from any patent and trademark risks. Hopefully, this all helps you distinguish the OpenZFS project from the ZFS technology. + There was further discussion of how the ZFSOnLinux repo will become the OpenZFS repo in the future once it also contains the bits to build on FreeBSD as well during the June 25th ZFS Leadership Meeting. The videos for all of the meetings are available here (https://www.youtube.com/channel/UC0IK6Y4Go2KtRueHDiQcxow) Beastie Bits How to safely and portably close a file descriptor in a multithreaded process without running into problems with EINTR (https://twitter.com/cperciva/status/1141852451756105729?s=03) KnoxBug Meetup June 27th at 6pm (http://knoxbug.org/2019-06-27) BSD Pizza Night, June 27th at 7pm, Flying Pie Pizzeria, 3 Monroe Pkwy, Ste S, Lake Oswego, OR (https://www.flying-pie.com/locations/lake-oswego/) Difference between $x and ${x} (https://moopost.blogspot.com/2019/06/difference-between-x-and-x.html) Beware of Software Engineering Media Sites (https://www.nemil.com/on-software-engineering/beware-engineering-media.html) How Verizon and a BGP optimizer knocked large parts of the internet offline today (https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/) DragonflyBSD - MDS mitigation added a while ago (http://lists.dragonflybsd.org/pipermail/commits/2019-May/718899.html) Reminder: Register for EuroBSDcon 2019 in Lillehammer, Norway (https://eurobsdcon.org) Feedback/Questions Dave - CheriBSD (http://dpaste.com/38233JC) Neb - Hello from Norway (http://dpaste.com/0B8XKXT#wrap) Lars - Ansible tutorial? (http://dpaste.com/3N85SHR) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) *** Your browser does not support the HTML5 video tag.
303: OpenZFS in Ports
OpenZFS-kmod port available, using blacklistd with NPF as fail2ban replacement, ZFS raidz expansion alpha preview 1, audio VU-meter increases CO2 footprint rant, XSAVE and compat32 kernel work for LLDB, where icons for modern X applications come from, and more. Headlines ZFSonFreeBSD ports renamed OpenZFS (https://www.freshports.org/sysutils/openzfs-kmod) The ZFS on FreeBSD project has renamed the userland and kernel ports from zol and zol-kmod to openzfs and openzfs-kmod The new versions from this week are IOCTL compatible with the command line tools in FreeBSD 12.0, so you can use the old userland with the new kernel module (although obviously not the new features) With the renaming it is easier to specify which kernel module you want to load in /boot/loader.conf: > zfs_load=”YES” or > openzfs_load=”YES” To load traditional or the newer version of ZFS The kmod still requires FreeBSD 12-stable or 13-current because it depends on the newer crypto support in the kernel for the ZFS native encryption feature. Allan is looking at ways to work around this, but it may not be practical. We would like to do an unofficial poll on how people would the userland to co-exist. Add a suffix to the new commands in /usr/local (zfs.new zpool.new or whatever). One idea i’ve had is to move the zfs and zpool commands to /libexec and make /sbin/zfs and /sbin/zpool a switcher script, that will call the base or ports version based on a config file (or just based on if the port is installed) For testing purposes, generally you should be fine as long as you don’t run ‘zpool upgrade’, which will make your pool only importable using the newer ZFS. For extra safety, you can create a ‘zpool checkpoint’, which will allow you to undo any changes that are made to the pool during your testing with the new openzfs tools. Note: the checkpoint will undo EVERYTHING. So don’t save new data you want to keep. Note: Checkpoints disable all freeing operations, to prevent any data from being overwritten so that you can re-import at the checkpoint and undo any operation (including zfs destroy-ing a dataset), so also be careful you don’t run out of space during testing. Please test and provide feedback. How to use blacklistd(8) with NPF as a fail2ban replacement (https://www.unitedbsd.com/d/63-how-to-use-blacklistd8-with-npf-as-a-fail2ban-replacement) About blacklistd(8) blacklistd(8) provides an API that can be used by network daemons to communicate with a packet filter via a daemon to enforce opening and closing ports dynamically based on policy. The interface to the packet filter is in /libexec/blacklistd-helper (this is currently designed for npf) and the configuration file (inspired from inetd.conf) is in etc/blacklistd.conf Now, blacklistd(8) will require bpfjit(4) (Just-In-Time compiler for Berkeley Packet Filter) in order to properly work, in addition to, naturally, npf(7) as frontend and syslogd(8), as a backend to print diagnostic messages. Also remember npf shall rely on the npflog* virtual network interface to provide logging for tcpdump() to use. Unfortunately (dont' ask me why :P) in 8.1 all the required kernel components are still not compiled by default in the GENERIC kernel (though they are in HEAD), and are rather provided as modules. Enabling NPF and blacklistd services would normally result in them being automatically loaded as root, but predictably on securelevel=1 this is not going to happen News Roundup [WIP] raidz expansion, alpha preview 1 (https://github.com/zfsonlinux/zfs/pull/8853) Motivation and Context > This is a alpha-quality preview of RAID-Z expansion. This feature allows disks to be added one at a time to a RAID-Z group, expanding its capacity incrementally. This feature is especially useful for small pools (typically with only one RAID-Z group), where there isn't sufficient hardware to add capacity by adding a whole new RAID-Z group (typically doubling the number of disks). > For additional context as well as a design overview, see my short talk from the 2017 OpenZFS Developer Summit: slides video Rant: running audio VU-meter increases my CO2 footprint (https://medium.com/@MartinCracauer/bug-rant-running-audio-vu-meter-increases-my-co2-footprint-871d5c1bee5a) A couple months ago I noticed that the monitor on my workstation never power off anymore. Screensaver would go on, but DPMs (to do the poweroff) never kicked in. I grovels the output of various tools that display DPMS settings, which as usual in Xorg were useless. Everybody said DPMS is on with a timeout. I even wrote my own C program to use every available Xlib API call and even the xscreensaver library calls. (should make it available) No go, everybody says that DPMs is on, enabled and set on a timeout. Didn’t matter whether I let xscreeensaver do the job or just the X11 server. After a while I noticed that DPMS actually worked between starting my X11 server and starting all my clients. I have a minimal .xinitrc and start the actual session from a script, that is how I could notice. If I used a regular desktop login I wouldn’t have noticed. A server state bug was much more likely than a client bug. + See the article for the rest... XSAVE and compat32 kernel work for LLDB (http://blog.netbsd.org/tnf/entry/xsave_and_compat32_kernel_work) Upstream describes LLDB as a next generation, high-performance debugger. It is built on top of LLVM/Clang toolchain, and features great integration with it. At the moment, it primarily supports debugging C, C++ and ObjC code, and there is interest in extending it to more languages. In February, I have started working on LLDB, as contracted by the NetBSD Foundation. So far I've been working on reenabling continuous integration, squashing bugs, improving NetBSD core file support and lately extending NetBSD's ptrace interface to cover more register types. You can read more about that in my Apr 2019 report. In May, I was primarily continuing the work on new ptrace interface. Besides that, I've found and fixed a bug in ptrace() compat32 code, pushed LLVM buildbot to ‘green’ status and found some upstream LLVM regressions. More below. Some things about where icons for modern X applications come from (https://utcc.utoronto.ca/~cks/space/blog/unix/ModernXAppIcons) If you have a traditional window manager like fvwm, one of the things it can do is iconify X windows so that they turn into icons on the root window (which would often be called the 'desktop'). Even modern desktop environments that don't iconify programs to the root window (or their desktop) may have per-program icons for running programs in their dock or taskbar. If your window manager or desktop environment can do this, you might reasonably wonder where those icons come from by default. Although I don't know how it was done in the early days of X, the modern standard for this is part of the Extended Window Manager Hints. In EWMH, applications give the window manager a number of possible icons, generally in different sizes, as ARGB bitmaps (instead of, say, SVG format). The window manager or desktop environment can then pick whichever icon size it likes best, taking into account things like the display resolution and so on, and display it however it wants to (in its original size or scaled up or down). How this is communicated in specific is through the only good interprocess communication method that X supplies, namely X properties. In the specific case of icons, the NETWMICON property is what is used, and xprop can display the size information and an ASCII art summary of what each icon looks like. It's also possible to use some additional magic to read out the raw data from _NETWM_ICON in a useful format; see, for example, this Stackoverflow question and its answers. Beastie Bits Recent Security Innovations (http://undeadly.org/cgi?action=article;sid=20190605110020) Old Unix books + Solaris (https://imgur.com/a/HbSYtQI) Pro-Desktop - A Tiling Desktop Environment (https://bitcannon.net/post/pro-desktop/) The Tar Pipe (https://blog.extracheese.org/2010/05/the-tar-pipe.html) At least one vim trick you might not know (https://www.hillelwayne.com/post/intermediate-vim/) Feedback/Questions Johnny - listener feedback (http://dpaste.com/0ZQCQ8Y#wrap) Brian - Questions (http://dpaste.com/1843RNX#wrap) Mark - ZFS Question (http://dpaste.com/3M83X9G#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Your browser does not support the HTML5 video tag.
302: Contention Reduction
DragonFlyBSD's kernel optimizations pay off, differences between OpenBSD and Linux, NetBSD 2019 Google Summer of Code project list, Reducing that contention, fnaify 1.3 released, vmctl(8): CLI syntax changes, and things that Linux distributions should not do when packaging. Headlines DragonFlyBSD's Kernel Optimizations Are Paying Off (https://www.phoronix.com/scan.php?page=article&item=dragonfly-55-threadripper&num=1) DragonFlyBSD lead developer Matthew Dillon has been working on a big VM rework in the name of performance and other kernel improvements recently. Here is a look at how those DragonFlyBSD 5.5-DEVELOPMENT improvements are paying off compared to DragonFlyBSD 5.4 as well as FreeBSD 12 and five Linux distribution releases. With Dillon using an AMD Ryzen Threadripper system, we used that too for this round of BSD vs. Linux performance benchmarks. The work by Dillon on the VM overhaul and other changes (including more HAMMER2 file-system work) will ultimately culminate with the DragonFlyBSD 5.6 release (well, unless he opts for DragonFlyBSD 6.0 or so). These are benchmarks of the latest DragonFlyBSD 5.5-DEVELOPMENT daily ISO as of this week benchmarked across DragonFlyBSD 5.4.3 stable, FreeBSD 12.0, Ubuntu 19.04, Red Hat Enterprise Linux 8.0, Debian 9.9, Debian Buster, and CentOS 7 1810 as a wide variety of reference points both from newer and older Linux distributions. (As for no Clear Linux reference point for a speedy reference point, it currently has a regression with AMD + Samsung NVMe SSD support on some hardware, including this box, prohibiting the drive from coming up due to a presumed power management issue that is still being resolved.) With Matthew Dillon doing much of his development on an AMD Ryzen Threadripper system after he last year proclaimed the greatness of these AMD HEDT CPUs, for this round of testing I also used a Ryzen Threadripper 2990WX with 32 cores / 64 threads. Tests of other AMD/Intel hardware with DragonFlyBSD will come as the next stable release is near and all of the kernel work has settled down. For now it's mostly entertaining our own curiosity how well these DragonFlyBSD optimizations are paying off and how it's increasing the competition against FreeBSD 12 and Linux distributions. What are the differences between OpenBSD and Linux? (https://cfenollosa.com/blog/what-are-the-differences-between-openbsd-and-linux.html) Maybe you have been reading recently about the release of OpenBSD 6.5 and wonder, "What are the differences between Linux and OpenBSD?" I've also been there at some point in the past and these are my conclusions. They also apply, to some extent, to other BSDs. However, an important disclaimer applies to this article. This list is aimed at people who are used to Linux and are curious about OpenBSD. It is written to highlight the most important changes from their perspective, not the absolute most important changes from a technical standpoint. Please bear with me. A terminal is a terminal is a terminal Practical differences Security and system administration Why philosophical differences matter So what do I choose? How to try OpenBSD *** News Roundup NetBSD 2019 Google Summer of Code (http://blog.netbsd.org/tnf/entry/announcing_google_summer_of_code1) We are very happy to announce The NetBSD Foundation Google Summer of Code 2019 projects: Akul Abhilash Pillai - Adapting TriforceAFL for NetBSD kernel fuzzing Manikishan Ghantasala - Add KNF (NetBSD style) clang-format configuration Siddharth Muralee - Enhancing Syzkaller support for NetBSD Surya P - Implementation of COMPATLINUX and COMPATNETBSD32 DRM ioctls support for NetBSD kernel Jason High - Incorporation of Argon2 Password Hashing Algorithm into NetBSD Saurav Prakash - Porting NetBSD to HummingBoard Pulse Naveen Narayanan - Porting WINE to amd64 architecture on NetBSD The communiting bonding period - where students get in touch with mentors and community - started yesterday. The coding period will start from May 27 until August 19. Please welcome all our students and a big good luck to students and mentors! A big thank to Google and The NetBSD Foundation organization mentors and administrators! Looking forward to a great Google Summer of Code! Reducing that contention (http://www.grenadille.net/post/2019/05/09/Reducing-that-contention) The opening keynote at EuroBSDCon 2016 predicted the future 10 years of BSDs. Amongst all the funny previsions, gnn@FreeBSD said that by 2026 OpenBSD will have its first implementation of SMP. Almost 3 years after this talk, that sounds like a plausible forecast... Why? Where are we? What can we do? Let's dive into the issue! State of affairs Most of OpenBSD's kernel still runs under a single lock, ze KERNEL_LOCK(). That includes most of the syscalls, most of the interrupt handlers and most of the fault handlers. Most of them, not all of them. Meaning we have collected & fixed bugs while setting up infrastructures and examples. Now this lock remains the principal responsible for the spin % you can observe in top(1) and systat(1). I believe that we opted for a difficult hike when we decided to start removing this lock from the bottom. As a result many SCSI & Network interrupt handlers as well as all Audio & USB ones can be executed without big lock. On the other hand very few syscalls are already or almost ready to be unlocked, as we incorrectly say. This explains why basic primitives like tsleep(9), csignal() and selwakeup() are only receiving attention now that the top of the Network Stack is running (mostly) without big lock. Next steps In the past years, most of our efforts have been invested into the Network Stack. As I already mentioned it should be ready to be parallelized. However think we should now concentrate on removing the KERNEL_LOCK(), even if the code paths aren't performance critical. See the Article for the rest of the post fnaify 1.3 released - more games are "fnaify & run" now (https://www.reddit.com/r/openbsd_gaming/comments/btste9/fnaify_13_released_more_games_are_fnaify_run_now/) This release finally addresses some of the problems that prevent simple running of several games. This happens for example when an old FNA.dll library comes with the games that doesn't match the API of our native libraries like SDL2, OpenAL, or MojoShader anymore. Some of those cases can be fixed by simply dropping in a newer FNA.dll. fnaify now asks if FNA 17.12 should be automatically added if a known incompatible FNA version is found. You simply answer yes or no. Another blocker happens when the game expects to check the SteamAPI - either from a running Steam process, or a bundled steam_api library. OpenBSD 6.5-current now has steamworks-nosteam in ports, a stub library for Steamworks.NET that prevents games from crashing simply because an API function isn't found. The repo is here. fnaify now finds this library in /usr/local/share/steamstubs and uses it instead of the bundled (full) Steamworks.NET.dll. This may help with any games that use this layer to interact with the SteamAPI, mostly those that can only be obtained via Steam. vmctl(8): command line syntax changed (https://www.openbsd.org/faq/current.html#r20190529) The order of the arguments in the create, start, and stop commands of vmctl(8) has been changed to match a commonly expected style. Manual usage or scripting with vmctl must be adjusted to use the new syntax. For example, the old syntax looked like this: # vmctl create disk.qcow2 -s 50G The new syntax specifies the command options before the argument: # vmctl create -s 50G disk.qcow2 Something that Linux distributions should not do when packaging things (https://utcc.utoronto.ca/~cks/space/blog/linux/PackageNameClashProblem) Right now I am a bit unhappy at Fedora for a specific packaging situation, so let me tell you a little story of what I, as a system administrator, would really like distributions to not do. For reasons beyond the scope of this blog entry, I run a Prometheus and Grafana setup on both my home and office Fedora Linux machines (among other things, it gives me a place to test out various things involving them). When I set this up, I used the official upstream versions of both, because I needed to match what we are running (or would soon be). Recently, Fedora decided to package Grafana themselves (as a RPM), and they called this RPM package 'grafana'. Since the two different packages are different versions of the same thing as far as package management tools are concerned, Fedora basically took over the 'grafana' package name from Grafana. This caused my systems to offer to upgrade me from the Grafana.com 'grafana-6.1.5-1' package to the Fedora 'grafana-6.1.6-1.fc29' one, which I actually did after taking reasonable steps to make sure that the Fedora version of 6.1.6 was compatible with the file layouts and so on from the Grafana version of 6.1.5. Why is this a problem? It's simple. If you're going to take over a package name from the upstream, you should keep up with the upstream releases. If you take over a package name and don't keep up to date or keep up to date only sporadically, you cause all sorts of heartburn for system administrators who use the package. The least annoying future of this situation is that Fedora has abandoned Grafana at 6.1.6 and I am going to 'upgrade' it with the upstream 6.2.1, which will hopefully be a transparent replacement and not blow up in my face. The most annoying future is that Fedora and Grafana keep ping-ponging versions back and forth, which will make 'dnf upgrade' into a minefield (because it will frequently try to give me a 'grafana' upgrade that I don't want and that would be dangerous to accept). And of course this situation turns Fedora version upgrades into their own minefield, since now I risk an upgrade to Fedora 30 actually reverting the 'grafana' package version on me. Beastie Bits [talk] ZFS v UFS on APU2 msata SSD with FreeBSD (http://lists.nycbug.org:8080/pipermail/talk/2019-May/017885.html) NetBSD 8.1 is out (http://www.netbsd.org/releases/formal-8/NetBSD-8.1.html) lazyboi – the laziest possible way to send raw HTTP POST data (https://github.com/ctsrc/lazyboi) A Keyboard layout that changes by markov frequency (https://github.com/shapr/markovkeyboard) Open Source Game Clones (https://osgameclones.com/) EuroBSDcon program & registration open (https://eurobsdcon.org) *** Feedback/Questions John - A segment idea (http://dpaste.com/3YTBQTX#wrap) Johnny - Audio only format please don't (http://dpaste.com/3WD0A25#wrap) Alex - Thanks and some Linux Snaps vs PBI feedback (http://dpaste.com/1RQF4QM#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) *** Your browser does not support the HTML5 video tag.
301: GPU Passthrough
GPU passthrough on bhyve, confusion with used/free disk space on ZFS, OmniOS Community Edition, pfSense 2.4.4 Release p3, NetBSD 8.1 RC1, FreeNAS as your Server OS, and more. Headlines GPU Passthrough Reported Working on Bhyve Normally we cover news focused on KVM and sometimes Xen, but something very special has happened with their younger cousin in the BSD world, Bhyve. For those that don’t know, Bhyve (pronounced bee-hive) is the native hypervisor in FreeBSD. It has many powerful features, but one that’s been a pain point for some years now is VGA passthrough. Consumer GPUs have not been useable until very recently despite limited success with enterprise cards. However, Twitter user Michael Yuji found a workaround that enables passing through a consumer card to any *nix system configured to use X11: https://twitter.com/michael_yuji/status/1127136891365658625 All you have to do is add a line pointing the X server to the Bus ID of the passed card and the VM will boot, with acceleration and everything. He theorizes that this may not be possible on windows because of the way it looks for display devices, but it’s a solid start. As soon as development surrounding VGA passthrough matures on Bhyve, it will become a very attractive alternative to more common tools like Hyper-V and Qemu, because it makes many powerful features available in the host system like jails, boot environments, BSD networking, and tight ZFS integration. For example, you could potentially run your Router, NAS, preferred workstation OS and any number of other things in one box, and only have to spin up a single VM because of the flexibility afforded by jails over Linux-based containers. The user who found this workaround also announced they’d be writing it up at some point, so stay tuned for details on the process. It’s been slow going on Bhyve passthrough development for a while, but this new revelation is encouraging. We’ll be closely monitoring the situation and report on any other happenings. Confusion with used/free disk space in ZFS I use ZFS extensively. ZFS is my favorite file system. I write articles and give lectures about it. I work with it every day. In traditional file systems we use df(1) to determine free space on partitions. We can also use du(1) to count the size of the files in the directory. But it’s different on ZFS and this is the most confusing thing EVER. I always forget which tool reports what disk space usage! Every time somebody asks me, I need to google it. For this reason I decided to document it here - for myself - because if I can’t remember it at least I will not need to google it, as it will be on my blog, but maybe you will also benefit from this blog post if you have the same problem or you are starting your journey with ZFS. The understanding of how ZFS is uses space and how to determine which value means what is a crucial thing. I hope thanks to this article I will finally remember it! News Roundup OmniOS Community Edition The OmniOS Community Edition Association is proud to announce the general availability of OmniOS - r151030. OmniOS is published according to a 6-month release cycle, r151030 LTS takes over from r151028, published in November 2018; and since it is a LTS release it also takes over from r151022. The r151030 LTS release will be supported for 3 Years. It is the first LTS release published by the OmniOS CE Association since taking over the reins from OmniTI in 2017. The next LTS release is scheduled for May 2021. The old stable r151026 release is now end-of-life. See the release schedule for further details. This is only a small selection of the new features, and bug fixes in the new release; review the release notes for full details. If you upgrade from r22 and want to see all new features added since then, make sure to also read the release notes for r24, r26 and r28. The OmniOS team and the illumos community have been very active in creating new features and improving existing ones over the last 6 months. pfSense 2.4.4 Release p3 is available We are pleased to announce the release of pfSense® software version 2.4.4-p3, now available for new installations and upgrades! pfSense software version 2.4.4-p3 is a maintenance release, bringing a number of security enhancements as well as a handful of fixes for issues present in the 2.4.4-p2 release. pfSense 2.4.4-RELEASE-p3 updates and installation images are available now! To see a complete list of changes and find more detail, see the Release Notes. We had hoped to bring you this release a few days earlier, but given the announcement last Tuesday of the Intel Microarchitectural Data Sampling (MDS) issue, we did not have sufficient time to fully incorporate those corrections and properly test for release on Thursday. We felt that it was worth delaying for a few days, rather than making multiple releases within a week. Upgrade Notes Due to the significant nature of the changes in 2.4.4 and later, warnings and error messages, particularly from PHP and package updates, are likely to occur during the upgrade process. In nearly all cases these errors are a harmless side effect of the changes between FreeBSD 11.1 and 11.2 and between PHP 5.6 and PHP 7.2. Always take a backup of the firewall configuration prior to any major change to the firewall, such as an upgrade. Do not update packages before upgrading pfSense! Either remove all packages or do not update packages before running the upgrade. The upgrade will take several minutes to complete. The exact time varies based on download speed, hardware speed, and other factors such installed packages. Be patient during the upgrade and allow the firewall enough time to complete the entire process. After the update packages finish downloading it could take 10-20 minutes or more until the upgrade process ends. The firewall may reboot several times during the upgrade process. Monitor the upgrade from the firewall console for the most accurate view. NetBSD 8.1 RC1 is out The NetBSD Project is pleased to announce NetBSD 8.1, the first update of the NetBSD 8 release branch. It represents a selected subset of fixes deemed important for security or stability reasons, as well as new features and enhancements. Some highlights of the 8.1 release are: x86: Mitigation for INTEL-SA-00233 (MDS) Various local user kernel data leaks fixed. x86: new rc.conf(5) setting smtoff to disable Simultaneous Multi-Threading Various network driver fixes and improvements. Fixes for thread local storage (TLS) in position independent executables (PIE). Fixes to reproducible builds. Fixed a performance regression in tmpfs. DRM/KMS improvements. bwfm(4) wireless driver for Broadcom FullMAC PCI and USB devices added. Various sh(1) fixes. mfii(4) SAS driver added. hcpcd(8) updated to 7.2.2 httpd(8) updated. FreeNAS as your Server OS What if you could have a server OS that had built in RAID, NAS and SAN functionality, and could manage packages, containers and VMs in a GUI? What if that server OS was also free to download and install? Wouldn’t that be kind of awesome? Wouldn’t that be FreeNAS? FreeNAS is the world’s number one, open source storage OS, but it also comes equipped with all the jails, plugins, and VMs you need to run additional server-level services for things like email and web site hosting. File, Block, and even Object storage is all built-in and can be enabled with a few clicks. The ZFS file system scales to more drives than you could ever buy, with no limits for dataset sizes, snapshots, and restores. FreeNAS is also 100% FreeBSD. This is the OS used in the Netflix CDN, your PS4, and the basis for iOS. Set up a jail and get started downloading packages like Apache or NGINX for web hosting or Postfix for email service. Just released, our new TrueCommand management platform also streamlines alerts and enables multi-system monitoring. Beastie Bits Keep Crashing Daemons Running on FreeBSD Look what I found today... my first set of BSD CDs... NetBSD - Intel MDS FreeBSD 11.3-BETA2 -- Please test! Feedback/Questions Anthony - Question Guntbert - Podcast Guillaume - Another suggestion for Ales from Serbia Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv Your browser does not support the HTML5 video tag.
300: The Big Three
FreeBSD 11.3-beta 1 is out, BSDCan 2019 recap, OpenIndiana 2019.04 is out, Overview of ZFS Pools in FreeNAS, why open source firmware is important for security, a new Opnsense release, wireguard on OpenBSD, and more. Headlines FreeBSD 11.3-b1 is out BSDCan 2019 Recap We’re back from BSDCan and it was a packed week as always. It started with bhyvecon on Tuesday. Meanwhile, Benedict spent the whole day in productive meetings: annual FreeBSD Foundation board meeting and FreeBSD Journal editorial board meeting. On Wednesday, tutorials for BSDCan started as well as the FreeBSD Developer Summit. In the mornings, there were presentations in the big auditorium, while working groups about networking, failsafe bootcode, development web services, swap space management, and testing/CI were held. Friday had a similar format with an update from the FreeBSD core team and the “have, need, want” session for FreeBSD 13. In the afternoon, there were working groups about translation tools, package base, GSoC/Outreachy, or general hacking. Benedict held his Icinga tutorial in the afternoon with about 15 people attending. Devsummit presentation slides can be found on the wiki page and video recordings done by ScaleEngine are available on FreeBSD’s youtube channel. The conference program was a good mixture of sysadmin and tech talks across the major BSDs. Benedict saw the following talks: How ZFS snapshots really work by Matt Ahrens, 20 years in Jail by Michael W. Lucas, OpenZFS BOF session, the future of OpenZFS and FreeBSD, MQTT for system administrators by Jan-Piet Mens, and spent the rest of the time in between in the hallway track. Photos from the event are available on Ollivier Robert’s talegraph and Diane Bruce’s website for day 1, day 2, conference day 1, and conference day 2. Thanks to all the sponsors, supporters, organizers, speakers, and attendees for making this yet another great BSDCan. Next year’s BSDCan will be from June 2 - 6, 2020. OpenIndiana 2019.04 is out We have released a new OpenIndiana Hipster snapshot 2019.04. The noticeable changes: Firefox was updated to 60.6.3 ESR Virtualbox packages were added (including guest additions) Mate was updated to 1.22 IPS has received updates from OmniOS CE and Oracle IPS repos, including automatic boot environment naming Some OI-specific applications have been ported from Python 2.7/GTK 2 to Python 3.5/GTK 3 Quick Demo Video: https://www.youtube.com/watch?v=tQ0-fo3XNrg News Roundup Overview of ZFS Pools in FreeNAS FreeNAS uses the OpenZFS (ZFS) file system, which handles both disk and volume management. ZFS offers RAID options mirror, stripe, and its own parity distribution called RAIDZ that functions like RAID5 on hardware RAID. The file system is extremely flexible and secure, with various drive combinations, checksums, snapshots, and replication all possible. For a deeper dive on ZFS technology, read the ZFS Primer section of the FreeNAS documentation. SUGGEST LAYOUT attempts to balance usable capacity and redundancy by automatically choosing an ideal vdev layout for the number of available disks. The following vdev layout options are available when creating a pool: Stripe data is shared on two drives, similar to RAID0) Mirror copies data on two drives, similar to RAID1 but not limited to 2 disks) RAIDZ1 single parity similar to RAID5 RAIDZ2 double parity similar to RAID6 RAIDZ3 which uses triple parity and has no RAID equivalent Why OpenSource Firmware is Important for Security Roots of Trust The goal of the root of trust should be to verify that the software installed in every component of the hardware is the software that was intended. This way you can know without a doubt and verify if hardware has been hacked. Since we have very little to no visibility into the code running in a lot of places in our hardware it is hard to do this. How do we really know that the firmware in a component is not vulnerable or that is doesn’t have any backdoors? Well we can’t. Not unless it was all open source. Every cloud and vendor seems to have their own way of doing a root of trust. Microsoft has Cerberus, Google has Titan, and Amazon has Nitro. These seem to assume an explicit amount of trust in the proprietary code (the code we cannot see). This leaves me with not a great feeling. Wouldn’t it be better to be able to use all open source code? Then we could verify without a doubt that the code you can read and build yourself is the same code running on hardware for all the various places we have firmware. We could then verify that a machine was in a correct state without a doubt of it being vulnerable or with a backdoor. It makes me wonder what the smaller cloud providers like DigitalOcean or Packet have for a root of trust. Often times we only hear of these projects from the big three or five. OPNsense This update addresses several privilege escalation issues in the access control implementation and new memory disclosure issues in Intel CPUs. We would like to thank Arnaud Cordier and Bill Marquette for the top-notch reports and coordination. Here are the full patch notes: system: address CVE-2019-11816 privilege escalation bugs[1] (reported by Arnaud Cordier) system: /etc/hosts generation without interfacehasgateway() system: show correct timestamp in config restore save message (contributed by nhirokinet) system: list the commands for the pluginctl utility when n+ argument is given system: introduce and use userIsAdmin() helper function instead of checking for 'page-all' privilege directly system: use absolute path in widget ACLs (reported by Netgate) system: RRD-related cleanups for less code exposure interfaces: add EN DUID Generation using OPNsense PEN (contributed by Team Rebellion) interfaces: replace legacygetallinterface_addresses() usage firewall: fix port validation in aliases with leading / trailing spaces firewall: fix outbound NAT translation display in overview page firewall: prevent CARP outgoing packets from using the configured gateway firewall: use CARP net.inet.carp.demotion to control current demotion in status page firewall: stop live log poller on error result dhcpd: change rule priority to 1 to avoid bogon clash dnsmasq: only admins may edit custom options field firmware: use insecure mode for base and kernel sets when package fingerprints are disabled firmware: add optional device support for base and kernel sets firmware: add Hostcentral mirror (HTTP, Melbourne, Australia) ipsec: always reset rightallowany to default when writing configuration lang: say "hola" to Spanish as the newest available GUI language lang: updates for Chinese, Czech, Japanese, German, French, Russian and Portuguese network time: only admins may edit custom options field openvpn: call openvpnrefreshcrls() indirectly via plugin_configure() for less code exposure openvpn: only admins may edit custom options field to prevent privilege escalation (reported by Bill Marquette) openvpn: remove custom options field from wizard unbound: only admins may edit custom options field wizard: translate typehint as well plugins: os-freeradius 1.9.3 fixes string interpolation in LDAP filters (contributed by theq86) plugins: os-nginx 1.12[2] plugins: os-theme-cicada 1.17 (contributed by Team Rebellion) plugins: os-theme-tukan 1.17 (contributed by Team Rebellion) src: timezone database information update[3] src: install(1) broken with partially matching relative paths[4] src: microarchitectural Data Sampling (MDS) mitigation[5] ports: carootnss 3.44 ports: php 7.2.18[6] ports: sqlite 3.28.0[7] ports: strongswan custom XAuth generic patch removed wiregaurd on OpenBSD Earlier this week I imported a port for WireGuard into the OpenBSD ports tree. At the moment we have the userland daemon and the tools available. The in-kernel implementation is only available for Linux. At the time of writing there are packages available for -current. Jason A. Donenfeld (WireGuard author) has worked to support OpenBSD in WireGuard and as such his post on ports@ last year got me interested in WireGuard, since then others have toyed with WireGuard on OpenBSD before and as such I've used Ted's article as a reference. Note however that some of the options mentioned there are no longer valid. Also, I'll be using two OpenBSD peers here. The setup will be as follows: two OpenBSD peers, of which we'll dub wg1 the server and wg2 the client. The WireGuard service on wg1 is listening on 100.64.4.3:51820. Conclusion WireGuard (cl)aims to be easier to setup and faster than OpenVPN and while I haven't been able to verify the latter, the first is certainly true...once you've figured it out. Most documentation out there is for Linux so I had to figure out the wireguardgo service and the tun parameters. But all in all, sure, it's easier. Especially the client configuration on iOS which I didn't cover here because it's essentially pkgadd libqrencode ; cat client.conf | qrencode -t ansiutf8, scan the code with the WireGuard app and you're good to go. What is particularly neat is that WireGuard on iOS supports Always-on. Beastie Bits Serenity OS vkernels vs pmap Brian Kernighan interviews Ken Thompson Improvements in forking, threading, and signal code DragonFly 5.4.3 NetBSD on the Odroid C2 Feedback/Questions Paulo - Laptops A Listener - Thanks Bostjan - Extend a pool and lower RAM footprint Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv Your browser does not support the HTML5 video tag.