Created by three guys who love BSD, we cover the latest news and have an extensive series of tutorials, as well as interviews with various people from all areas of the BSD community. It also serves as a platform for support and questions. We love and advocate FreeBSD, OpenBSD, NetBSD, DragonFlyBSD and TrueOS. Our show aims to be helpful and informative for new users that want to learn about them, but still be entertaining for the people who are already pros. The show airs on Wednesdays at 2:00PM (US Eastern time) and the edited version is usually up the following day.
Similar Podcasts
Elixir Outlaws
Elixir Outlaws is an informal discussion about interesting things happening in Elixir. Our goal is to capture the spirit of a conference hallway discussion in a podcast.
The Cynical Developer
A UK based Technology and Software Developer Podcast that helps you to improve your development knowledge and career,
through explaining the latest and greatest in development technology and providing you with what you need to succeed as a developer.
Programming Throwdown
Programming Throwdown educates Computer Scientists and Software Engineers on a cavalcade of programming and tech topics. Every show will cover a new programming language, so listeners will be able to speak intelligently about any programming language.
157: ZFS, The “Universal” File-system
This week on BSDNow, we have an interview with Richard Yao, who will be telling us about the experience and challenges of porting ZFS to Linux. That plus the latest news and feedback is coming your way, on your place This episode was brought to you by Headlines Registration for MeetBSD 2016 is now Open (https://www.meetbsd.com/) “Beastie’s coming home!” This year, MeetBSD will be held at UC Berkeley’s Clark Kerr Campus November 11th and 12th, preceded by a two day FreeBSD Vendor/Dev Summit (Nov 9th and 10th) MeetBSD can be traced back to its humble roots as a local workshop for BSD developers and users, hosted annually in Poland since 2004. Since then, MeetBSD’s popularity has spread, and it’s now widely recognized as its own conference with participants from all over the world. The US version runs every two years in California since 2008, and now trades off with the east coast vBSDCon which runs on the odd years. “MeetBSD 2016 uses a mixed unConference format featuring both scheduled talks and community-driven events such as birds-of-a-feather meetings, lightning talks, hackable presentations, stump the chumps, and speed geeking sessions. Speakers are to be determined – stay tuned for more information!” Register before September 30th, and get $30 off Kris and I will be there, along with lots of other FreeBSD Developers, Vendors, and Users. MeetBSD’s unconference style does a very good job of mingling users with developers and is one of my favourite conferences. *** Dual Booting FreeBSD and Windows UEFI (http://kev009.com/wp/2016/07/freebsd-uefi-root-on-zfs-and-windows-dual-boot/) Looking to install FreeBSD alongside Windows 10? What happens if that that system is pre-installed and UEFI? Well you could run TrueOS, but if that isn’t your bag and you want vanilla FreeBSD we have you covered this week! Over on Kevin Bowling’s blog, we have a detailed article showing exactly how to do that. First up, as prep you’ll need to go into the Windows disk manager and shrink your existing NTFS partition. You’ll need to next boot FreeBSD 11 or later. From there the walkthrough takes us through disk partitioning using gpart, and setup of ZFS into a boot-environment friendly layout. Once you get through the typical FreeBSD setup / extraction, the tutorial gives us a nice bonus, showing how to setup “rEFInd” for a graphical boot-menu. A great walkthrough, and hopefully it encourages others to try out dual-booting “EFI-style”. *** ZFS High-Availability NAS (https://github.com/ewwhite/zfs-ha/wiki) Interested in a DiY HA ZFS NAS? Edmund White (ewwhite on github) has posted a very detailed look at how he has custom-rolled his own Linux + ZFS + HA setup. Most of the concepts are already ones used in various other HA products, but it is interesting and informative to see a public detailed look at how ZFS and HA works. In particular this setup require some very specific hardware, such as dual-port SAS drives, so you will have to pre-plan according. The only bummer is this is a ZFS on Linux setup. Maybe this can serve as the guide / inspiration for somebody in our community to do their own FreeBSD + HA + ZFS setup and blog about it in similar detail. *** First public release of chyves - version 0.1.0 (http://chyves.org/) As bhyve continues to mature we are seeing tooling evolve around it. Enter ‘chyves’ which started life as a fork of iohyve. We are looking to do an interview with the author in the near future, but we still want to bring you some of the new features / changes in this evolution of bhyve management. First up, nearly every function from iohyve has either been re-written in part or full. Among the new features, a full logging system (master and per-vm logs), multiple pool configurations, properties stored outside of ZFS (for speed) and self-upgrading. (Will that work with pkg’d version?) In addition to the above features, the website has a large chart showing the original ‘iohyve’ commands, and how that usage has changed moving to chyves. Give it a spin, let the author know of issues! *** Interview - Richard Yao - ryao@gentoo.org (mailto:ryao@gentoo.org) Sr. Kernel Engineer at ClusterHQ - Major Contributor to ZFS on Linux News Roundup ZFS Deadlock: 'Directory of Death' (http://lists.freebsd.org/pipermail/freebsd-hackers/2016-July/049740.html) A user reports that when they try to install npm (the Node.js package manager), their system deadlocks It turns out, this was also hitting the FreeBSD package building machines PR 209158 (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209158) The problem was a race condition in the way renames are handled in the FreeBSD VFS vs how ZFS does them internally This bug has existed since the original import of ZFS, but some other change caused it to happen much more frequently “ZFS POSIX Layer is originally written for Solaris VFS which is very different from FreeBSD VFS. Most importantly many things that FreeBSD VFS manages on behalf of all filesystems are implemented in ZPL in a different Way. Thus, ZPL contains code that is redundant on FreeBSD or duplicates VFS functionality or, in the worst cases, badly interacts / interferes with VFS.” “The most prominent problem is a deadlock caused by the lock order reversal of vnode locks that may happen with concurrent zfsrename() and lookup(). The deadlock is a result of zfsrename() not observing the vnode locking contract expected by VFS.” The fixes have been merged to the 10.x and 11.x branches *** New BSD Magazine out (2016-07) (https://bsdmag.org/download/implementing-memory-cache-beast-architecture/) Articles include: Implementing in-memory cache in the BeaST architecture, Docker Cleanup, FreeNAS Getting Started Guide, and starting at the very beginning with open source The August issue is also out (https://bsdmag.org/download/minix-3-free-open-source-operating-system-highly-reliable-flexible-secure/) This issue features two articles about MINIX 3, continues the FreeNAS getting started guide, Optimizes the in-memory cache for the BeaST architecture, and talks about fixing failed ports for Hardened and LibreBSD We hope to have an interview with the creator of the BeaST architecture in the coming weeks *** DragonflyBSD and UEFI (http://lists.dragonflybsd.org/pipermail/users/2016-July/270796.html) We’ve featured a few stories and walkthroughs about using UEFI to dual-boot BSD, and now its Dragonfly BSD’s turn. Dave McFarlane writes into the DF mailing lists, telling us about the specific steps taken to get UEFI installed and boot-strapped on his system. If you’ve done a FreeBSD manual UEFI install, the process looks very similar, but you will end up manually running ‘gpt’ to create partitions, installing dist files, and eventually installing boot1.efi into the FAT EFI partition. Dave also ran into an issue with resulted in no /etc/fstab being present, and helpfully includes what his system needed to fully boot hammer properly. Somebody should document this fully for DFLY, since I would expect to become more commonplace as commodity hardware is shipped with UEFI on by default. *** Netflix and Fill (http://techblog.netflix.com/2016/08/netflix-and-fill.html) The Netflix team has produced a technical blog post describing how their OpenConnect appliances work First the content is received from the content provider, and the Netflix content team makes it ready for deployment, by transcoding the various bitrates, packaging the subtitles, etc. The finished files are then pushed to Amazon S3 storage “We deploy the majority of our updates proactively during configured fill windows. An important difference between our OpenConnect CDN and other commercial CDNs is the concept of proactive caching. Because we can predict with high accuracy what our members will watch and what time of day they will watch it, we can make use of non-peak bandwidth to download most of the content updates to the OCAs in our network during these configurable time windows. By reducing disk reads (content serving) while we are performing disk writes (adding new content to the OCAs), we are able to optimize our disk efficiency by avoiding read/write contention. The predictability of off-peak traffic patterns helps with this optimization, but we still only have a finite amount of time every day to get our content pre-positioned to where it needs to be before our traffic starts to ramp up and we want to make all of the OCA capacity available for content serving.” The OCA may actually contain more than one copy of the same video, because each disk in the OCA is independent, storing the same video on two different disks will provide twice the available read bandwidth Normally the filesystem cache would obviate the need for this, but the Netflix OCA has so much storage, and not a lot of memory, and the requests from users are offset enough that the cache is useless “OCAs communicate at regular intervals with the control plane services, requesting (among other things) a manifest file that contains the list of titles they should be storing and serving to members. If there is a delta between the list of titles in the manifest and what they are currently storing, each OCA will send a request, during its configured fill window, that includes a list of the new or updated titles that it needs. The response from the control plane in AWS is a ranked list of potential download locations, aka fill sources, for each title.” “It would be inefficient, in terms of both time and cost, to distribute a title directly from S3 to all of our OCAs, so we use a tiered approach. The goal is to ensure that the title is passed from one part of our network to another using the most efficient route possible.” The article then goes on to explain how they calculate the least cost filling source “Now that Netflix operates in 190 countries and we have thousands of appliances embedded within many ISP networks around the world, we are even more obsessed with making sure that our OCAs get the latest content as quickly as possible while continuing to minimize bandwidth cost to our ISP partners.” *** Beastie Bits: Cover reveal for “PAM Mastery” (http://blather.michaelwlucas.com/archives/2734) LibertyBSD 5.9 is out - looking for mirrors (http://libertybsd.net/download.html) Unix for Poets (https://web.stanford.edu/class/cs124/lec/124-UnixForPoets.pdf) Feedback/Questions Chuck / Ingo - Get Involved (http://pastebin.com/ksq0rfph) Oskar - Thanks (http://pastebin.com/YqzcHEMg) Alex - SMF (http://pastebin.com/WvdVZbYc) Raymond - RPI3 (http://pastebin.com/JPWgzSGv) ***
156: The Fresh BSD experience
This week on BSDNow, Allan is back from his UK trip and we’ll get to hear his thoughts on the developer summit. That plus all the This episode was brought to you by Headlines FreeBSD 11.0-RC1 Available (https://lists.freebsd.org/pipermail/freebsd-stable/2016-August/085277.html) FreeBSD is marching onwards to 11.0, and with it the first RC1 was released. In addition to the usual amd64 architectures, you may want to give it a whirl on your various ARM boards as well, as it includes images for the following systems: 11.0-RC1 amd64 GENERIC 11.0-RC1 i386 GENERIC 11.0-RC1 powerpc GENERIC 11.0-RC1 powerpc64 GENERIC64 11.0-RC1 sparc64 GENERIC 11.0-RC1 armv6 BANANAPI 11.0-RC1 armv6 BEAGLEBONE 11.0-RC1 armv6 CUBIEBOARD 11.0-RC1 armv6 CUBIEBOARD2 11.0-RC1 armv6 CUBOX-HUMMINGBOARD 11.0-RC1 armv6 GUMSTIX 11.0-RC1 armv6 RPI-B 11.0-RC1 armv6 RPI2 11.0-RC1 armv6 PANDABOARD 11.0-RC1 armv6 WANDBOARD 11.0-RC1 aarch64 GENERIC For those wondering the list of changes between this and BETA4, we have that as well: A NULL pointer dereference in IPSEC has been fixed. Support for SSH Protocol 1 has been removed. OpenSSH DSA keys have been disabled by default. Users upgrading from prior FreeBSD versions are urged to update their SSH keys to RSA or ECDSA keys before upgrading to 11.0-RC1. PCI-e hotplug on bridges with power controllers has been disabled. A loader tunable (hw.pci.enablepciehp) to disable PCI-e HotPlug has been added. A VESA panic on suspend has been fixed. Google Compute Engine image publication has been fixed. An AES-ICM heap corruption typo bug has been fixed. A regression in pf.conf while parsing the 'interval' keyword has been fixed. A ZFS/VFS deadlock has been fixed. RC2 is delayed while some issues are sorted out (https://lists.freebsd.org/pipermail/freebsd-stable/2016-August/085323.html) RC2 is looming large, but was pushed back a few days while the following bugs are sorted out: Issue with IPv6 UDP traffic being sent from wrong MAC address Layer2 violation with IPv6 *** OpenBSD just added initial support for the RaspberryPi 2 and 3 devices (https://marc.info/?l=openbsd-cvs&m=147059203101111&w=2) It’s a good time to be an ARM and BSD enthusiast. In addition to all the ARM images in FreeBSD 11.0, we also have word that initial support for RPi2 and RPi3 has started to land in OpenBSD. Mark Kettenis has posted the following with his Commit: Initial support for Raspberry Pi 2/3. All the hard work done by patrick@, I just cleaned things up a bit. Any bugs introduced in that process are entirely mine. This doesn't work yet. But when it does, you'll need recent firmware from the Raspberry Pi Foundation git repository at: https://github.com/raspberrypi/firmware The device tree for the Raspberry Pi is somewhat in flux as bits and pieces to support the Raspberry Pi 2 and 3 are committed to the mainline Linux kernel.“ + Exciting news! We will of course keep you informed as to when we have images to play with. Running OpenBSD / PF on a RPi does sound intriguing. drm-4.8-rc2 tagged in drm-next (https://lists.freebsd.org/pipermail/freebsd-x11/2016-August/017840.html) Remember when FreeBSD lagged so far behind in Graphics support? Well, those days are rapidly coming to an end. Matt Macy has posted an update to the FreeBSD X11 list with news of his DRM branch being caught up all the way to Linux 4.8-RC2 now. This is a huge accomplishment, with Matt commenting: As of this moment sys/dev/drm in the drm-next tree is sync with https://github.com/torvalds/linux drivers/gpu/drm (albeit only for the subset of drivers that FreeBSD supports - i915, radeon, and amdgpu). I feel this is a bit of a milestone as it means that it is possible that in the future graphics support on FreeBSD could proceed in lockstep with Linux. For those who want to try out the latest support, you can build from his branch at the following GitHub location: (https://github.com/FreeBSDDesktop/freebsd-base-graphics) Or, if compiling isn’t your thing, TrueOS (The re-branded PC-BSD) will be releasing the a new ISO based upon his update to Linux 4.7 in the coming days, with 4.8-RC2 to follow in the next week or two. *** Installing FreeBSD for Raspberry Pi (https://www.freebsdfoundation.org/freebsd/how-to-guides/installing-freebsd-for-raspberry-pi/) People have been running FreeBSD on various RPi devices for a while now, however there are still a lot of people who probably need a hand to get boot-strapped on their RPi system. The FreeBSD foundation has put together a nice tutorial which walks even the most novice user through getting FreeBSD up and running. In particular this could become a good way for students or other FreeBSD newcomers to try out the OS on a relatively low-cost platform outside of a VM. The tutorial starts of with a check-list of the specific items you’ll need to get started, for RPi 1 (a/b) or RPi 2 hardware. From there, instructions on how to get the downloaded images onto a sdcard are provided, including Mac and Windows image burning details. With this done, it’s really only a matter of plugging in your device to be presented with your new RPi + FreeBSD system. The most important details (the default username/password) at also provided, so don’t skim too quickly. *** Interview - Drew Gurkowski Foundation Intern: First time FreeBSD User and Writing Tutorials *** News Roundup FreeBSD’s ipfw gets a NAT64 implementation (https://svnweb.freebsd.org/base?view=revision&revision=304046) A new feature has been added to FreeBSD’s native firewall, ipfw2 The new loadable module implements stateless and stateful NAT64 “Stateless translation is appropriate when a NAT64 translator is used in front of IPv4-only servers to allow them to be reached by remote IPv6-only clients.” With this setup, you map specific IPv6 addresses to the corresponding IPv4 address, allowing IPv4 only servers to be reachable on the v6 network. “Stateful translation is suitable for deployment at the client side or at the service provider, allowing IPv6-only client hosts to reach remote IPv4-only nodes.” This configuration allows many IPv6 only clients to reach the “legacy” internet. The FreeBSD cluster has been waiting for this feature for a while, because they have limited IP addresses, but many service jails that require access to services like GitHub that are not IPv6 enabled. The work was sponsored by Yandex, the Russian search engine and long time FreeBSD user Example configurations for both types are included in the commit message If you would find this feature useful, please take the time to set it up and document the steps and contribute that to the FreeBSD Handbook. *** Update on using LLVM's lld linker in the FreeBSD base system (https://lists.freebsd.org/pipermail/freebsd-toolchain/2016-August/002240.html) Ed Maste has written a lengthy update on the progress being made towards using LLVM’s lld linker as a replacement for GNU’s ‘ld’. Ed starts off by giving us some of the potential benefits of using lld vs the 2.17.50 ‘ld’ version FreeBSD currently uses: AArch64 (arm64) support Link Time Optimization (LTO) New ABI support Other linker optimization Much faster link times Maintained code base Ed also gives us an update on several of the major blockers: Since the last update in March several lld developers have implemented much of the missing functionality. The main blockers were symbol version support and expression evaluation in the linker script expression parser. Both are now nearly complete“ A detailed plan was also articulated in respect to switching over: Update lld along with the Clang/LLVM 3.9 update that dim@ is working on. Add the bmake build infrastructure, installing as /usr/bin/ld.lld on the same architectures that use Clang (amd64, arm, arm64, i386). I don't think there's a need for a WITH_LLD src.conf knob, but will add one if desired. Update lld again (most likely to a snapshot from upstream SVN) once it is able to link an unmodified FreeBSD kernel. Modify the boot loader and kernel builds to avoid using features not implemented by lld. Introduce a WITHLLDAS_LD knob to have /usr/bin/ld be a ld.lld hardlink instead of /usr/bin/ld.bfd. Request ports exp-runs and issue a call for testing with 3rd party software. Fix issues found during this process. Switch /usr/bin/ld to ld.lld by default in head for the Clang-using architectures. Add a WITHOUTLLDAS_LD knob to switch back to GNU ld. *** How to install FreeBSD with ZFS filesystem on DigitalOcean (https://github.com/fxlv/docs/blob/master/freebsd/freebsd-with-zfs-digitalocean.md) I know we’ve mentioned using FreeBSD + ZFS on digital ocean in the past, but today we have a nice HowTo by Kaspars Mickevics (fxlv) on GitHub. Before getting started, kaspars mentions some pre-reqs. First up he recommends starting with a Minimum of 2GB of RAM. (The $20/mo droplet). This is to ensure you have plenty of cushion to avoid running out of memory during the process. It is possible to use ZFS with less, but depending on your desired workload this does make sense. From there, checking out “mfsBSD” is discussed, along with details on how to make it suitable for a DO installation. (Mostly just disabling DHCP for the network device) For good measure ‘pkg-static’ is also included. With that done, using mfsBSD you will create a tar file, which is then extracted on top of the running system. After rebooting, you will be able to run “bsdinstall” and proceed to installing / formatting your disk with ZFS as normal. A good tutorial, something I may need to do here in the near future. User manages to get OpenBSD and FreeBSD working with Libreboot (https://lists.nongnu.org/archive/html/libreboot/2016-08/msg00058.html) In a short drive-by post to the Libreboot mailing list Piotr Kubaj gives a quick notice that he managed to get OpenBSD and FreeBSD both booting. > I know GNU people don't like BSD, so let me make it quick :) > > > I've succeeded in booting FreeBSD 11.0-RC1 using txt mode on my X200 > with the newest Libreboot. > > To get installer to boot, I used: > kfreebsd (usb0,gpt3)/boot/kernel/kernel > set FreeBSD.vfs.mountfrom=ufs:/dev/da1p3 > boot > > I didn't try to install yet. > The trick looks relatively simple (looks like GRUB), manually loading the kernel with ‘kfreebsd’ and then setting the vfs.root.mountfrom variable to find the USB stick. In an update he also mentions booting OpenBSD with ‘kopenbsd’ instead of ‘kfreebsd’ (again GRUB syntax) Now somebody will need to test installation of the system (he didn’t) and see what other issues may crop up in running BSD on a free BIOS. *** Beastie Bits: The ACPICA (ACPI Component Architecture) coding language AML now in DragonFly BSD (http://lists.dragonflybsd.org/pipermail/commits/2016-July/624192.html) Release announcement for 4.3BSD Tahoe from 1988 (https://groups.google.com/forum/#!topic/comp.sys.tahoe/50ManvdM1-s) Feedback/Questions Mike - Jail Uptime (http://pastebin.com/FLpybL6D) Greg - Router Hardware (http://pastebin.com/RGuayhB3) Kristof writes in (http://pastebin.com/NT4zmHiG) Ty - Updates and Logs (http://pastebin.com/CtetZdFg) Benjamin - MTA Bug (http://pastebin.com/Qq3VbQG2) ***
155: Cabling up FreeBSD
This week on BSDNow, Allen is away in the UK (For BSDCam), but we still have a full episode for you! Don’t miss our interview with This episode was brought to you by Headlines My two year journey to becoming an OS Developer (http://zinascii.com/2016/going-to-joyent.html) A blog post by Ryan Zezeski about how he ended doing OS Development instead of working on application We have featured his posts before, including The illumos SYSCALL Handler (http://zinascii.com/2016/the-illumos-syscall-handler.html) It started in the summer of 2014: I had just left Basho after 3.5 years of working on Riak, when I decided I wanted to become an OS developer. I purchased Solaris Internals, cloned illumos-gate, fired up cscope, and got to work. I hardly knew any C, x86 might as well have been Brainfuck, and, frankly, I knew shit about operating systems. But I was determined. I’ve always learned best by beating my head against something until it makes sense. I’m not a fast learner; I’m persistent. What others have in ability I make up for in effort. And when it comes to OS internals it’s all about work ethic. The more you look, the more you realize it’s just another program. The main difference being: it’s the program all the other programs run on. My strategy: to pick something, anything, that looked interesting, and write a post describing how it works. I wrote several of these posts in 2014 and 2015. More important, it put me in touch with Roger Faulkner: the creator of truss(1), the Solaris process model, and the real /proc filesystem. At the time I didn’t like my interaction with Roger. He explained, in what I would later find out to be his typical gruff manner, that I was wrong; so I concluded he is a prick. But over the years I realized that I was being a brat—he was trying to teach me something and I let my ego get in the way. I’ve come to view that interaction as a blessing. I interacted with one of the greats, a mentor of my mentor’s mentor (a Great Great Mentor). A couple of weeks later something even more surreal happened, at illumos Day 2014. Bryan Cantrill was the last speaker of the day. One of my mentors and someone I admire greatly. He was there to regale us with the story of Joyent’s resurrection of lx-branded zones: Linux system call emulation on top of the illumos kernel. But before he would do that he decided to speak about me! I couldn’t believe it. I was so overwhelmed that I don’t remember most of what he said. I was too busy flipping shit—Bryan Cantrill is on stage, in front of other kernel developers I look up to, saying my name. I was in a dream. It turns out, unknown to me at the time, that he wrote the POSIX queue code for both Solaris and QNX, which I wrote about. He compared me to the great expository technical writers Elliott Organick and Richard Stevens. And it was at this moment that I knew I could do this: I could become an OS developer. Never underestimate the effect kind words can have on someone that looks up to you. There is a lot more to the story, and it is definitely worth the read The story then goes on to talk about his recent run in with Bryan Cantrill > A week from now my two year journey to become an OS developer comes to an end; and a new chapter begins. I don’t know what specific things I’m going to work on, but I’m sure it will push me to the limit. I look forward to the challenge. *** Version 1.0 of the Lumina Desktop released (https://lumina-desktop.org/version-1-0-0-released/) After 4 years of development, Lumina Desktop has now hit version 1.0! This release brings with it a slew of new features and support: Completely customizable interface! Rather than having to learn how to use a new layout, change the desktop to suit you instead! Simple shortcuts for any application! The “favorites” system makes it easy to find and launch applications at any time. Extremely lightweight! Allows applications to utilize more of your system hardware and revitalizes older systems! Multiple-monitor support! Each monitor is treated as an independent entity – making it great for presentation systems which use a temporary monitor or for workstations which utilize an array of monitors for various tasks. While originally developed on PC-BSD, it already has been ported to a variety of different platforms, including OpenBSD, DragonFly, NetBSD, Debian and Gentoo Lumina has become the defacto desktop environment for TrueOS (Formerly PC-BSD), and looks like will provide a solid framework to continue growing desktop features. *** n2k16 hackathon report: Ken Westerback on dhclient, bridges, routing and more (http://undeadly.org/cgi?action=article&sid=20160804200232) Next up, we have a report from Ken Westerback talking about the recent OpenBSD hackathon in Prague He starts by telling us about the work in bpf: First order of business, stsp@'s weird setup involving bridges and multiple dhclient clients. A bit of bpf(4) programming to restrict dhclient to handling ethernet packets unicast to its interface worked. Cool. Unfortunately it turned out some lazy dhcp servers always use ethernet broadcasts just because some lesser, non-OpenBSD clients ignore unicast packets until they have configured IP. Classic chicken and egg. So this was backed out just before 6.0. Sigh. Next up, he talks about an idea he had on the flight over, specifically with regard to how DHCP leases are stored, and how keeping the SSID information with them could speed up re-connection times, by only trying leases for current SSID’s connected. After a day or so of hacking, it was working! However for $REASONS it was shelved for post 6.0, bummer! He then discusses an on-going project with Peter Hessler on passing along relevant PIDs in response to routing messages generated by kernel from ioctl events. This is something they’ve been hacking at, in order to allow dhclient to recognize its own routing messages. Sounds like they are both still works-in-progress. However, Ken did get something in for 6.0: Diving back into dhclient code I discovered that in situations where multiple offers were received the unused offers were not being declined and discarded. Despite a clear comment saying that's what was being done! Thus dhclient might gradually use up more and more memory. And possibly be retrying offers that should have been discarded. The fix for this did make 6.0! Yay! In Memoriam Roger Faulkner (https://www.usenix.org/memoriam-roger-faulkner) USENIX has re-released Roger Faulkner’s original paper on /proc as a free download The UNIX community recently lost one of its original pioneers, Roger Faulkner, whom one commenter described as “The godfather of post-AT&T UNIX” In his memory, the USENIX group as re-released his original paper on the /proc file-system from 1991. Roger worked in many area’s of UNIX, however the process file system /proc was his special baby. “/proc began as a debugger interface superseding ptrace(2) but has evolved into a general interface to the process model.” The original /proc only had a file for each process, not a directory. "Data may be transferred from or to any valid locations in the process's address space by applying lseek(2) to position the file at the virtual address of interest followed by read(2) or write(2)." Processes could be controlled using IOCTLs on the file As the USENIX article states: Roger believed that terrible things were sometimes required to create beautiful abstractions, and his trailblazing work on /proc embodies this burden: the innards may be delicate and nasty ("vile," as Roger might say in his distinguished Carolinian accent)—but the resulting abstractions are breathtaking in their power, scope and robustness. RIP Roger, and thanks for the wonderful UNIX legacy you’ve left us all. Interview - Myke Geiger - myke@servernorth.net (mailto:myke@servernorth.net) / @mWare (https://twitter.com/mWare) Using FreeBSD at a DSL/Cable ISP *** News Roundup New options in bsdinstall - some sysctls and date/time settings (https://www.reddit.com/r/freebsd/comments/4vxnw3/new_options_in_bsdinstall_some_sysctls_and/) bsdinstall in FreeBSD 11.0 will feature a number of new menus. The first, well allow you to set the date and time. Often on computers that have been in storage, or some embedded type devices that have no RTC, the date will be wildly wrong, and ntpd will refuse to run until the date is correctly set. This feature makes it easy to enter the date and time using dialog(1) The second menu, inspired by the existing ‘services’ menu, offers a number of ‘hardening’ options This menu allows users to easily enable a number of security features, including: Hide processes running as other users/groups Disable reading the kernel message buffer and debugging processes for unprivileged users Randomize the PID of newly created processes Enable the stack guard Erase /tmp at boot Disable remote syslog Disable sendmail All of these options are off by default, so that an install done with the installer will be the same as an install from source, or an upgrade. A number of these options are candidates to become on-by-default in the future, so the hope is that this menu will get more users to test these features and find any negative interactions with applications or general use, so they can be fixed. *** Rawrite32: the NetBSD image writing tool (https://www.netbsd.org/~martin/rawrite32/) Martin of the NetBSD project has released a new version of his USB imaging tool, rawrite32 For those who’ve not used this tool before, it is a Windows Application that allows writing NetBSD images directly to USB media (other other disk media) This update brings with it support for writing .xz file, and binary signing This may come in handy for writing other OS images to memory sticks as well, especially for those locked into a windows environment who need to switch. *** ZFS-Snap-Diff -- A pretty interface for viewing what changed after a ZFS snapshot (https://github.com/j-keck/zfs-snap-diff) There are lots of nice little utilities to help create and maintain your ZFS snapshots. However today we have something unique to look at, ‘zfs-snap-diff’. What makes it unique, is that it ships with a built-in golang / angularjs GUI for snapshot management It looks very powerful, including a built-in diff utility, so you can even see the changes in text-files, in addition to downloading files, restoring old versions and more. Its nice to see so many ZFS utilities starting to take off, and evolve file-management further. *** Dtrace Conf 2016 Event Videos (https://www.joyent.com/about/events/2016/dtrace-conf) The videos from Dtrace.conf 2016 have been posted Some highlights: Useful DTrace Intro CTF Everywhere Distributed DTrace DTrace for Apps DTrace json() subroutine Implementing (or not) fds[] in FreeBSD OpenDTrace DTrace performance improvements with always-on instrumentation D Syntactic Sugar DTrace and Go, DTrace and Postgres dtrace.conf(16) wrap-up by Bryan Cantrill (https://www.joyent.com/blog/dtrace-conf-16-wrap-up) Once again, it was an eclectic mix of technologists — and once again, the day got kicked off with me providing an introduction to dtrace.conf and its history. (Just to save you the time filling out your Cantrill Presentation Bingo Card: you can find me punching myself at 16:19, me offering unsolicited personal medical history at 20:11, and me getting trolled by unikernels at 38:25.) The next DTrace.conf isn’t until 2020 *** Beastie Bits The BSD Daemon features in Mexican candy packaging (https://www.reddit.com/r/BSD/comments/4vngmw/the_bsd_daemon_feature_in_mexican_candy_packaging/) Remove PG_ZERO and zeroidle (page-zeroing) entirely (http://lists.dragonflybsd.org/pipermail/commits/2016-August/624202.html) OpenBSD: Release Songs: 6.0: "Black Hat" (https://www.openbsd.org/lyrics.html#60b) OpenBSD Gaming Resource (http://satterly.neocities.org/openbsd_games.html) LibreSSL 2.4.2 and 2.3.7 Released (http://bsdsec.net/articles/libressl-2-4-2-and-2-3-7-released) Feedback/Questions Pedja - Bhyve GUI (http://pastebin.com/LJcJmNsR) Tim - Jail Management (http://pastebin.com/259x94Rh) Don - X260 (http://pastebin.com/A86yHnzz) David - Updates (http://pastebin.com/wjtcuVSA) Ghislain - Jail Management (http://pastebin.com/DgH9G7p5) ***
154: Myths, Pi’s & Features, oh my!
This week on BSDNow, we are taking a look at a few different tutorials, including running your very own RPi web-server. (Come-on, you This episode was brought to you by Headlines broken features aren't used (http://www.tedunangst.com/flak/post/broken-features-arent-used) This post from TedU talks about the difficulty of removing features from an operating system “One of the difficulties in removing a feature is identifying all the potential users. A feature here could be a program bundled with an operating system, or a command line option, or maybe just a function in a library. If we remove a feature, users that depend on it will be sad. Unfortunately, absence of evidence is not evidence of absence. I’ve never heard of anybody running ls -p but it’s not impossible that somebody does.” “The reasons why we want to remove an existing feature can vary. Sometimes it’s old code that interferes with maintenance. Sometimes a nearly complete rewrite can improve performance. In other cases, the feature in question is really more of a misfeature. It may have security implications, where the existence of the feature can be used to facilitate the exploitation of other vulnerabilities, and removing the feature will help mitigate the exploit.” “There’s no general test that can be used, but there is one test that works in many cases. Test that the feature works. If the feature doesn’t work, that’s compelling evidence that nobody is using it, because nobody can be using it. You don’t need to fix it. You can just remove it.” He makes some interesting comments about exhaustive unit tests and the push to keep everything working all the time. If you never break anything to see if someone complains, how do you know if it is still being used? *** A Raspberry Pi FreeBSD Web Server (http://devio.us/~spacemonkey/raspberry_pi_freebsd_web_server_howto.html) Looking at a super-low power solution to host some webpages? If so, we have the tutorial for you. Specifically a walkthrough of getting FreeBSD up on a Pi, and setting up nginx, OpenNTPD, LibreSSL and friends. The walkthrough starts with grabbing a FreeBSD 11 snapshot for arm64 and doing the initial setup process to get to a bootable FreeBSD system. If you are an extreme noob, not to fear. The tutorial walks you through setting up usernames, timezones, even a larger /tmp directory on your new MiniBSD setup. The tedious part comes to play during the setup of packages. The author walks us through setting up LibreSSL and various other packages via ports (Since LibreSSL isn’t the default in FreeBSD). This will take some time to compile on your humble RPi device. (Go make a sandwich, walk the dog, fix the gutters, etc) When it’s all said and done, you’ll end up with a secure little web-server that you’ve configured all by yourself! (Wondering what the word-press performance would be like on that box) *** Uber switches from PostgreSQL back to MySQL (https://eng.uber.com/mysql-migration/) We often hear success stories of people switching to PostgreSQL and getting huge performance gains, but this stories is the reverse Uber’s engineering team has switched back to MySQL, because for their specific workload and design, MySQL’s innodb has better performance Of course, it is not just vanilla MySQL, but “Schemaless”, a sharding system that sits on top of MySQL The article goes into detail about the on-disk format used by Postgres, and the specific shortcomings that Uber encountered Uber admits that all of its testing was against the older PostgreSQL 9.2, but one of their complaints is about having difficulty upgrading “We started out with Postgres 9.1 and successfully completed the upgrade process to move to Postgres 9.2. However, the process took so many hours that we couldn’t afford to do the process again. By the time Postgres 9.3 came out, Uber’s growth increased our dataset substantially, so the upgrade would have been even lengthier. For this reason, our legacy Postgres instances run Postgres 9.2 to this day, even though the current Postgres GA release is 9.5.” There is a followup, from the Postgres side “Why we lost Uber as a user” (https://www.postgresql.org/message-id/5797D5A1.5030009%40agliodbs.com) This thread goes into detail about the specific types of problematic queries that Uber was using “The Uber guy is right that InnoDB handles this better as long as you don't touch the primary key (primary key updates in InnoDB are really bad)” “This is a common problem case we don't have an answer for yet.” The thread then goes on to discuss possibly supporting a “pluggable heap storage layer”, to allow different workloads to use different on-disk formats for best performance *** Getting started with GhostBSD and FreeBSD Part 1 (http://unsolicitedbutoffered.blogspot.com/2016/07/getting-started-with-ghostbsd-and.html) Part 2 (http://unsolicitedbutoffered.blogspot.com/2016/07/getting-started-with-ghostbsd-and_31.html) Part 3 (http://unsolicitedbutoffered.blogspot.com/2016/08/getting-started-with-ghostbsd-and.html) Part 4 (http://unsolicitedbutoffered.blogspot.com/2016/08/getting-started-with-ghostbsd-and_2.html) In what may be our first GhostBSD tutorial, we have a nice walkthrough on the initial getting started with it. For those who don’t know, GhostBSD provides a nice XFCE or Mate desktop out of box, and still supports 32bit installs for those who want to keep that older hardware running. The walkthough takes us through the process of grabbing GhostBSD images and getting the installer up and running via bootable USB stick. Once booted, the graphical installer is straight-forward and short, allowing you to get the bits on disk as quickly as possible. (The actual installation took around 45 Minutes on an old Toshiba NB520) The author then takes us on a tour of some of GhostBSD’s out-of-box bundled applications (Along with XFCE) and how it compares to similar Linux setups. Lastly covered is the setup of Wireless (The manual way with WPA supplicant, since the GUI tool appeared to not work in this particular case) All in all a good walkthrough, especially if you’ve not seen GhostBSD in action before, the screenshots are very informative! *** News Roundup Steam on FreeBSD 11-CURRENT (https://github.com/SteamOnFreeBSD/SteamOnFreeBSD) Steam on FreeBSD. Yes, we’ve heard of setups using WINE, but what about running the Linux binaries natively? Well you are in luck. We have a github project that details getting the Linux native client up and running on a FreeBSD 11-CURRENT system. This github project is rather mysterious, with only the instructions to download a pre-packaged steam.txz file, extract and run the provided install.sh script. Curious I inspected some of the scripts, the installer.sh is fairly straight-forward, but does some ‘non-standard’ freebsd things, like fetching packages and extracting specific files/libaries into a new /compat/ubuntu directory. After that, it goes through a huge list of debian/ubuntu packages, also throwing them into the aforementioned ubuntu directory. At runtime, the wrapper script ensures that various linux compat file-systems are mounted in the correct location, then proceeds to run steam with some LDLIBRARYFLAGS set from the users .local/share/steam directory. A tad scary if honest, however it is a neat PoC to see Steam working on FreeBSD. Hopefully somebody can turn this into a more traditional package which can be easily removed / cleaned up afterwards. *** How to run Enlightenment on OpenBSD (http://enform.haxlab.org/) Are you an enlightenment fan? Be honest, we know there are quite a few of you out there! If so, we have a tutorial for you today, which talks about how to run E on OpenBSD. The process is pretty easy, but some steps might be overlooked if you are new to OpenBSD or don’t know how to tune / compile things on your own. It starts out with adjusting some sysctl’s for better tuning that works on an E based desktop. Next is installing from package some pre-reqs that will enable us to build E from source. After that, we need to download and install EFL from github, and the autogen / gmake commands are helpfully provided for you. Lastly the same is done for E itself, and TADA, E is installed and ready to go on your OpenBSD system. If you do this right, should only take 5-10 minutes to be up and running. *** Myths about FreeBSD (https://wiki.freebsd.org/Myths) Over on the FreeBSD wiki, we have a new “Myths” page which we’ve never highlighted on the show before. First up, and one I’ve very familiar with, is the usual “FreeBSD is only for Servers and not Desktops”, along with a good rebuttal about what it does offer and mention of projects such as PC-BSD which do it also. Another prevalent one is the “FreeBSD has a closed development model”, which is easily refuted: FreeBSD has over 400 developers around the world who have commit access to the repository. Many of these are willing to commit patches from third parties. If you want to get an idea of the number of patches that have been committed on behalf of other developers, then search for 'Submitted by' in the commit logs. At the time of writing, this is just under twenty thousand, or about ten percent of all commits. After having a few patches accepted, regular contributors are usually encouraged to apply for commit access. Another one that we are still hearing (Although it is less and less now) was the common “FreeBSD makes me compile everything from source”. Listeners of this show will know that pkg has pretty much made this irrelevant in recent years. However the option to compile yourself from source still exists, but most users won’t find this ever necessary. A good list, with many more items on it than we’ve mentioned here. Take a look, you might find something there you’ve heard in the wild, or maybe even thought yourself at one point! FreeBSD Area51 testing repo, KDE 5 (https://gist.github.com/swills/8a9a9f8a529256f134830c2a1476db0c) This github GIST from Steve Wills provides the instructions to enable the FreeBSD xorg teams unofficial ‘Area51’ testing repo on your machine This gives you access to the new KDE 5 Users should obviously be careful testing early-access software, but bug reports are very welcome, and important to getting KDE 5 working well under FreeBSD *** Beastie Bits NetBSD Machines at Open Source Conference 2016 Kyoto (http://mail-index.netbsd.org/netbsd-advocacy/2016/08/01/msg000712.html) How to install htop on pfSense (http://www.cyberciti.biz/faq/pfsense-install-htop-using-pkg-command/) The first ever patch for #Chromium on #NetBSD has been committed! (https://twitter.com/krytarowski/status/759909139300491265) CharmBug - Adventures in Hardened BSD - August 24th 2016 (http://www.meetup.com/CharmBUG/events/232474857/) Feedback/Questions Emanual - GoTTY (http://pastebin.com/q24sNcNY) Joe - Disable Device (http://pastebin.com/iTkmpVr9) Allen - ZFS Send/Recv (http://pastebin.com/bPHEP4Ya) Mica - Adobe Replacements (http://pastebin.com/L0ttuNx3) Robroy - RAID (http://pastebin.com/Dr92CGCU) ***
153: Big int trouble
This week on BSDNow, we have a variety of news to discuss, covering quite the spectrum of BSD. (Including a new DragonFly release!). This episode was brought to you by Headlines my int is too big (http://www.tedunangst.com/flak/post/my-int-is-too-big) “The NCC Group report (http://marc.info/?l=oss-security&m=146853062403622&w=2) describes the bugs, but not the history of the code.” “Several of them, as reported by NCC, involved similar integer truncation issues. Actually, they involved very similar modern 64 bit code meeting classic 32 bit code” “The thrsleep system call is a part of the kernel code that supports threads. As the name implies, it gives userland a measure of control over scheduling and lets a thread sleep until something happens. As such, it takes a timeout in the form of a timespec. The kernel, however, internally implements time keeping using ticks (there are HZ, 100, ticks per second). The tsleep function (t is for timed) takes an int number of ticks and performs basic validation by checking that it’s not negative. A negative timeout would indicate that the caller has miscalculated. The kernel panics so you can fix the bug, instead of stalling forever.” “The trouble therefore is when userland is allowed to specify a timeout that could be negative. The existing code made an attempt to handle various tricks by converting the timespec to a ticks value stored as a 64 bit long long which was checked against INTMAX before passing to sleep. Any value over INTMAX would be truncated, so we can’t allow that. Instead, we saturate the value to INT_MAX. Unfortunately, this check didn’t account for the possibility that the tick conversion from the timespec could also overflow and result in a negative value.” Then there is the description of the kqueue flaw: “Every kqueue keeps a list of all the attached events it’s watching for. A simple array is used to store file events, indexed by fd.” “This array is scaled to accommodate the largest fd that needs to be stored. This would obviously cause trouble, consuming too much memory, if the identifier were not validated first. Which is exactly what kqueue tries to do. The fdgetfile function checks that the identifier is a file that the process has open. One wrinkle. fdgetfile takes an int argument but ident is a uintptr_t, possibly 64 bits. An ident of 2^32 + 2 will look like a valid file descriptor, but then cause the array to be resized to gargantuan proportions.” “Again, the fix is pretty simple. We must check that the ident is bounded by INTMAX before calling fdgetfile. This bug likely would have been exploitable beyond a panic, but the array allocation was changed to use mallocarray instead of multiplying arguments by hand, thus preventing another overflow.” Then there is a description of the anonymous mmap flaw, and the “secret magic” _MAPNOFAULT flag *** FreeBSD Quarterly Status Report Q2 2016 (https://www.freebsd.org/news/status/report-2016-04-2016-06.html) It’s time for another round of FreeBSD Quarterly Status Reports! In this edition, we have status updates from the various teams, including IRC/Bugs/RE/Ports/Core and Foundation We also have updates on some specific projects, including from Konstantin on the on-going work for his implementation of ASLR, including the new ‘proccontrol’ command which provides the following: > “The proccontrol(1) utility was written to manage and query ASLR enforcement on a per-process basis. It is required for analyzing ASLR failures in specific programs. This utility leverages the procctl(2) interface which was added to the previous version of the patch, with some bug fixes.” Next are updates on porting CEPH to FreeBSD, the ongoing work to improve EFI+GELI (touched on last week) and more robust Mutexes. Additionally we have an update from Matt Macy and the Xorg team discussing the current work to update FreeBSD’s graphic stack: > “All Intel GPUs up to and including the unreleased Kaby Lake are supported. The xf86-video-intel driver will be updated soon. Updating this driver requires updating Xorg, which in turn is blocked on Nvidia updates.” The kernel also got some feature status updates, including on the new Allwinner SoC support, an update on FreeBSD in Hyper-V and VIMAGE In addition to a quick update on the arm64 architecture (It’s getting there, RPi3 is almost a thing), we also have a slew of port updates, including support for GitLab in ports, updates on GNOME / KDE and some additional Intel-specific networking tools. *** Vulnerabilities discovered in freebsd-update and portsnap (https://lists.freebsd.org/pipermail/freebsd-security/2016-July/009016.html) There are two vulnerabilities discovered in freebsd-update and portsnap, where an attacker could place files in the portsnap directory and they would be used without being subject to having their checksum verified (but this requires root access), and the second where a man-in-the-middle attacker could guess the name of a file you will fetch by exploiting the time-gap between when you download the initial snapshot, and when you fetch the updated files. There are a number of vulnerabilities that were discovered in libarchive/tar as well There is also an issue with bspatch. A security advisory for bspatch has already been released, as this vulnerabilities was also discovered by the Chromium team, which uses this same code. The patch discussed in this mailing list thread is larger, but secteam@ believes at least one of the additional checks introduced is incorrect and may prevent a valid patch from being applied. The smaller patch was pushed out first, to solve the main attack vector, while the larger patch is investigated. Automated fuzz testing is underway. Great care is being taken fixing bspatch, as if it is broken installing future updates becomes much more difficult secteam@ and core@ would like to emphasize that the FreeBSD project takes these issue very seriously and are working on it > “As a general rule, secteam@ does not announce vulnerabilities for which we don't have patches, but we concede that we should have considered making an exception in this case” Work is underway to re-architect freebsd-update and portsnap to do signature verification on all files before they are passed to libarchive/tar, to help protect users from any future vulnerabilities in libarchive. However, this requires changes to the metadata format to provide these additional signatures, and backwards compatibilities must be preserved, so people can update to the newer versions to get these additional security features There is also discussion of using HTTPS for delivery of the files, but certificate verification and trust are always an issue. FreeBSD does not distribute a certificate trust store by default. There will be more on this in the coming days. *** OpenSSH 7.3 Released (http://www.openssh.com/txt/release-7.3) OpenSSH 7.3 has landed! Primarily a bug-fix release, the release notes do mention the pending deprecation of some more legacy Crypto in the future, including denying all RSA keys < 1024bit, and removal of SSHv1 support. (Already disabled via compile option) On the bug side, there was a security issue addressed in sshd: “sshd(8): Mitigate a potential denial-of-service attack against the system's crypt(3) function via sshd(8). An attacker could send very long passwords that would cause excessive CPU use in crypt(3). sshd(8) now refuses to accept password authentication requests of length greater than 1024 characters” Also a timing issue was resolved in regard to password auth, which could possibly allow an attacker to discern between valid/invalid account names. On the feature side, we have the new ProxyJump option (-J flag) which allows you to do simplified indirection through various SSH jump hosts. Various bugs were fixed, and some compile failures resolved in the portable version to auto-disable some ciphers not supported by OpenSSL. News Roundup OpenBSD Ports - Integrating Third Party Applications [pdf] (http://jggimi.homeip.net/semibug.pdf) A talk from Josh Grosse, presented at SEMIBUG (South-East Michigan BSD Users Group), about OpenBSD Ports It opens by explaining the separation of the ‘base system’ from ‘packages’, as is common in most all BSDs It explains the contents of OpenBSD package tar file, which contain some metadata files (+CONTENTS and +DESC) and then the actual package files The talk goes on to explain the different branches (-release, -stable, and -current), and warn users that there are no official -stable packages from the project Then it goes on into the development model, including what new contributors should expect Then it walks through the entire process of creating a port and getting it contributed *** NetBSD removes last RWX page in amd64 kernel (http://mail-index.netbsd.org/source-changes/2016/07/27/msg076413.html) NetBSD has purged the last holdout RWX page on the amd64 platform > “Use UVMPROTALL only if UVMKMFEXEC is given as argument. Otherwise, if UVMKMFPAGEABLE is also given as argument, only the VA is allocated and UVM waits for the page to fault before kentering it. When kentering it, it will use the UVMPROT flag that was passed to uvm_map; which means that it will kenter it as RWX. With this change, the number of RWX pages in the amd64 kernel reaches strictly zero.” Break out the party favors! Hopefully any last stragglers in any of the other BSD’s gets retired soon as well. *** DragonFly BSD 4.6 launches with home-grown support for NVMe Controllers (http://linux.softpedia.com/blog/dragonfly-bsd-4-6-0-launches-with-home-grown-support-for-nvme-controllers-506908.shtml) Softpedia picked up on the release of DragonFlyBSD 4.6, specifically about their new home-grown NVMe driver. > “We now have a NVMe driver (PCIe SSDs). It currently must be kldloaded with nvme_load="YES" in /boot/loader.conf. The driver uses all concurrency features offered by the chip and will distribute queues and interrupts across multiple CPUs to maximize performance. It has been tested up to around 1.05M IOPS @4K, and roughly 6.5 GBytes/sec @32K (random read from urandom-filled partition, physio, many threads), with the 2xE5-2620v4 (xeon) test server 78% idle in the IOPS test and 72% idle on the bandwidth test. In other words, we maxed out the three NVMe devices we had plugged in and the system still had plenty of suds left over. Please note that a machine's ability to boot from an NVMe device depends on the BIOS, and not DragonFly. Most BIOSes cannot boot from NVMe devices and those that can probably only do it through UEFI. Info on device state is available with the new utility nvmectl.“ In addition to this improved support, 4.6 also brings in the improved graphics support, matching what is in Linux 4.4 and support for Broadwell/Skylake. SMP also got some love: > “SMP performance was already very good. As part of the NVMe driver work we revamped the buffer cache subsystem and a number of other I/O related paths, further reducing lock contention and IPI signalling overheads. We also put topology-aware cpu cache localization into the kernel memory allocator (primarily helps multi-socket systems and systems with high core counts). The network subsystem also continues to receive significant improvement, with modest machine configurations now capable of handling upwards of 580K conns/sec.“ +Full Release Notes (https://www.dragonflybsd.org/release46/) *** The powerd++ daemon monitors the system load and adjusts the CPU clock accordingly and is a drop-in replacement for FreeBSD's native powerd(8). (http://www.freshports.org/sysutils/powerdxx/) As mentioned in our EuroBSDCon 2016 rundown, Dominic Fandrey will be giving a presentation about his powerd replacement, powerd++ The source code is already available on github, and is in ports The major difference is the newer design handle many-core systems much better. The original powerd was written at a time when most laptops only had a single core, and maybe a hyperthread. The new design decides which CPU frequency to use by looking at the busiest core, rather than the average across the cores, resulting in a more meaningful result. It also supports averaging over a longer period of time, to avoid jumping to a higher frequency to quickly powerd++ also avoids ‘slewing’ the cpu frequency, ratching it up and down one step at a time, and instead jumps directly to the target frequency. Often times, you will use less battery by jumping to maximum frequency, finishing the work, and going back to a low power state, than trying to do that work over a longer period of time in low power mode *** Beastie Bits Hyper-V: Unmapped I/O improves userland direct disk performance by 35% ~ 135% (https://svnweb.freebsd.org/base?view=revision&revision=303474) One does not simply remove FreeBSD (https://imgur.com/a/gjGoq) A new BSD Podcast "BSD Synergy" has started (https://www.youtube.com/channel/UCBua6yMtJ6W5ExYSREnS3UQ) KnoxBug - Next Meeting - Aug 30th (http://knoxbug.org/content/2016-08-30) Feedback/Questions Daniel - Root/Wheel (http://pastebin.com/8sMyKm6c) Joe - IPV6 Frag (http://pastebin.com/r5Y0gbxf) Paul - ChicagoBug (http://pastebin.com/iVYPYcVs) Chris - SSH BruteBlock (http://pastebin.com/597m9gHa) Todd - Jails (http://pastebin.com/xjbKwSaz) ***
152: The Laporte has landed!
This week on BSDNow, we have some big breaking news about another major switcher to FreeBSD, plus early information about the pending This episode was brought to you by Headlines Leo Laporte tries FreeBSD (http://www.leolaporte.com/blog/a-grand-experiment) Leo Laporte, formerly of TechTV, and now of TWiT.tv, is switching to FreeBSD “The latest debacle over the "forced" upgrade to Windows 10 and Apple's increasingly locked-in ecosystem has got me thinking. Do I really need to use a proprietary operating system to get work done? And while I'm at it, do I need to use commercial cloud services to store my data?” A sometimes Linux user since the mid 90s, Leo talks about his motivations: “But as time went by, even Ubuntu began to seem too commercial to me” “So now for the grand experiment. Is it possible, I wonder, to do everything I need to do on an even more venerable, more robust system: a true UNIX OS, FreeBSD? Here are my requirements” Browsing Email with PGP signing and encryption Coding - I'm a hobbyist programmer requiring support for lisp/scheme/racket, rust, and python (and maybe forth and clojure and meteor and whatever else is cool and new) Writing A password vault. I currently use Lastpass because it syncs with mobile but eventually I'll need to find a FOSS replacement for that, too Photo editing - this is the toughest to replace. I love Photoshop and Lightroom. Can I get by with, say, GIMP and Darktable? I do all of those things on my PCBSD machine all the time “I love Linux and will continue to use it on my laptops, but for my main workhorse desktop I think FreeBSD will be a better choice. I also look forward to learning and administering a true UNIX system.” He got a nice SuperMicro based workstation, with an Intel Xeon E3-1275v5 and an NVIDIA GeForce GTX 960 GPU I have a server with one of those Skylake E3s, it is very nice “450Mbps Wireless N Dual Band PCI-e Adapter w/ 3x 2dBi Antennas (Yes, sad to say, unless I rewire my house I'll have to use Wi-Fi with this beast. I'll probably rewire my house.)” He plans to have a 4x 1TB ZFS pool, plus a second pool backed by a 512 GB NVMe m.2 for the OS “And I'll continue to chronicle my journey into the land of FOSS here when The Beast arrives. But in the meantime, please excuse me, I've got some reading to do.” Leo went so far as to slap a “Power By FreeBSD” sticker (https://youtu.be/vNVst_rxxm0?t=270) on the back of his new Tesla *** OpenBSD 6.0 to be released on Sept 1st, 2016 (http://undeadly.org/cgi?action=article&sid=20160725100831) OpenBSD 6.0 Tenative Released Notes (https://www.openbsd.org/60.html) OpenBSD 6.0 is just around the corner, currently slated for Sept 1st and brings with it a whole slew of exciting new features First up, and let’s get this right out of the way.. VAX support has been dropped!! Oh no! However to make up for this devastating loss, armv7 has been added to this release. The tentative release notes are very complete and marks 6.0 as quite an exciting release OpenBSD 6.0 Pre-orders up (http://undeadly.org/cgi?action=article&sid=20160726230851) OpenBSD 6.0 tightens security by losing Linux compatibility (http://www.infoworld.com/article/3099038/open-source-tools/openbsd-60-tightens-security-by-losing-linux-compatibility.html) In related news, infoworld picked up on the pending removal of Linux compat from OpenBSD 6.0. Touted as a security feature, you will soon be unable to run legacy linux binaries on OpenBSD. This has both positives and negatives depending upon your use case. Ironically we’re excitedly awaiting improved Linux Compat support in FreeBSD, to allow running some various closed-source applications. (Netflix DRM, Steam, Skype to name a few) *** EuroBSDCon 2016 Schedule released (https://2016.eurobsdcon.org/talks-schedule/) EuroBSDCon 2016 Tutorial Schedule released (https://2016.eurobsdcon.org/tutorials/) EuroBSDCon has announced the list of talks and tutorials for September 22nd-25th’s conference! George Neville Neil (Who we’ve interviewed in the past) is giving the keynote about “The Coming Decades of BSD” *** News Roundup Blast from the past No interview again this week, we’re working on getting some people lined up. The Leo Laporte story brought these old gem from TechTV into my youtube playlist: Matt Olander and Murrey Stokey explain FreeBSD on TechTV (https://www.youtube.com/watch?v=d0UsXwRvaIg) Matt Olander and Brooks Davis explain building a cluster with FreeBSD on TechTV (https://www.youtube.com/watch?v=bAsYz5pVwyc) FreeBSD vs Linux Part 1 (https://www.youtube.com/watch?v=91igg2UX7o8) FreeBSD vs Linux Part 2 (https://www.youtube.com/watch?v=oU88fQkwfws) *** Running FreeBSD on the LibreM (https://ericmccorkleblog.wordpress.com/2016/07/16/freebsd-librem-update/) Eric McCorkle (Who has worked on the EFI loader for a while now) has written an update on his efforts to get FreeBSD working properly on the LibreM 13 laptop. Since April the work seems to be progressing nicely Matt Macy’s i915 graphics patch works well on the Librem 13, and I personally made sure that the suspend/resume support works. The patch is very stable on the Librem, and I’ve only had one kernel panic the entire time testing it. The HDMI output Just Works™ with the i915 driver. Even better, it works for both X11 and console modes. Full support for the Atheros 9462 card has been merged in. I’ve had some occasional issues, but it works for the most part. The vesa weirdness is obviated by i915 support, but it was resolved by using the scfb driver. Some of the outstanding issues still being worked on are support for Synaptics on this particular touchpad, as well as hotkey support for the keyboard, and brightness controls. In addition Eric is still working on the EFI + Geli support, with the eventual goal of getting EFI secure-boot working out of box as well. More OpenBSD syscall fuzzing (http://seclists.org/oss-sec/2016/q3/157) NCC Group’s Project Triforce continues its work of fuzzing OpenBSD This time they have found a flaw that allows any user to panic the kernel Attempting to read from the tmpfs_vfsops sysctl tree will panic the system: “attempt to execute user address 0x0 in supervisor mode” This is actually a “good” thing… “Impact: Any user can panic the kernel by using the sysctl call. If a user can manage to map a page at address zero, they may be able to gain kernel code execution and escalate privileges” OpenBSD’s default configuration prevents mapping a page at address zero, so the code execution is prevented So while a panic is a bad outcome, it is a lot better than it could have been *** Root privilege escalation on NetBSD (http://akat1.pl/?id=2) This post described a root privilege escalation in NetBSD mail.local is a utility included in the base system for delivering mail to other users on the same system, rather than invoking a mail client and going through the mail server. The mail.local utility contains a ‘time of check / time of use’ vulnerability. This means that it checks if a file or permission is valid, and then later accesses that file. If an attacker can change that file between the time when it is checked, and the time when it is used, they may be able to exploit the system by evading the check This is exactly what happens in this case mail.local appends a message to the indicated user’s mailbox It first checks if the target user already has an existing mailbox file. If the file exists, but is a link, mail.local exits with an error (to prevent exploits) If the file does not exist, it is created The message is then appended to the file If the file needed to be created, it is chown’d to the owner of the mailbox This is where the problem lies, if mail.local checks and does not find the mailbox, but an attacker then creates a link from the target mailbox to some other file mail.local then appends to that file instead, thinking it is creating the new mailbox Then, mail.local chown’s the target file to the user the attacker was trying to send mail to The article explains how this could be used to replace /etc/master.passwd etc, but opts for an easier proof of concept, replacing /usr/bin/atrun, which is run as root every 5 minutes from crontab with a script that will copy the shell to /tmp and mark it setuid The attacker can then run that shell out of /tmp, and be root NetBSD fixed the vulnerability by changing the code flow, separating the cases for opening an existing file from creating a new file. In the case where an existing file is opened, the code then verifies that the file that was opened has the same inode number and is on the same device, as the file that was checked earlier, to ensure it was not a link *** FreeBSD Heap vulnerability in bspatch (https://www.freebsd.org/security/advisories/FreeBSD-SA-16:25.bspatch.asc) An important vuln has been found and fixed in FreeBSD this past week, specifically relating to the ‘bspatch’ utility. “Upstream's bspatch.c implementation doesn't check for negative values on the number of bytes to read from the "diff" and "extra" streams, allowing an attacker controlling the patch file to write at arbitrary locations in the heap.” This could result in a crash, or running arbitrary code as the user running bspatch. (Often root) “bspatch's main loop reads three numbers from the "control" stream in the patch: X, Y and Z. The first two are the number of bytes to read from "diff" and "extra" (and thus only non-negative), while the third one could be positive or negative and moves the oldpos pointer on the source image. These 3 values are 64bits signed ints (encoded somehow on the file) that are later passed the function that reads from the streams, but those values are not verified to be non-negative.” “Chrome[OS] has four different implementations of this program, all derived from the same original code by Colin Percival.” Chromium Issue Tracker (https://bugs.chromium.org/p/chromium/issues/detail?id=372525) Patch your systems now! *** Beastie Bits: If you're a BUG member or Organizer, please contact BSD Now (https://twitter.com/q5sys/status/758087886927388673) TedU writes about some interesting localizations to gcc in openbsd, and why they are there (http://www.tedunangst.com/flak/post/one-reason-to-hate-openbsd) List of Products based on FreeBSD -- Help complete the list (https://en.wikipedia.org/wiki/List_of_products_based_on_FreeBSD) Virtualbox v5 hits the FreeBSD Ports tree (http://www.freshports.org/emulators/virtualbox-ose/) Skull Canyon NUC booting FreeBSD 11.0-BETA2 (https://gist.github.com/gonzopancho/b71be467f45594822131f4816d6cb718) 2016 BSDCan Trip Report : Trent Thompson (https://www.freebsdfoundation.org/blog/2016-bsdcan-trip-report-trent-thompson/) August London BSD Meetup (http://mail-index.netbsd.org/regional-london/2016/07/25/msg000542.html) Feedback/Questions Michael Open-Source Alts (http://pastebin.com/eiWbDXTd) Herminio - AP Troubles (http://pastebin.com/w9aCDBut) Jake - Plasma (http://pastebin.com/d15QpVFw) Morgan - Clean DO Droplets (http://pastebin.com/Wj1P7jq8) Chris - Auditd (http://pastebin.com/U9PYEH6K) ***
151: Fuzzy Auditing
This week on BSDNow, we have all sorts of interesting news, including a Kernel Fuzzing audit done for OpenBSD, a much improved This episode was brought to you by Headlines Multiple Bugs in OpenBSD Kernel (http://marc.info/?l=oss-security&m=146853062403622&w=2) Its patch Wednesday! (OR last Thursday if you were watching the mailing lists) Jesse Hertz and Tim Newsham (part of the NCC Group calling themselves project Triforce) have been working with the OpenBSD team to fix some newly discovered bugs in the kernel using fuzzing. Specifically they were able to track down several potential methods to corrupt memory or panic the kernel: mmap_panic: Malicious calls to mmap() can trigger an allocation panic or trigger memory corruption. kevent_panic: Any user can panic the kernel with the kevent system call. thrsleeppanic: Any user can panic the kernel with the _thrsleep system Call. thrsigdivertpanic: Any user can panic the kernel with the _thrsigdivert system call. ufsgetdentspanic: Any user can panic the kernel with the getdents system call. mount_panic: Root users, or users on systems with kern.usermount set to true, can trigger a kernel panic when mounting a tmpfs filesystem. unmount_panic: Root users, or users on systems with kern.usermount set to true, can trigger a kernel panic when unmounting a filesystem. tmpfsmknodpanic: Root can panic kernel with mknod on a tmpfs filesystem. This was a great find, and we have a link to more of the results, if you would like to explore them in more detail. NCC Group OpenBSD Kernel fuzzing results (http://www.openwall.com/lists/oss-security/2016/07/14/5) Would like to see more work like this done in all of the BSDs *** Running CockroachDB in a FreeBSD Jail (https://www.cockroachlabs.com/blog/critters-in-a-jar-running-cockroachdb-in-a-freebsd-jail/) The developers behind CockroachDB have written up a nice walkthrough of getting their software to run inside FreeBSD jails. “Manually encapsulating CockroachDB using Linux cgroups is no easy task, which is why tools like Docker exist in the first place. By comparison, running server processes natively in FreeBSD jails is straightforward and robust.” The walkthrough begins with compiling CockroachDB straight from source (A port is pending), which is pretty easy relying upon bash / git / gmake and GO. With the compile finished, the next step will be mounting linprocfs, although that may be going away in the future: “(Note: Linux compatibility files / packages / libraries are not needed further. CockroachDB uses Linux’s procfs to inspect system properties via gosigar. If/when gosigar evolves to read FreeBSD properties natively, CockroachDB will not need linprocfs any more.)” With the initial setup complete, the walkthrough then takes us through the process of creating the rc.d script (Which should be included with the port) and ultimately setting up ezjail and deploying CockroachDB within. With the word getting out about jails and their functionality, we hope to see more projects also provide walkthroughs and FreeBSD support natively. Kudos to the CockroachDB team! *** Usermount bugs (https://marc.info/?l=openbsd-announce&m=146854517406640&w=2) kern.usermount, (vfs.usermount on FreeBSD) is a sysctl that can be enabled to allow an unprivileged user to mount filesystems. It is very useful for allowing non-root users to mount a USB stick or other external media. It is not without its dangers though: > “kern.usermount=1 is unsafe for everyone, since it allows any non-pledged program to call the mount/umount system calls. There is no way any user can be expected to keep their system safe / reliable with this feature. Ignore setting to =1, and after release we'll delete the sysctl entirely.” In OpenBSD 6.0 and forward, the setting will no longer work, and root privileges will be required to mount a filesystem If there is a bug in the filesystem driver, the user could potentially exploit that and root the system > “In addition to the patched bugs, several panics were discovered by NCC that can be triggered by root or users with the usermount option set. These bugs are not getting patched because we believe they are only the tip of the iceberg. The mount system call exposes too much code to userland to be considered secure” This is a very pragmatic way of dealing with these issues, as it is not really possible to be sure that EVERY bug has been fixed, and that this feature is no longer an exploit vector usermount being removed from OpenBSD (http://undeadly.org/cgi?action=article&sid=20160715125022) I use this facility in FreeBSD extensively, combined with ZFS permission delegation, to allow non-root users to create and mount new ZFS datasets, and to do replication without requiring any root access There are some safety belts, for instance: the user must own the directory that the new filesystem will be mounted to, so they can’t mount to /etc and replace the password file with their own *** Let's Encrypt client from BSD in C (https://kristaps.bsd.lv/letskencrypt/) File this one under the category of “It’s about time!”, but Kristaps (Who we’ve interviewed in the past) has released some new software for interacting with letsencrypt. The header for the project site sums it up nicely: “Be up-front about security: OpenSSL is known to have issues, you can't trust what comes down the pipe, and your private key's integrity is a hard requirement. Not a situation where you can be careless. letskencrypt is a client for Let's Encrypt users, but one designed for security. No Python. No Ruby. No Bash.A straightforward, open source implementation in C that isolates each step of the sequence.” What specifically does it isolate you ask? Right now it is broken down into 6 steps: read and parse an account and domain private key authenticate with the Let's Encrypt server authorise each domain listed for the certificate submit the X509 request receive and serialise the signed X509 certificate request, receive, and serialise the certificate chain from the issuer I don’t know about all of you, but I’m going to be switching over one of my systems this weekend. *** News Roundup Videos from the FOSDEM BSD Dev room are now online (https://video.fosdem.org/2016/k4601/) The videos from the BSD Dev room at FOSDEM have been stealthily posted online at some point since last I checked The videos are individually linked from the talks on the Schedule (https://archive.fosdem.org/2016/schedule/track/bsd/) The talk pages also include the slides, which can help you to follow along *** FreeBSD on Jetson TK1 (http://kernelnomicon.org/?p=628) The nVidia Jetson TK1 is a medium sized ARM device that is a big more than your standard Raspberry Pi The device has: NVIDIA 4-Plus-1™ Quad-Core ARM® Cortex™-A15 CPU (2.3 GHz) NVIDIA Kepler GPU with 192 CUDA Cores 2 GB DDR3L x16 Memory with 64-bit Width 16 GB 4.51 eMMC Memory 1 Half Mini-PCIE Slot 1 Full-Size SD/MMC Connector 1 Full-Size HDMI Port 1 USB 2.0 Port, Micro AB 1 USB 3.0 Port, A 1 RS232 Serial Port 1 ALC5639 Realtek Audio Codec with Mic In and Line Out 1 RTL8111GS Realtek GigE LAN 1 SATA Data Port SPI 4 MByte Boot Flash The following signals are available through an expansion port: DP/LVDS Touch SPI 1x4 + 1x1 CSI-2 GPIOs UART HSIC i2c The device costs $192 USD from nVidia or Amazon Oleksandr Tymoshenko (gonzo@freebsd.org) has a post describing what it takes to get FreeBSD running on the Jetson TK1 > “First of all – my TK1 didn’t have U-Boot. Type of bootloader depends on the version of Linux4Tegra TK1 comes with. Mine had L4T R19, with some kind of “not u-boot” bootloader.” They tried using the provided tool, compiled on FreeBSD since it uses libusb, but it gave an error. Falling back to trying from Ubuntu, they got the same error. They then flashed the TK1 with newer firmware, and suddenly, uboot is available. The post then walks through pxe booting FreeBSD on the TK1 The guide then walks through replacing the UBoot with a version compatible with UBLDR, for more features We’ll have to wait for another post to get FreeBSD burned onto the device, but at this point, you can reliably boot it without any user interaction I have one of these devices, so I am very interested in this work *** Why we use OpenBSD at VidiGaurd (https://blog.vidiguard.com/why-we-use-openbsd-at-vidiguard-4521f217b2b7#.9r86v742v) VidiGuard (Which makes autonomous drone solutions for security monitoring) has posted an interesting write-up on why they use OpenBSD. Specifically they start by mentioning while they are in business to provide physical security, they just as equally value their data security, especially their customer data. They name 4 specific features that matter to them, starting with Uncompromising Quality and Security: “Over the past 20 years, OpenBSD’s focus on uncompromising quality and code correctness has yielded an operating system second-to-none. Code auditing and review is core to the project’s development process. The team’s focus on security includes integrated cryptography, new security mitigation techniques, and an optional-security-is-no-security stance, making it arguably the most secure operating system available today. This approach pays off in the form of only a few security updates for a given release, compared to other operating systems that might release a handful of updates every week.” High praise indeed! They also mention the sane-defaults, documentation and last but not least, the license as also winning factors in making OpenBSD their operating system of choice. Thanks to VidiGuard for publically detailing the use of BSD, and we hope to see other business follow suit! *** "You can (and should) slow down and learn how things work" – Interview with Dru Lavigne (https://bsdmag.org/dru_lavigne/) If you’ve been around the BSD community for any length of time, you no doubt have heard of Dru Lavigne (Or perhaps own one of her books!) She was recently interviewed by Luca Ferrari for BSD Magazine and you may find it a fascinating read. The 2nd question asked sounded a lot like our opener to an interview (How did you get into BSD) “ In the mid 90s, I went back to school to learn network and system administration. As graduation grew near and I started looking for a work, I noticed that all the interesting jobs wanted Unix skills. Wanting to increase my skills, and not having any money, I did an Internet search for “Free Unix”. The first hit was freebsd.org. I went to the website and started reading the Handbook and thought “I can do this”. Since I only had access to one computer and wanted to ramp up my skills quickly, I printed out the installation and networking chapters of the Handbook. I replaced the current operating system with FreeBSD and forced myself to learn how to do everything I needed to do on that computer in FreeBSD. It was a painful (and scary) few weeks as I figured out how to transition the family’s workflow to FreeBSD, but it was also exhilarating to learn that “yes, I can do this!. Since then, I’ve had the opportunity to try out or administer the other BSDs, several Linux distros, SCO, and Solaris. I found that the layout, logic, and release engineering process of the BSDs makes the most sense to me and I’m happiest when on a BSD system.” When asked, Dru also had a good response to what challenges potential new UNIX or BSD users may face: “Students who haven’t been exposed to open source before are used to thinking of technology in terms of a purchasable brand consisting of “black boxes” that are supposed to “just work”, without having to think about how they work. You can (and should) slow down and learn how things work. It can be a mind shift to learn that the freedom to use and change how something works does exist, and isn’t considered stealing. And that learning how something works, while hard, can be fun. BSD culture, in particular, is well suited for those who have the time and temperament to dive into how things work. With over 40 years of freely available source and commit messages, you can dive as deep as you want into learning how things came to be, how they evolved over the years, how they work now, and how they can be improved. There is a diverse range of stuff to choose from: from user tools to networking to memory management to hardware drivers to security mechanisms and so on. There is also a culture of sharing and learning and encouragement for users who demonstrate that they have done their homework and have their own ideas to contribute.” The interview is quite long, and Dru provides fantastic insights into more aspects of BSD in general. Well worth your time to read! *** Beastie Bits: Ed Maste is seeking testing 'withoutgpldtc' (https://twitter.com/ed_maste/status/755474764479672321) “PAM Mastery” tech reviewers wanted (http://blather.michaelwlucas.com/archives/2717) OPNsense 16.7 RC2 (https://opnsense.org/opnsense-16-7-rc2-released/) Jupyter Notebook for bootstrapping Arduino on FreeBSD (https://nbviewer.jupyter.org/github/DadAtH-me/Projects/blob/master/arduino-on-nix.ipynb) The Design and Implementation of the Anykernel and Rump Kernels (second edition) (http://www.fixup.fi/misc/rumpkernel-book/) Complete desktop synchronisation with Unison and FreeBSD jails (xjails) (https://github.com/kbs1/freebsd-synced-xjails) Feedback/Questions Eric - List most popular files (http://pastebin.com/S7u0VeVi) Robroy - ZFS Write Cache (http://pastebin.com/81Zmj0cX) Luis - FreeNAS HW Setup (http://pastebin.com/SfeKR7v2) Emett - Python Followup (http://pastebin.com/wy4ar0YH) Peter - Multicast + Jails (http://pastebin.com/zd2QAu25) ***
150: Sprinkle a little BSD into your life.
Today on the show, we are going to be talking to Jim Brown (of BSD Cert Fame) about his home-brew sprinkler system… Wait for it… This episode was brought to you by Headlines Distrowatch reviews OpenBSD and PCBSD's live upgrade method (http://distrowatch.com/weekly.php?issue=20160620#upgrade) Upgrading… The bane of any sysadmin! Distrowatch has recently done a write-up on the in-place upgrading of various distros / BSDs including PC-BSD and OpenBSD. Lets look first at the PC-BSD attempt, which was done going from 9.2 -> 10. “I soon found trying to upgrade either the base system or pkg would fail. The update manager did not provide details as to what had gone wrong and so I decided to attempt a manual upgrade by following the FreeBSD Handbook as I had when performing a live upgrade of FreeBSD back in May. At first the manual process seemed to work, downloading the necessary patches for FreeBSD 10 and getting me to resolve conflicts between my existing configuration files and the new versions. Part way through, we are asked to reboot and then continue the upgrade process using the freebsd-update command utility. PC-BSD failed to reboot and, in fact, the boot loader no longer found any operating systems to run.” Ouch! I’m not sure on the particular commands used, but to lose the boot-loader indicates something went horribly wrong. There is good news in this though. After the pain experienced in the 9.X upgrade process, 11.0 has been vastly improved to help fix this going forward. The updater is also self-updating, which means future changes to tools such as package can be accounted for in previously released versions. Moving on to OpenBSD, Jesse had much better luck: > “The documentation provided explains how to upgrade OpenBSD 5.8 to version 5.9 step-by-step and the instructions worked exactly as laid out. Upgrading requires two reboots, one to initiate the upgrade process and one to boot into the new version of OpenBSD. Upgrading the base operating system took approximately ten minutes, including the two reboots. Upgrading the third-party packages took another minute or two. The only quirk I ran into was that I had to manually update my repository mirror information to gain access to the new packages available for OpenBSD 5.9. If this step is not done, then the pkg_add package manager will continue to pull in packages from the old repository we set up for OpenBSD 5.8. “ A good read, and they covered some Linux distros such as Mint and OpenMandriva as well, if you want to find out how they fared. *** A curated list of awesome DTrace books, articles, videos, tools and resources (http://awesome-dtrace.com/) The website awesome-dtrace.com compiles a list of resources, including books, articles, videos, tools, and other resources, to help you get the most out of DTrace The list of books includes 2 open source books that are available on the web, and of course Brendan Gregg’s official DTrace book There are also cheat sheets, one-liner collections, and a set of DTrace war stories A breakdown of different PID providers and the userspace statically defined tracepoints The videos from DTrace.conf 2008, 2012, and soon 2016 And links to the tools to start using DTrace with your favourite programming language, including Erlang, Node.JS, Perl, PHP, Python, or Ruby There are also DTrace setups for MySQL/MariaDB, and PostreSQL Joyent has even written a mod_usdt DTrace module for the Apache web server This seems like a really good resource, and with the efforts of the new OpenDTrace project, to modernize the dtracetoolkit and make it more useful across the different supported operating systems, there has never been a better time to start learning DTrace *** Installing OpenBSD using a serial console with no external monitor (http://unix.stackexchange.com/questions/292891/how-can-i-install-openbsd-using-the-serial-console-without-external-monitor-wi) Have you found yourself needing to install OpenBSD from USB, but with a twist, as in no external monitor? Well somebody has and asked the question on stackexchange. The answer provided is quite well explained, but in a nut-shell the process involves downloading the USB image and making some tweaks before copying it to the physical media. Specifically with a couple of well-placed echo’s into boot.conf, the serial-port can be enabled and ready for use: echo "stty com0 115200" > /mnt/etc/boot.conf echo "set tty com0" >> /mnt/etc/boot.conf + After that, simply boot the box and you are ready to access the serial console and drive the installation as normal! #bsdhacks GSoC 2016 Reports: Split debug symbols for pkgsrc builds (https://blog.netbsd.org/tnf/entry/gsoc_2016_reports_split_debug) The NetBSD blog provides a status report on one of the GSoC projects that is nearing its midterm evaluation The project to split debugging data into separate pkgsrc packages, so that users can install the debugging symbols if they need them to debug a failing application The report is very detailed, and includes “A quick introduction to ELF and how debug information are stored/stripped off” It walks through the process of writing a simple example application, compiling it, and dealing with the debug data It includes a number of very useful diagrams, and a summary of what changes needed to be make to the pkgsrc makefile infrastructure With this as a recipe, someone should be able to do something quite similar for FreeBSD’s ports tree *** iXsystems iXsystems’ TrueNAS Firmware Update Delivers Compelling Performance, Replication, and Graphing Improvements (https://www.ixsystems.com/blog/ixsystems-truenas-firmware-update-delivers-compelling-performance-replication-graphing-improvements/) *** Interview - Jim Brown - jpb@jimby.name (mailto:jpb@jimby.name) FreeBSD+BBB Sprinkler System News Roundup From the past : A Research Unix Reader (http://www.cs.dartmouth.edu/~doug/reader.pdf) A paper by by Douglas McIlroy “Selected pages from the nine research editions of the UNIX® Programmer’s Manual illustrate the development of the system” “Accompanying commentary recounts some of the needs, events, and individual contributions that shaped this evolution.” Interesting insight into the evolution of the origin UNIX operating system *** Evolution of C programming practices – Unix 1973–2015 (http://kristerw.blogspot.com/2016/06/evolution-of-c-programming-practices.html) From the author of the recent post we covered, “20 years of NetBSD code bloat”, comes a new post “I found a recent paper that also looks at how the BSD code base has evolved, but from a very different perspective compared to my code-size investigation.” The paper "The Evolution of C Programming Practices: A Study of the Unix Operating System 1973–2015" investigates coding style, and tests seven hypotheses by looking at metrics (line length, number of volatile in the source code, etc.) in 66 releases of Unix from 1973 to 2014. The hypotheses are: > + Programming practices reflect technology affordances (e.g. developers may be more liberal with screen space when using high resolution displays) > + Modularity increases with code size > + New language features are increasingly used to saturation point > + Programmers trust the compiler for register allocation > + Code formatting practices converge to a common standard > + Software complexity evolution follows self correction feedback mechanisms > + Code readability increases and the result is that they seem to be true, as interpreted through the metrics. > “The data points for the releases have somewhat random dates. One issue is that the paper use each release's mean file date (the average of the files' last modification time) instead of the release date (that is why the graphs stop at November 2010, even though FreeBSD 10 was released in 2014). The idea is that this better reflects the age of the code base, but this has the effect of compressing some of the data points (especially the clustering around 1993-1994), and it makes the spline fitting even more suspect.” > “One other problem is that the original data used by the researchers seems to have incorrect timestamps. For example, 4.3BSD Net/1 was released in 1989, but is listed as 1993-12-25 in the paper. The same is true for at least the Net/2 release too, which was released in 1991, but the paper list it as 1993-07-02.” *** [old release pictures] openbsd 2.1 - 5.9, straight from theo's bookshelf. (https://twitter.com/blakkheim/status/747540167112671232) Speaking of old releases, our Producer JT picked up this gem at Southeast Linuxfest this year (https://twitter.com/q5sys/status/748003859012984837) Noah Axon shares a scan of his NetBSD 1.4 disc (https://plus.google.com/+NoahAxon/posts/VsiQhUn3tHb) Jan van den broek shares a pic of his FreeBSD 2.2.5 set (https://plus.google.com/101232368324501316985/posts/4QsaJE2KxXh) *** FreeBSD: Just in Time (https://bsdmag.org/just_in_time/) Another BSDMag goodie this week, we have a small article written by Jonathan Garrido which details their experience switching to FreeBSD for a NTP server. The article is short, but a good read: > “A Few years ago we had a time problem. Suddenly our linux NTP server, for a reason that I still do not know, started to fail giving us a lot of issues within all the equipment and services within our network. After a quick and brief meeting with management, I found out that there was not sufficient budget left for a fancy and well-suited appliance. So, with no time (literally) and no money to spend, I decided to give it a try and utilized a homemade open source solution, and the operating system of choice was FreeBSD 10.0.” “Now, let’s pause for a second. You may be thinking, why in the world is this guy doing this, when he has never installed a BSD machine in his life? The answer is very simple; here, in the Dominican Republic, in the heart of the Caribbean, FreeBSD has a very good reputation when it comes to reliability and security. In fact, there is some collective thought within the sysadmin community that says something like: “If you want to deal only once with a service, install it over FreeBSD.”” Jonathan then goes through some of the steps taken to initial deploy NTP services, but with that out of the way, he has a great summary: > “Fascinated with the whole experience, we migrate one of our internal dns servers to a second FreeBSD machine and at the moment of this writing we are testing haproxy, an open source load-balancing proxy into a another server with the same OS. > After all this, no time issues have been reported in the past 2 years, so at least for my environment, FreeBSD came just in time.“ *** Beastie Bits MiniBSD laptop computer (https://hackaday.io/project/643-minibsd-laptop-computer) The state of LibreSSL in FreeBSD (https://attilagyorffy.com/2016/07/02/the-state-of-libressl-in-freebsd/) Justin Sherrill is looking for someone willing to run a Go builder with DragonflyBSD (https://www.dragonflydigest.com/2016/07/01/18372.html) Tiny Unix tools for Windows (https://tinyapps.org/blog/windows/201606040700_tiny_unix_tools_windows.html) OpenBSD's doas added to the FreeBSD Ports Tree (http://www.freshports.org/security/doas/) ubuntuBSD 16.04 to feature a combo of BusyBox and OpenRC, no systemd (http://linux.softpedia.com/blog/ubuntubsd-16-04-will-feature-a-combination-of-busybox-and-openrc-but-no-systemd-505463.shtml) Syncast Podcast 4 : Curl, libcurl and the future of the web, with Daniel Stenberg (http://podcast.sysca.st/podcast/4-curl-libcurl-future-web-daniel-stenberg/) Feedback/Questions Harri - Using beadm / zfssnap (http://pastebin.com/qKeCd63F) Jonathan - bhyve vs Proxmox (http://pastebin.com/EhXDwbWQ) Mohammad - Bhyve gfx passthrough (http://pastebin.com/ZCNk4Bga) Jeremy - Shapshots and more Snapshots (http://pastebin.com/xp7nzEYa) Ron - Microphone (http://pastebin.com/H2xr53CR) ***
149: The bhyve has been disturbed, and a wild Dexter appears!
Today on the show, we are going to be chatting with Michael Dexter about a variety of topics, but of course including bhyve! That plus This episode was brought to you by Headlines NetBSD Introduction (https://bsdmag.org/netbsd_intr/) We start off today’s episode with a great new NetBSD article! Siju Oommen George has written an article for BSDMag, which provides a great overview of NetBSD’s beginnings and what it is today. Of course you can’t start an article about NetBSD without mentioning where the name came from: “The four founders of the NetBSD project, Chris Demetriou, Theo de Raadt, Adam Glass, and Charles Hannum, felt that a more open development model would benefit the project: one centered on portable, clean and correct code. They aimed to produce a unified, multi-platform, production-quality, BSD-based operating system. The name “NetBSD” was suggested by de Raadt, based on the importance and growth of networks, such as the Internet at that time, the distributed and collaborative nature of its development.” From there NetBSD has expanded, and keeping in line with its motto “Of course it runs NetBSD” it has grown to over 57 hardware platforms, including “IA-32, Alpha, PowerPC,SPARC, Raspberry pi 2, SPARC64 and Zaurus” From there topics such as pkgsrc, SMP, embedded and of course virtualization are all covered, which gives the reader a good overview of what to expect in the modern NetBSD today. Lastly, in addition to mentioning some of the vendors using NetBSD in a variety of ways, including Point-Of-Sale systems, routers and thin-clients, you may not have known about the research teams which deploy NetBSD: NASA Lewis Research Center – Satellite Networks and Architectures Branch use NetBSD almost exclusively in their investigation of TCP for use in satellite networks. KAME project – A research group for implementing IPv6, IPsec and other recent TCP/IP related technologies into BSD UNIX kernels, under BSD license. NEC Europe Ltd. established the Network Laboratories in Heidelberg, Germany in 1997, as NEC’s third research facility in Europe. The Heidelberg labs focus on software-oriented research and development for the next generation Internet. SAMS-II Project – Space Acceleration Measurement System II. NASA will be measuring the microgravity environment on the International Space Station using a distributed system, consisting of NetBSD.“ My condolences, you’re now the maintainer of a popular open source project (https://runcommand.io/2016/06/26/my-condolences-youre-now-the-maintainer-of-a-popular-open-source-project/) A presentation from a Wordpress conference, about what it is like to be the maintainer of a popular open source project The presentation covers the basics: Open Source is more than just the license, it is about community and involvement The difference between Maintainers and Contributors It covers some of the reasons people do not open up their code, and other common problems people run into: “I'm embarrassed by my code” (Hint: so is everyone else, post it anyway, it is the best way to learn) “I'm discouraged that I can't finish releases on time” “I'm overwhelmed by the PR backlog” “I'm frustrated when issues turn into flamewars” “I'm overcommitted on my open source involvement” “I feel all alone” Each of those points is met with advice and possible solutions So, there you have it. Open up your code, or join an existing project and help maintain it *** FreeBSD Committer Allan Jude Discusses the Advantages of FreeBSD and His Role in Keeping Millions of Servers Running (http://www.hostingadvice.com/blog/freebsd-project-under-the-hood/) An interesting twist on our normal news-stories today, we have an article featuring our very own Allan Jude, talking about why FreeBSD and the advantages of working on an open-source project. “When Allan started his own company hosting websites for video streaming, FreeBSD was the only operating system he had previously used with other hosts. Based on his experience and comfort with it, he trusted the system with the future of his budding business.A decade later, the former-SysAdmin went to a conference focused on the open-source operating system, where he ran into some of the folks on its documentation team. “They inspired me,” he told our team in a recent chat. He began writing documentation but soon wanted to contribute improvements beyond the docs.Today, Allan sits as a FreeBSD Project Committer. It’s rare that you get to chat with someone involved with a massive-scale open-source project like this — rare and awesome.” From there Allan goes into some of the reasons “Why” FreeBSD, starting with Code Organization being well-maintained and documented: “The FreeBSD Project functions like an extremely well-organized world all its own. Allan explained the environment: “There’s a documentation page that explains how the file system’s laid out and everything has a place and it always goes in that place.”” + In addition, Allan gives us some insight into his work to bring Boot-Environments to the loader, and other reasons why FreeBSD “just makes sense” + In summary Allan wraps it up quite nicely: “An important take-away is that you don’t have to be a major developer with tons of experience to make a difference in the project,” Allan said — and the difference that devs like Allan are making is incredible. If you too want to submit the commit that contributes to the project relied on by millions of web servers, there are plenty of ways to get involved! We’re especially talking to SysAdmins here, as Allan noted that they are the main users of FreeBSD. “Having more SysAdmins involved in the actual build of the system means we can offer the tools they’re looking for — designed the way a SysAdmin would want them designed, not necessarily the way a developer would think makes the most sense” A guide to saving electricity and time with poudriere and bhyve (http://justinholcomb.me/blog/2016/07/03/poudriere-in-bhyve-and-bare-metal.html) “This article goes over running poudriere to built packages for a Raspberry Pi with the interesting twist of running it both as a bhyve guest and then switching to running on bare metal via Fiber Channel via ctld by sharing the same ZFS volume.” “Firstly, poudriere can build packages for different architectures such as ARM. This can save hours of build time compared to building ports from said ARM device.” “Secondly, let’s say a person has an always-on device (NAS) running FreeBSD. To save power, this device has a CPU with a low clock-rate and low core count. This low clock-rate and core count is great for saving power but terrible for processor intensive application such as poudriere. Let’s say a person also has another physical server with fast processors and a high CPU count but draws nearly twice the power and a fan noise to match.” “To get the best of both worlds, the goal is to build the packages on the fast physical server, power it down, and then start the same ZFS volume in a bhyve environment to serve packages from the always-on device.” The tutorial walks through setting up ‘ahost’, the always on machine, ‘fhost’ the fast but noisy build machine, and a raspberry pi It also includes creating a zvol, configuring iSCSI over fibre channel and exporting the zvol, booting an iSCSI volume in bhyve, plus installing and setting up poudriere This it configures booting over fibre channel, and cross-building armv6 (raspberry pi) packages on the fast build machine Then the fast machine is shut down, and the zvol is booted in bhyve on the NAS Everything you need to know to make a hybrid physical/virtual machine The same setup could also work to run the same bhyve VM from either ahost or fhost bhyve does not yet support live migration, but when it does, having common network storage like the zvol will be an important part of that *** Interview - Michael Dexter - editor@callfortesting.org (mailto:editor@callfortesting.org) / @michaeldexter (https://twitter.com/michaeldexter) The RoloDexter *** iXSystems Children's Minnesota Star Studio Chooses iXsystems' TrueNAS Storage (https://www.youtube.com/watch?v=FFbdQ_05e-0) *** News Roundup FreeBSD Foundation June 2016 Update (https://www.freebsdfoundation.org/wp-content/uploads/2016/06/FreeBSD-Foundation-June-2016-Update.pdf) The FreeBSD Foundation’s June newsletter is out Make sure you submit the FreeBSD Community Survey (https://www.surveymonkey.com/r/freebsd2016) by July 7th: In addition to the opening message from the executive director of the foundation, the update includes details to sponsored work on the FreeBSD VM system, reports from a number of conferences the Foundation attended, including BSDCan The results of the foundation's yearly board meeting People the foundation recognized for their contributions to FreeBSD at BSDCan And an introduction to their new “Getting Started with FreeBSD” project *** [How-To] Building the FreeBSD OS from scratch (http://www.all-nettools.com/forum/showthread.php?34422-Building-the-FreeBSD-OS-from-scratch) A tutorial over at the All-NetTools.com forums that walks through building FreeBSD from scratch I am not sure why anyone would want to build Xorg from source, but you can It covers everything in quite a bit of detail, from the installation process through adding Xorg and a window manager from source It also includes tweaking some device node permissions for easier operation as a non-root user, and configuring the firewall *** Window Systems Should Be Transparent (http://doc.cat-v.org/bell_labs/transparent_wsys/) + Rob Pike of AT&T Labs writes about why Window Systems should be transparent This is an old paper (undated, but I think from the late 80s), but may contain some timeless insights “UNIX window systems are unsatisfactory. Because they are cumbersome and complicated, they are unsuitable companions for an operating system that is appreciated for its technical elegance” “A good interface should clarify the view, not obscure it” “Mux is one window system that is popular and therefore worth studying as an example of good design. (It is not commercially important because it runs only on obsolete hardware.) This paper uses mux as a case study to illustrate some principles that can help keep a user interface simple, comfortable, and unobtrusive. When designing their products, the purveyors of commercial window systems should keep these principles in mind.” There are not many commercial window systems anymore, but “open source” was not really a big thing when this paper was written *** Roger Faulkner, of Solaris fame passed away (http://permalink.gmane.org/gmane.comp.standards.posix.austin.general/12877) “RIP Roger Faulkner: creator of the One and True /proc, slayer of the M-to-N threading model -- and the godfather of post-AT&T Unix” @bcantrill: Another great Roger Faulkner story (https://twitter.com/bcantrill/status/750442169807171584) The story of how pgrep -w saved a monitor -- if not a life (https://news.ycombinator.com/item?id=4306515) @bcantrill: With Roger Faulkner, Tim led an engineering coup inside Sun that saved Solaris circa 2.5 (https://twitter.com/bcantrill/status/750442169807171584) *** Beastie Bits: Developer Ed Maste is requesting information from those who are users of libvgl. (https://lists.freebsd.org/pipermail/freebsd-stable/2016-June/084843.html) HEADS UP: DragonFly 4.5 world reneeds rebuilding (http://lists.dragonflybsd.org/pipermail/users/2016-June/249748.html) Chris Buechler is leaving the pfSense project, the entire community thanks you for your many years of service (https://blog.pfsense.org/?p=2095) GhostBSD 10.3-BETA1 now available (http://ghostbsd.org/10.3_BETA1) DragonFlyBSD adds nvmectl (http://lists.dragonflybsd.org/pipermail/commits/2016-June/500671.html) OPNsense 16.1.18 released (https://opnsense.org/opnsense-16-1-18-released/) bhyve_graphics hit CURRENT (https://svnweb.freebsd.org/base?view=revision&revision=302332) BUG Update FreeBSD Central Twitter account looking for a new owner (https://twitter.com/freebsdcentral/status/750053703420350465) NYCBUG meeting : Meet the Smallest BSDs: RetroBSD and LiteBSD, Brian Callahan (http://lists.nycbug.org/pipermail/talk/2016-July/016732.html) NYCBUG install fest @ HOPE (http://lists.nycbug.org/pipermail/talk/2016-June/016694.html) SemiBUG is looking for presentations for September and beyond (http://lists.nycbug.org/pipermail/semibug/2016-June/000107.html) Caleb Cooper is giving a talk on Crytpo at KnoxBUG on July 26th (http://knoxbug.org/content/2016-07-26) Feedback/Questions Leif - ZFS xfer (http://pastebin.com/vvASr64P) Zach - Python3 (http://pastebin.com/SznQHq7n) Dave - Versioning (http://pastebin.com/qkpjKEr0) David - Encrypted Disk Images (http://pastebin.com/yr7BUmv2) Eli - TLF in all the wrong places (http://pastebin.com/xby81NvC) ***
148: The place to B...A Robot!
This week on the show, Allan and I are going to be showing you a very interesting interview we did talking about using FreeBSD to drive This episode was brought to you by Headlines FreeBSD Core Team Election (https://www.freebsd.org/administration.html#t-core) Core.9 has been elected, and will officially take over from Core.8 on Wednesday, 6 July 2016 Many thanks to the outgoing members of the core team for their service over the last 2 years 214 out of 325 eligible voters (65.8%) cast their votes in an election counting 14 candidates. The top nine candidates are, in descending order of votes received: 180 84.1% Ed Maste (incumbent) 176 82.2% George V. Neville-Neil (incumbent) 171 79.9% Baptiste Daroussin (incumbent) 168 78.5% John Baldwin 166 77.6% Hiroki Sato (incumbent) 147 68.7% Allan Jude 132 61.7% Kris Moore 121 56.5% Benedict Reuschling 108 50.5% Benno Rice There was no tie for ninth. BSDNow and the entire community would also like to extend their thanks to all those who stood for election to the core team Next week’s core meeting will encompass the members of Core.8 and Core.9, as responsibility for any outstanding items will be passed from outgoing members of core to the new incoming members *** Why I run OpenBSD (http://deftly.net/posts/2016-05-31-why-i-run-openbsd.html) This week we have a good article / blog post talking about why the posted has moved to OpenBSD from Linux. “One thing I learned during my travels between OSs: consistency is everything. Most operating systems seem to, at least, keep a consistent interface between themselves and binaries / applications. They do this by keeping consistent APIs (Application Programming Interfaces) and ABIs (Application Binary Interfaces). If you take a binary from a really old version of Linux and run or build it on a brand-spanking new install of Linux, it will likely Just Work™. This is great for applications and developers of applications. Vendors can build binaries for distribution and worry less about their product working when it gets out in the wild (sure this binary built in 2016 will run on RedHat AS2.1!!).“ The author then goes through another important part of the consistency argument, with what he calls “UPI” or “User Program Interfaces”. In other words, while the ABI may be stable, what about the end-user tooling that the user directly has to interact with on a daily basis? “This inconsistency seems to have come to be when Linux started getting wireless support. For some reason someone (vendors, maybe?) decided that ifconfig wasn’t a good place to let users interact with their wireless device. Maybe they felt their device was special? Maybe there were technical reasons? The bottom line is, someone decided to create a new utility to manage a wireless device… and then another one came along… pretty soon there was iwconfig(8), iw(8), ifconfig(8), some funky thing that let windows drivers interface with Linux.. and one called ip(8) I am sure there are others I am forgetting, but I prefer to forget. I have moved onto greener pastures and the knowledge of these programs no longer serves me.” The article then goes through the rundown of how he evaluated the various BSD’s and ultimately settled on OpenBSD: “OpenBSD won the showdown. It was the most complete, simple, and coherent system. The documentation was thorough, the code was easy to follow and understand. It had one command to configure all of the network interfaces! I didn’t have wireless, but I was able to find a cheap USB adapter that worked by simply running man -k wireless and reading about the USB entries. It didn’t have some of the applications I use regularly, so I started reading about ports (intuitively, via man ports!).” The ultimate NetBSD Router (http://blog.tbrodel.me/2016/#netbsd-router) “So yesterday I spent the day setting up a new firewall at home here, based off of this BSD Now tutorial. Having set up a couple of OpenBSD routers before, either based on old laptops, bulky old power-sucking desktops or completely over-specced machines like the Intel NUC, I wanted to get some kind of BSD onto a low-powered ARM board and use that instead.” “I've had a couple of Cubietrucks lying around for a while now, I've used them in a couple of art installations, running Debian and Pure Data, but over all they've been a bit disappointing. It's more the manufacturer's fault but they require blobs for the graphics and audio, which Debian won't allow, so as a multimedia board they're dud for video, and only passable for audio work with a usb sound card. So they've been collecting dust.” “Only thing missing is a second NIC, luckily I had an Apple USB->Ethernet dongle lying around, which when I bought it was the cheapest thing I could find on eBay that OpenBSD definitely supported. There, and on NetBSD, it's supported by the axe(4) driver. USB 2.0 works fine for me as I live in Australia and my ISP can only give me 30Mbps, so this should do for the forseeable future.” + The article then walks through installing and configuring NetBSD + Configuration includes: pf, unbound, and dhcpd “This project has been really fun, I started with basically no experience with NetBSD and have finished with a really useful, low-powered and robust appliance. It's a testament to the simplicity of the NetBSD system, and the BSD design principles in general, that such a novice as myself could figure this out. The NetBSD project has easily the most polished experience on Allwinner ARM boards, even Debian doesn't make it this easy. It's been a joy running the system, it has the bits I love from OpenBSD; ksh(1), tmux(1), an http daemon in base and of course, pf(4). This is mixed with some of the pragmatism I see in FreeBSD; a willingness to accept blobs if that really is the only way to boot, or get audio, or a video console.” bhyve-Bootable Boot Environments (http://callfortesting.org/bhyve-boot-environments/) We have a lengthy article also today from our friend Michael Dexter, who asks the basic question “What if multibooting and OS upgrades weren't horrible?” No doubt if you’ve been a frequent listener to this show, you’ve heard Allan or Myself talking about ZFS Boot Environments, and how they can “change your life”. Well today Michael goes further into detail on how the BE’s work, and how they can be leveraged to do neat things, like installing other versions of an operating system from the original running system. “If you are reading this, you have probably used a personal computer with a BSD or GNU/Linux operating system and at some point attempted to multiboot between multiple operating systems on the same computer. This goal is typically attempted with complex disk partitioning and a BSD or GNU/Linux boot loader like LILO or GRUB, plus several hours of frustrating experimentation and perhaps data loss. While exotic OS experimentation has driven my virtualization work since the late 1990s, there are very pragmatic reasons for multibooting the same OS on the same hardware, notable for updates and failback to "known good" versions. To its credit, FreeBSD has long had various strategies including the NanoBSD embedded system framework with primary and secondary root partitions, plus the nextboot(8) utility for selecting the "next" kernel with various boot parameters. Get everything set correctly and you can multiboot "with impunity". “That's a good start, and over time we have seen ZFS "boot environments" be used by PC-BSD and FreeNAS to allow for system updates that allow one to fall back to previous versions should something go wrong. Hats off to these efforts but they exist in essentially purpose-built appliance environments. I have long sensed that there is more fun to be had here and a wonderful thing happened with FreeBSD 10.3 and 11.0: Allan Jude added a boot environment menu to the FreeBSD loader” From here Michael takes us through the mechanical bits of actually creating a new ZFS dataset (BE) and performing a fresh FreeBSD 10.3 installation into this new boot-environment. The twist comes at the end, where he next sets up the BE to be a root NFS for booting in bhyve! This is interesting and gives you a way to test booting into your new environment via a VM, before rebooting the host directly into it. *** Interview - Edicarla Andrade & Vinícius Zavam - @egypcio (https://twitter.com/egypcio) BSD-Powered Robots News Roundup Tomohiro Kasumi explains what “@@” means, in the context of the Hammer filesystem (http://lists.dragonflybsd.org/pipermail/users/2016-June/249717.html) A post from the Dragonfly users’ mailing list about what the @@ construct means in the Hammer filesystem “@@ represents the existence of a PFS which is logically separated pseudo filesystem space within HAMMER's B-Tree” “HAMMER only has 1 large B-Tree per filesystem (not per PFS), so all the PFS exist within that single B-Tree. PFS are separated by localization parameter which is one of the B-Tree keys used to lookup the tree.” Each substring in "@@-1:00001" means: "@@" means it's a PFS or snapshot. "-1" means it's a master. ":" is just a separator. "00001" means it's PFS#1, where PFS#0 is the default PFS created on newfs. There is no "00000" because that's what's mounted on /HAMMER. PFS# is used for localization parameter. “Localization parameter has the highest priority when inserting or looking up B-Tree elements, so fs elements that belong to the same PFS# tend to be localized (clustered) within the B-Tree” There is also a note about how snapshots are named: "@@0x00..." A user points out that having : in the path can confuse some applications, such as in the case of adding the current directory or a relative path to the $PATH environment variable, which is a colon delimited list of paths This seems quite a bit more confusing that the datasets created by ZFS, but they might have other useful properties *** FreeBSD 11.0 nearing RC1 (https://www.freebsd.org/releases/11.0R/schedule.html) We’ve all been eagerly awaiting the pending release of FreeBSD 11.0, and the schedule has now been updated! The first release candidate is slated for July 29th! If all goes well (and we stick to schedule) there will be another RC2 and possible RC3 release, before 11.0 officially drops near the end of August. Start playing with those builds folks, be sure to send your feedback to the team to make this the best .0 release ever! *** TensorFlow on FreeBSD (http://ecc-comp.blogspot.com/2016/06/tensorflow-on-freebsd.html) Next we have a blog post about the experience of a “new” FreeBSD user trying to deploy some non-ported software to his new system. Specifically he was interested in running TensorFlow, but not doing a port himself, because in his words: “First, I apologize for not supplying a port archive myself. After reading the FreeBSD handbook for creating a port, it's too complex of a task for me right now. I've only been using FreeBSD for two weeks. I would also not like to waste anyone's time giving them a terrible port archive and mess up their system.” First of all, good ports are often born out of bad ports! Don’t let the porting framework daunt you, give it a go, since that's the only way you are going to learn how to write “good” ports over time. The porters-handbook is a good first place to start, plus the community usually is very helpful in providing feedback. He then walks us through the changes made to the TensorFlow code (starting with the assumption that OSX was a good “flavor” to begin porting from) and ultimately compiling. This ends up with the creation of a pip package which works! A good tutorial, and also very similar to what goes on in the porting process. With this write-up perhaps somebody will take up creating a port of it… hint hint! *** NetBSD: A New Beginning? (http://jamesdeagle.blogspot.ca/2016/06/netbsd-new-beginning.html) We don’t get enough NetBSD news at times, but this post by James Deagle talks about his adventure with NetBSD 7.0 and making it his “new beginning” “After a few months of traipsing around the worlds of SunOS and Linux, I'm back to NetBSD for what I hope will be a lengthy return engagement. And while I'm enamored of NetBSD for all the previously-mentioned reasons, I'm already thinking ahead to some problems to solve, some of which have also been mentioned before.” He then goes through and lists some of the small nits he’s still running into during the daily workflow YouTube audio - Specifically he mentions that no audio is playing, but wonders if Flash plays some part. (Ideally you’re not using Flash though, in which case you need to check the audio backend FF is using. Try PulseAudio since it seems the best supported. If pulse is already enabled, install ‘pavucontrol’ to make sure audio is playing to the correct sound device) Slow gaming performance (TuxKart and Celestia) - Check DRI / Xorg? Or is it CPU bound? Lastly some unspecified Wireless issues, which typically end up being driver related. (Or use another chipset) Beastie Bits Reproducible NetBSD? 77.7% of the way there (https://reproducible.debian.net/netbsd/netbsd.html) Create FreeBSD virtual machine using qemu. Run the VM using xhyve. (https://gist.github.com/zg/38a3afa112ddf7de4912aafc249ec82f) FreeBSD PowerPC 32bit pkg repository (unofficial). ~19,500 packages, more to come (https://joshcummings.net/pub/FreeBSD) NetBSD machines at Open Source Conference 2016 Gunma (http://mail-index.netbsd.org/netbsd-advocacy/2016/05/16/msg000706.html) Adam Leventhal (of ZFS and DTrace) does an analysis of APFS (http://arstechnica.com/apple/2016/06/a-zfs-developers-analysis-of-the-good-and-bad-in-apples-new-apfs-file-system/) SemiBug June meeting summary (http://lists.nycbug.org/pipermail/semibug/2016-June/000106.html) KnoxBug Meeting (http://knoxbug.org/content/2016-07-26) Feedback/Questions Andrew - iocage (http://pastebin.com/nuYTzaG6) Florian - Arm + GitHub (http://pastebin.com/PzY68hNS) Clint - Synth (http://pastebin.com/JESGZjLu) Leonardo - Translations (http://pastebin.com/b4LAiPs4) Zachary - Moving things to VMs (http://pastebin.com/VRc8fvBk) ***
147: Release all the things!
On this episode of BSDNow, we will be talking to Glen Barber and Peter Wemm of the FreeBSD RE and Cluster Admin teams! That plus our This episode was brought to you by Headlines 2016 FreeBSD Community Survey (https://www.surveymonkey.com/r/freebsd2016) We often get comments from our listeners, “I’m not a developer, how can I help out”? Well today is your chance to do something. The FreeBSD Foundation has its 2016 Community Survey online, where they are asking for feedback from you! I just did the survey, it’ll take you about 5 minutes, but gives you a chance to provide valuable feedback to the foundation about things that are important to you. Be sure to answer in as much detail as possible and the foundation will review and use this feedback for its operations going forward. *** ART, OpenBSDs new routing table, single thread performances (http://www.grenadille.net/post/2016/06/17/ART-single-thread-performances) OpenBSD has changed the way routes are looked up in the kernel as part of their path to an SMP networking stack The “Allotment Routing Table” (ART) is a performance tradeoff, where more memory is used to store the routing table, in exchange for faster lookups With this new arrangement, a full BGP routing table will grow from 130MB to 180MB of memory “ART is a free multibit trie based routing table. To keep it simple, it can be seen as using more memory for fewer CPU cycles. In other words, we get a faster lookup by wasting memory. The original paper (http://www.hariguchi.org/art/art.pdf) presents some performance comparisons between two ART configurations and the BSD Radix. But how does this apply to OpenBSD?” “I asked Hrvoje Popovski to run his packet forwarding test on his Xeon box (E5-2620 v2 @ 2.10GHz, 2400.34 MHz) with ix(4) (82599) interfaces. The test setup consist of three machines with the OpenBSD box in the middle” “The simulations have been performed with an OpenBSD -current from June 9th. The machine is configured with pf(4) disabled in order to force a single route lookup for every IPv4 packet. Based on the result of the lookup the kernel decide if it should forward, deliver or drop the packet” *** BSDCan 2016 Playlist (https://www.youtube.com/playlist?list=PLeF8ZihVdpFfoEV67dBSrKfA8ifpUr6qC) The complete set of videos from BSDCan is online and ready to be consumed Remember the good-ole days where we would wait months (or years) to get videos posted from conferences? Well, who are we kidding, some conferences STILL do that, but we can’t count BSDCan among them. Only two weeks out from this years exciting BSDCan, and all the videos have now landed on YouTube. Granted, this is no substitute for actually being at the conference, but even if you attended you probably missed quite a few of the talks. There are no videos of the hallway track, which is the best part of the conference Except the dinner discussion of course. and don’t forget the hacker lounge *** Should you be scared of Unix signals? (http://jvns.ca/blog/2016/06/13/should-you-be-scared-of-signals/) Do you know much about UNIX Signals? Are you afraid of their complexity? Do you know there are signals other than SIGKILL? This article talks about the practical implications of signals from a programming perspective The things you need to consider when dealing with signals Basically, you register a “signal handler”, the function that will be run when a signal arrives As you program is running, if a signal arrives, your program will be interrupted. Its current state will be saved and any system calls in progress will return EINTR (Error, Interrupted), then your signal handler will be run. Once the signal handler is complete, the state of your application will be restored, and execution will resume As long as your program properly handles this interruption, and errors that might result from it (getting EINTR from a read() call, instead of the data you expected), then everything should be fine. Of course, you need to be careful what you do inside your signal handler, as if you modify any variables or state in your application, it might be very confused when it resumes. *** Interview - Glen and Peter- News Roundup Unik - The Unikernel Compilation and Deployment Platform (uses NetBSD's Rump) (https://github.com/emc-advanced-dev/unik) We’ve talked a bit about NetBSD’s RUMP (unikernel) in the past, including articles on how to deploy services using it. Now we have an interesting project which makes the process super-easy, and dare-we-say almost “Docker-Like?” The Unik project has a fairly complete walkthrough right on their GitHub project page, including details on installation and creating your own unikernel containers. In addition, it provides instructions on boot-strapping your own Go/Node.js/Python/Java applications, and supports out of Box VCenter / AWS / Qemu / VirtualBox providers. *** PkgSrc 50th Release Highlights () pkgsrc is celebrating its 50th release, and to highlight this, they have posted a series of interviews from people who have been active in the project pkgsrc 50th release interviews - Jonathan Perkin (http://blog.netbsd.org/tnf/entry/pkgsrc_50th_release_interviews_jonathan) pkgsrc 50th release interviews - Ryo ONODERA (http://blog.netbsd.org/tnf/entry/pkgsrc_50th_release_interviews_ryo) pkgsrc 50th release interviews - Joerg Sonnenberg (http://blog.netbsd.org/tnf/entry/pkgsrc_50th_release_interview_with) pkgsrc 50th release interviews - Sevan Janiyan (https://blog.netbsd.org/tnf/entry/pkgsrc_50th_release_interviews_sevan) *** Migrating to FreeBSD from Solaris 11 (http://justinholcomb.me/blog/2016/02/28/migration-to-freebsd-part1.html) Part 2 (http://justinholcomb.me/blog/2016/03/12/migration-to-freebsd-part2.html) Part 3 (http://justinholcomb.me/blog/2016/03/19/migration-to-freebsd-part3.html) Part 4 (http://justinholcomb.me/blog/2016/03/26/migration-to-freebsd-part4.html) Part 5 (http://justinholcomb.me/blog/2016/04/03/migration-to-freebsd-part5.html) *** How to chroot www/firefox on NetBSD (https://github.com/alnsn/localpkgsrc/tree/master/firefox-chroot) Looking for a jail-like method of running FireFox on NetBSD? (Or possibly other BSDs?) We have a github repo with details on how to setup and run FireFox using a chroot using a “webuser” account for safety. Think of this as a jail alternative, may be useful on systems with no jail support. Of interest is the method used to do X forwarding. It uses Xorg TCP listen option (which is often off by default for security reasons). Perhaps SSH X forwarding would be a better alternative. (Or nullfs mounts of /tmp) *** Beastie Bits Tredly - V1 Release Candidate (https://github.com/tredly/tredly/releases/tag/v1.0.0-rc.1) Call for Testing - ypldap testing against OpenLDAP and Microsoft Active Directory (http://lists.freebsd.org/pipermail/freebsd-current/2016-June/061775.html) BSD Magazine, June 2016 Out Now (https://bsdmag.org/) Hammer2 - Add xxhash to H2 and throw in debug stuff for performance testing (http://lists.dragonflybsd.org/pipermail/commits/2016-June/500610.html) chyves pre-announcement (http://justinholcomb.me/blog/2016/06/14/chyves-project-preannouncement.html) *** Feedback/Questions Michael - Versioning (http://pastebin.com/1hpGrmuL) Michael - Removing Encryption (http://pastebin.com/2PkrMGGx) Bostjan - PC-BSD Questions (http://pastebin.com/q5VdmNxG) Fong - ZFS Rollback (http://pastebin.com/2aedLV7d) Jochen - Docker on FBSD (http://pastebin.com/dneVZkXc) ***
146: Music to Beastie’s ears
Kris is on vacation this week, so allan flies solo, provides a recap of BSDCan & cover's a boatload of news including Microsoft This episode was brought to you by Headlines BSDCan Recap and Live Stream Videos (http://www.bsdcan.org/2016/) OpenBSD BSDCan 2016 papers now available (http://www.openbsd.org/papers) Allan’s slides (http://allanjude.com/bsd/BSDCan2016_-_GELIBoot.pdf) and Paper (http://allanjude.com/bsd/AsiaBSDCon2016_geliboot_pdf1a.pdf) Michael W Lucas presents Allan with a gift (https://www.youtube.com/watch?v=LFgxAHkrSTg) “FreeBSD Mastery: Advanced ZedFS” (http://blather.michaelwlucas.com/archives/2698) Highlighted Tweets: Groff Arrives at BSDCan (https://twitter.com/Keltounet/status/740344735194320896) FreeBSD Foundation recognizes the contributions of Bryan Drewery, Rod Grimes, Warren Block, & Gleb Smirnoff (https://twitter.com/freebsdfndation/status/742456950676393984) A moment of silence and shots in memory in Benjamin Perrault @creepingfur (https://twitter.com/__briancallahan/status/741854476340858880) @gvnn3 sells the FreeBSD Foundation shirt off of his back for Charity (https://twitter.com/Keltounet/status/741763867471155201) Michael W. Lucas asks Matt Ahrens how to pronounce ZFS, “You can pronounce ZFS however you like, but if you pronounce it 'reiserfs', people might be confused.” (https://twitter.com/cperciva/status/741375414967410688) Sysadmin T-Shirt (https://twitter.com/BSDCan/status/741420633007874050) FreeBSD Dev Summit ran out of room on the chalkboards listing accomplishments of 11.0 (https://twitter.com/SeanChittenden/status/740904105388978176) List of things people have or want for FreeBSD 12 (https://twitter.com/Keltounet/status/740928627471159296) Matt Ahrens signing Allan’s ZFS book (https://twitter.com/kprovst/status/741322268480049152?cn=bWVudGlvbg%3D%3D&refsrc=email) FreeBSD’s new marketing strategy (https://twitter.com/cperciva/status/741707948469157889) Charity Auction: systemd whoopie cushion (https://twitter.com/HippyWizard/status/741768670704066560) Embarass OpenBSD’s @HenningBrauer by donating $10 to charity for a selfie with him wearing a Linux t-shirt (https://twitter.com/juliefriday/status/741948048788586496) @GroffTheBSDGoat changes handlers, from @HenningBrauer to @GavinAtkinson (https://twitter.com/GroffTheBSDGoat/status/742415390798716928) Day 1 Video (https://www.youtube.com/watch?v=AOidjSS7Hsg) Day 2 Video (https://www.youtube.com/watch?v=z7pDnBO5wSM) Allan’s GELIBoot talk (day 2) (https://www.youtube.com/watch?v=z7pDnBO5wSM&feature=youtu.be&list=PLeF8ZihVdpFfoEV67dBSrKfA8ifpUr6qC&t=4440) *** Media Coverage of Microsoft + FreeBSD story (https://azure.microsoft.com/en-us/blog/freebsd-now-available-in-azure-marketplace/) Microsoft has released their own custom image of FreeBSD 10.3 for the Azure Cloud “This means that not only can you quickly bring-up a FreeBSD VM in Azure, but also that in the event you need technical support, Microsoft support engineers can assist.” “Microsoft is the publisher of the FreeBSD image in the marketplace rather than the FreeBSD Foundation. The FreeBSD Foundation is supported by donations from the FreeBSD community, including companies that build their solutions on FreeBSD. They are not a solution provider or an ISV with a support organization but rather rely on a very active community that support one another. In order to ensure our customers have an enterprise SLA for their FreeBSD VMs running in Azure, we took on the work of building, testing, releasing and maintaining the image in order to remove that burden from the Foundation. We will continue to partner closely with the Foundation as we make further investments in FreeBSD on Hyper-V and in Azure.” "It's quite a significant milestone for FreeBSD community and for Microsoft to publish a supported FreeBSD image on Azure Marketplace. We really appreciate Microsoft's commitment and investment in FreeBSD project". - Justin T. Gibbs, President of FreeBSD Foundation Microsoft took a FreeBSD 10.3-RELEASE image and added additional patches, most of which they have upstreamed but that were too late for the regular 10.3 release cycle. Rather than requiring users to use a snapshot of the stable/10 branch, which would complicate the user experience, and complicate the job of the Microsoft support engineers, they created their own “certified” release This allows Microsoft to selectively deploy errata fixes to the image as well It is not clear how this affects update mechanisms like freebsd-update(8) The Register (http://www.theregister.co.uk/2016/06/09/microsoft_freebsd/) The Inquirer (http://www.theinquirer.net/inquirer/news/2461070/microsoft-creates-own-distribution-of-freebsd-for-azure-developers) Infoworld (http://www.infoworld.com/article/3082090/open-source-tools/is-microsoft-publishing-its-own-freebsd-yes-and-no.html) The Hacker News (http://thehackernews.com/2016/06/microsoft-azure-freebsd.html) Windows Report (http://windowsreport.com/microsoft-freebsd-10-3-ready-made-vm-image-azure/) Windows Club (http://news.thewindowsclub.com/microsoft-freebsd-operating-system-84375/) *** Select works poorly (http://www.tedunangst.com/flak/post/select-works-poorly) “At the bottom of the OpenBSD man page for select is a little note. “Internally to the kernel, select() and pselect() work poorly if multiple processes wait on the same file descriptor.” There’s a similar warning in the poll man page. Where does this warning come from and what does it mean?” Ted found that at first glance, OpenBSD’s select() appears to be quite bad: “whenever some data gets written, we call wakeup(&selwait);. Based on what we’ve seen so far, one can conclude that this is likely to be inefficient. Every time any socket has some data available, we wake up every selecting process in the system. Works poorly indeed.” After further investigation, it turns out to not be quite as bad When the select() is first setup, the PID of the process that cares about the FD is recorded in the selinfo struct If a second process runs select() on the same FD, the SI_COLL (Select Collision) flag is set on the selinfo struct When selwakeup() is called, if SI_COLL is set, all select()ing processes are woken up, and the sysctl kern.nselcoll is incremented. If the flag is not set, and only a single PID is waiting for activity on that FD, only that process is woken up “This is not an intractable problem. kevent avoids it entirely. Other implementations may too. But practically, does it need to be solved? My laptop says it’s happened 43 times. A server with substantially more uptime says 0. Doesn’t seem so bad.” *** Interview - Hans Petter Selasky - hps@freebsd.org (mailto:hps@freebsd.org) / @twitter (https://twitter.com/user) Designing FreeBSD’s USB drivers, hooking up a piano to FreeBSD & more! *** News Roundup Timeline of libexpat random vulnerability (http://www.tedunangst.com/flak/post/timeline-of-libexpat-random-vulnerability) Do you use FreeBSD as web server? Why or why not? (https://news.ycombinator.com/item?id=11804565) 20 years of NetBSD code Bloat (http://kristerw.blogspot.sg/2016/05/20-years-of-netbsd-code-bloat.html) HP Chromebook 13 now booting OpenBSD (https://jcs.org/statuses/2016/06/08/740606952149942272/) UNIX for Poets (https://web.stanford.edu/class/cs124/lec/124-UnixForPoets.pdf) Comparing live version upgrade methods (https://distrowatch.com/weekly.php?issue=20160530#upgrades) My life with FreeBSD on a Thinkpad X220 (https://www.reddit.com/r/BSD/comments/4n3flx/my_life_with_freebsd_on_a_thinkpad_x220/)
145: At the Core of it all
It’s BSDCan time! Allan and I are both enjoying what is sure to be a super-busy week, but don’t think we’ve forgotten about This episode was brought to you by Interview - Benno Rice - benno@freebsd.org (mailto:benno@freebsd.org) / @jeamland (https://twitter.com/jeamland) Manager, OS & Networking at EMC Isilon Emily Dunham: Community Automation (https://www.youtube.com/watch?v=dIageYT0Vgg) iXsystems 1U Rackmount Server - 4 Bay Hot-Swap SAS/SATA Drive Bays 400W Redundant Power Supply - Single Socket Embedded CPU (48 cores) - 8 DIMM Slots with 16GB DIMMs for a total of 128GB RAM – Dual Gigabit LAN, Dual 10GbE SFP+ and 1 x 40Gb QSFP+ port, (1) PCI-E Expansion Slots + IPMI Dedicated LAN - Cavium ThunderX ARM CN8890 48 Core ThunderX CPU - 2.5GHz per core System has 128GB RAM, 4 x 2TB SATA HDD, Additional Intel i350 (2 x 1GbE) Beastie Bits file considered harmful (http://www.tedunangst.com/flak/post/file-considered-harmful) An open source talk on ZFS. “Intro to ZFS” as a set of open source slides for the community to build on, and to reuse. Go give this talk at your local conference. (https://github.com/problame/talkintrozfs2016) ARMv7 now has a bootloader (http://undeadly.org/cgi?action=article&sid=20160529145411) SHA256/512 speed improvements in FreeBSD 11 (https://svnweb.freebsd.org/base?view=revision&revision=300966) pkgsrc 50th release interviews - Joerg Sonnenberg (http://blog.netbsd.org/tnf/entry/pkgsrc_50th_release_interview_with) DFly versus PC-BSD on a Laptop (http://lists.dragonflybsd.org/pipermail/users/2016-May/249636.html) FreeBSD ifconfig can print subnet masks in CIDR or dotted-quad, finally (https://svnweb.freebsd.org/base?view=revision&revision=301059) Feedback/Questions Eli - Getting rid of ports? (http://pastebin.com/4Y6VYSyN) Morgan - Best way to admin jails? (http://pastebin.com/w8hsMtbc) Simon - Use existing pkgs in poudriere (http://pastebin.com/mqSJk0pP) Pete - Lots of Q’s (http://pastebin.com/1M7HLAXs) Van - Made the switch (http://pastebin.com/NTVBvtC5) ***
144: The PF life
It’s only one-week away from BSDCan, both Allan and I are excited to meet some of you in person! However, the show keeps on This episode was brought to you by Headlines dotSecurity 2016 - Theo de Raadt - Privilege Separation and Pledge (http://www.dotsecurity.io/) Video (https://www.youtube.com/watch?v=a_EYdzGyNWs) Slides (https://www.openbsd.org/papers/dot2016.pdf) Interested in Privilege Separation and security in general? If so, then you are in for a treat, we have both the video and slides from Theo de Raadt at dotSecurity 2016. Specifically the the talk starts off looking at Pledge (no copyright issues with the pictures I hope??) and how their NTP daemon uses it. After going through some internals, Theo reveals that around 10% of programs “pledged” so far were found to be trying to do actions outside of their security scope. On the future-work side, they mention going back and looking at OpenSSH privilege separation next, as well as working with other OS’s that may want pledge support. *** bhyve now supports UEFI GOP (https://lists.freebsd.org/pipermail/freebsd-virtualization/2016-May/004471.html) The log awaited UEFI GOP (Graphics Output Protocol (https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface#GOP)) features has landed in bhyve This provides emulated graphics via an internal VNC server, allowing users to have full graphical access to the guest OS This allows installation of Windows guests without needing to create a modified ISO with an unattended installation script The code has not actually landed in FreeBSD head yet, but has been committed to a project branch Following a few simple commands, you can compile the new bhyve binary on your -CURRENT system and get started right away This feature is expected to be included in the upcoming FreeBSD 11.0 This commit drop also brings with it: XHCI -- an emulated usb tablet device that provides exact mouse positioning in supported OSs PS2 mouse for fallback if the guest does not support XHCI (Windows 7) PS2 keyboard “The code has been tested with Windows 7/8/8.1/10 and Server 2k12/2k16, Ubuntu 15.10, and FreeBSD 10.3/11-CURRENT” “For VNC clients, TightVNC, TigherVNC, and RealVNC (aka VNC Viewer) have been tested on various hosts. The OSX VNC client is known not to work.” The VNC server supports an optional ‘wait’ parameter, that causes the VM to not actually boot until the VNC client connects, allowing you to interrupt the boot process if need be Related user blog post (http://justinholcomb.me/blog/2016/05/28/bhyve-uefi-gop-support.html) SVN commit (https://svnweb.freebsd.org/base?view=revision&revision=300829) *** zfsd lands in FreeBSD HEAD, in time for 11.0-RELEASE (https://svnweb.freebsd.org/base?view=revision&revision=300906) zfsd has been committed to FreeBSD -CURRENT in time to be included in FreeBSD 11.0 zfsd is the missing piece required to make ‘hot spares’ work properly in FreeBSD ZFS “zfsd attempts to resolve ZFS faults that the kernel can't resolve by itself. It listens to devctl(4) events, which is how the kernel notifies of events such as I/O errors and disk removals. Zfsd attempts to resolve these faults by activating or deactivating hotspares and onlining offline vdevs.” “The administrator never interacts with zfsd directly. Instead, he controls its behavior indirectly through zpool configuration. There are two ways to influence zfsd: assigning hotspares and setting pool properties. Currently, only the autoreplace property has any effect. See zpool(8) for details.” So, what example does it do? Device Removal: “When a leaf vdev disappears, zfsd will activate any available hotspare.” Device Arrival: “When a new GEOM device appears, zfsd will attempt to read its ZFS label, if any. If it matches a previously removed vdev on an active pool, zfsd will online it. Once resilvering completes, any active hotspare will detach automatically.” So if you disconnect a drive, then reconnect it, it will automatically be brought back online. Since ZFS is smart, the resilver will only have to copy data that has changed since the device went offline. “If the new device has no ZFS label but its physical path matches the physical path of a previously removed vdev on an active pool, and that pool has the autoreplace property set, then zfsd will replace the missing vdev with the newly arrived device. Once resilvering completes, any active hotspare will detach automatically.” If the new drive is in the same slot in your hot swap array as a failed device, it will be used as a replacement immediately. vdev degrade or fault events: “If a vdev becomes degraded or faulted, zfsd will activate any available hotspare. If a leaf vdev generates more than 50 I/O errors in a 60 second period, then zfsd will mark that vdev as FAULTED. zfs(4) will no longer issue any I/Os to it. zfsd will activate a hotspare if one is available.” Same for checksum errors. So if zfsd detects a drive is going bad, it brings the hotspare online before it is too late Spare addition: “If the system administrator adds a hotspare to a pool that is already degraded, zfsd will activate the spare.” Resilver complete: “zfsd will detach any hotspare once a permanent replacement finishes resilvering.” Physical path change: “If the physical path of an existing disk changes, zfsd will attempt to replace any missing disk with the same physical path, if its pool's autoreplace property is set.” In general, this tool means less reliance on the system administrator to keep the pool healthy *** W^X now mandatory in OpenBSD (http://undeadly.org/cgi?action=article&sid=20160527203200) We’ve talked a bit about W^X in the past. (Refresher: Memory being writable and executable at once) Well, this major security no-no is no-more on OpenBSD. Theo has committed a change which now prevents violations of this policy: “W^X violations are no longer permitted by default. A kernel log message is generated, and mprotect/mmap return ENOTSUP. If the sysctl(8) flag kern.wxabort is set then a SIGABRT occurs instead, for gdb use or coredump creation.” There are a few cases where you may still need W^X, which Theo points out can be enabled on a file-system basis. “W^X violating programs can be permitted on a ffs/nfs filesystem-basis, using the "wxallowed" mount option. One day far in the future upstream software developers will understand that W^X violations are a tremendously risky practice and that style of programming will be banished outright. Until then, we recommend most users need to use the wxallowed option on their /usr/local filesystem. At least your other filesystems don't permit such programs.” This is a great ability to grow, since now users can begin doing auditing of programs that violate this principle and making noise to upstream. *** Interview - Kristof Provost - kp@freebsd.org (mailto:kp@freebsd.org) @kprovst (https://twitter.com/kprovst) pf improvements on FreeBSD *** News Roundup GELI Support for the EFI Loader (https://ericmccorkleblog.wordpress.com/2016/05/28/freebsd-geli-support/) We’ve had Allan’s work to bring GELI support to the GPT / BIOS / ZFS loader for a while now, but the missing piece has been support for EFI. No longer, Eric McCorkle has posted a blog entry (with relevant github links) introducing us to his work to bring GELI encryption support to EFI. First the bad-news. This won’t make it into 11.0. (Maybe PC-BSD, TBD) Next he explains why this is more than just a new feature, but a re-factor of the EFI boot code: I have already written extensively about my EFI refactoring here. The reason for undertaking this effort, however, was driven by GELI support. Early in my work on this, I had implemented a non-EFI “providers” framework in boot1 in order to support the notion of disk partitions that may contain sub-partitions. This was deeply unsatisfying to me for several reasons: It implemented a lot of the same functionality that exists in the EFI framework. It involved implementing a GPT partition driver to deal with partition tables inside GELI partitions (GPT detection and support is guaranteed by the EFI spec). The interface between the EFI framework and the custom “providers” framework was awkward. The driver was completely boot1-specific, and exporting it to something like GRUB probably involved a total rewrite. Implementing it within loader was going to involve a lot of code duplication. There was no obvious was to pass keys between boot1, loader, and the kernel. With the issues known, Eric seems pleased with the results of the conversion so far: The GELI driver can be extracted from the FreeBSD codebase without too much trouble. While I was unable to go all the way to the EFI driver model, the only blocker is the bcache code, and once that is resolved, we can have hotplug support in the boot loader! The boot1 and loader codebases are now sharing all the backend drivers, and boot1 has been reduced to one very small source file. An interesting read, looking forward to playing with EFI more in the future! *** Faces of FreeBSD 2016: Michael W. Lucas (https://www.freebsdfoundation.org/blog/faces-of-freebsd-2016-michael-lucas/) On this edition of “Faces of FreeBSD”, Michael W Lucas tells the story of how he got started with FreeBSD After an amusing re-telling of his childhood (The words “Purina Monkey Chow” were mentioned), he then tells us how he got into BSD. His being thrown into the project may sound familiar to many: I came in at 11 PM one night and was told “The DNS administrator just got walked out the door. You’re the new lead DNS administrator. Make those servers work. Good luck.” From there (because he wanted more sleep), he began ripping out the systems that had been failing and waking him up at night. Good-bye UnixWare, Good-bye Solaris, hello BSD! A very amusing read, check it out! *** High Availability with PostgreSQL on FreeBSD (https://www.youtube.com/watch?v=ugct9-Mm7Ls) A talk by Sean Chittenden, who we interviewed previously on episode Episode 95 (http://www.bsdnow.tv/episodes/2015_06_24-bitrot_group_therapy) Explains how to setup Multi Data Center High Availability for PostgreSQL using consul Goes into how consul works, how it does the election, the gossip protocol, etc The HA setup uses DNS Failover, and the pros and cons of that approach are discussed Then he walks through the implementation details, and example configuration *** New FreeBSD i915 testing images (http://www.bsddesktop.com/images/) Still need users to test the Linux Kernel 4.6 DRM update to FreeBSD’s graphics stack Download the test image and write it to a USB stick and boot from it It will not modify your installed system, it runs entirely off of the USB drive Allows you to test the updated drivers without having to install the development branch on your device you can tell them that ATI/AMD support will be coming shortly and that stability has been steadily improving and that I'll do another announcement as soon as I've had a chance to test the newest Xorg bits *** Beastie Bits Comfortable on the CLI: Series Part 1 (https://www.cotcli.com/post/The-Very-Basics/) FreeBSD Booting on the Netgate uFW, a smaller-than-a-raspberry-pi dual port firewall (https://gist.github.com/gonzopancho/8e7df7a826e9a2949b36ed2a9d30312e) Picture of uFW (https://twitter.com/gonzopancho/status/737874921435594753) uFW OpenSSL Benchmarks (https://gist.github.com/gonzopancho/8f20b50487a4f7de56e99448866a147d) ***
143: One small step for DRM, one giant leap for BSD
This week on BSDNow, we have an interview with Matthew Macy, who has some exciting news to share with us regarding the state of graphics This episode was brought to you by Headlines How the number of states affects pf’s performance of FreeBSD (http://blog.cochard.me/2016/05/playing-with-freebsd-packet-filter.html) Our friend Olivier of FreeNAS and BSDRP fame has an interesting blog post this week detailing his unique issue with finding a firewall that can handle upwards of 4 million state table entries. He begins in the article with benchmarking the defaults, since without that we don’t have a framework to compare the later results. All done on his Netgate RCC-VE 4860 (4 cores ATOM C2558, 8GB RAM) under FreeBSD 10.3. “We notice a little performance impact when we reach the default 10K state table limit: From 413Kpps with 128 states in-used, it lower to 372Kpps.” With the initial benchmarks done and graphed, he then starts the tuning process by adjusting the “net.pf.states_hashsize”sysctl, and then playing with the number of states for the firewall to keep. “For the next bench, the number of flow will be fixed for generating 9800 pf state entries, but I will try different value of pf.states_hashsize until the maximum allowed on my 8GB RAM server (still with the default max states of 10k):” Then he cranks it up to 4 million states “There is only 12% performance penalty between pf 128 pf states and 4 million pf states.” “With 10M state, pf performance lower to 362Kpps: Still only 12% lower performance than with only 128 states” He then looks at what this does of pfsync, the protocol to sync the state table between two redundant pf firewalls Conclusions: There need to be a linear relationship between the pf hard-limit of states and the pf.stateshashsize; RAM needed for pf.stateshashsize = pf.stateshashsize * 80 Byte and pf.stateshashsize should be a power of 2 (from the manual page); Even small hardware can manage large number of sessions (it's a matter of RAM), but under too lot's of pressure pfsync will suffer. Introducing the BCHS Stack = BSD, C, httpd, SQLite (http://www.learnbchs.org/) Pronounced Beaches “It's a hipster-free, open source software stack for web applications” “Don't just write C. Write portable and secure C.” “Get to know your security tools. OpenBSD has systrace(4) and pledge(2). FreeBSD has capsicum(4).” “Statically scan your binary with LLVM” and “Run your application under valgrind” “Don't forget: BSD is a community of professionals. Go to conferences (EuroBSDCon, AsiaBSDCon, BSDCan, etc.)” This seems like a really interesting project, we’ll have to get Kristaps Dzonsons back on the show to talk about it *** Installing OpenBSD's httpd server, MariaDB, PHP 5.6 on OpenBSD 5.9 (https://www.rootbsd.net/kb/339/Installing-OpenBSDandsharp039s-httpd-server-MariaDB-PHP-56-on-OpenBSD-59.html) Looking to deploy your next web-stack on OpenBSD 5.9? If so this next article from rootbsd.net is for you. Specifically it will walk you through the process of getting OpenBSD’s own httpd server up and running, followed by MariaDB and PHP 5.6. Most of the setup is pretty straight-forward, the httpd syntax may be different to you, if this is your first time trying it out. Once the various packages are installed / configured, the rest of the tutorial will be easy, walking you through the standard hello world PHP script, and enabling the services to run at reboot. A good article for those wanting to start hosting PHP/DB content (wordpress anyone?) on your OpenBSD system. *** The infrastructure behind Varnish (https://www.varnish-cache.org/news/20160425_website.html) Dogfooding. It’s a term you hear often in the software community, which essentially means to “Run your own stuff”. Today we have an article by PKH over at varnish-cache, talking about what that means to them. Specifically, they recently went through a website upgrade, which will enable them to run more of their own stuff. He has a great quote on what OS they use:“So, dogfood: Obviously FreeBSD. Apart from the obvious reason that I wrote a lot of FreeBSD and can get world-class support by bugging my buddies about it, there are two equally serious reasons for the Varnish Project to run on FreeBSD: Dogfood and jails.Varnish Cache is not “software for Linux”, it is software for any competent UNIX-like operating system, and FreeBSD is our primary “keep us honest about this” platform.“ He then goes through the process of explaining how they would setup a new Varnish-cache website, or upgrade it. All together a great read, and if you are one of the admin-types, you really should pay attention to how they build from the ground up. Some valuable knowledge here which every admin should try to replicate. I can not reiterate the value of having your config files in a private source control repo strongly enough The biggest take-away is: “And by doing it this way, I know it will work next time also.” *** Interview - Matt Macy - mmacy@nextbsd.org (mailto:mmacy@nextbsd.org)Graphics Stack Update (https://lists.freebsd.org/pipermail/freebsd-x11/2016-May/017560.html) News Roundup Followup on packaging base with pkg(8) (https://lists.freebsd.org/pipermail/freebsd-pkgbase/2016-May/000238.html) In spite of the heroic last minute effort by a team of contributors, pkg’d base will not be ready in time for FreeBSD 11.0 There are just too many issues that were discovered during testing The plan is to continue using freebsd-update in the meantime, and introduce a pkg based upgrade mechanism in FreeBSD 11.1 With the new support model for the FreeBSD 11 branch, 11.1 may come sooner than with previous major releases *** FreeBSD Core Election (https://www.freebsd.org/internal/bylaws.html) It is time once again for the FreeBSD Core Election Application period begins: Wednesday, 18 May 2016 at 18:00:00 UTC Application period ends: Wednesday, 25 May 2016 at 18:00:00 UTC Voting begins: Wednesday, 25 May 2016 at 18:00:00 UTC Voting ends: Wednesday, 22 June 2016 at 18:00:00 UTC Results announced Wednesday, 29 June 2016 New core team takes office: Wednesday, 6 July 2016 As of the time I was writing these notes, 3 hours before the application deadline, the candidates are: Allan Jude: Filling in the potholes Marcelo Araujo: We are not vampires, but we need new blood. Baptiste Daroussin (incumbent): Keep on improving Benedict Reuschling: Learn and Teach Benno Rice: Revitalising The Community Devin Teske: Here to help Ed Maste (incumbent): FreeBSD is people George V. Neville-Neil (incumbent): There is much to do… Hiroki Sato (incumbent): Keep up with our good community and technical strength John Baldwin: Ready to work Juli Mallett: Caring for community. Kris Moore: User-Focused Mathieu Arnold: Someone ask for fresh blood ? Ollivier Robert: Caring for the project and you, its developers The deadline for applications is around the time we finish recording the live show We welcome any of the candidates to schedule an interview in the next few weeks. We will make an attempt to hunt many of them down at BSDCan as well. *** Wayland/Weston with XWayland works on DragonFly (http://lists.dragonflybsd.org/pipermail/users/2016-May/249620.html) We haven’t talked a lot about Wayland on BSD recently (or much at all), but today we have a post from Peter to the dragonfly mailing list, detailing his experience with it. Specifically he talks about getting XWayland working, which provides the compat bits for native X applications to run on WayLand displays. So far on the working list of apps: “gtk3: gedit nautilus evince xfce4: - xfce4-terminal - atril firefox spyder scilab” A pretty impressive list, although he said “chrome” failed with a seg-fault This is something I’m personally interested in. Now with the newer DRM bits landing in FreeBSD, perhaps it’s time for some further looking into Wayland. Broadcom WiFi driver update (http://adrianchadd.blogspot.ca/2016/05/updating-broadcom-softmac-driver-bwn-or.html) In this blog post Adrian Chadd talks about his recent work on the bwn(4) driver for Broadcom WiFi chips This work has added support for a number of older 802.11g chips, including the one from 2009-era Macbooks Work is ongoing, and the hope is to add 802.11n and 5ghz support as well Adrian is mentoring a number of developers working on embedded or wifi related things, to try to increase the projects bandwidth in those areas If you are interested in driver development, or wifi internals, the blog post has lots of interesting details and covers the story of Adrian’s recent adventures in bringing the drivers up *** Beastie Bits The Design of the NetBSD I/O Subsystems (2002) (http://arxiv.org/abs/1605.05810) ZFS, BTRFS, XFS, EXT4 and LVM with KVM – a storage performance comparison (http://www.ilsistemista.net/index.php/virtualization/47-zfs-btrfs-xfs-ext4-and-lvm-with-kvm-a-storage-performance-comparison.html?print=true) Swift added to FreeBSD Ports (http://www.freshports.org/lang/swift/) misc@openbsd: 'NSA addition to ifconfig' (http://marc.info/?l=openbsd-misc&m=146391388912602&w=2) Papers We Love: Memory by the Slab: The Tale of Bonwick's Slab Allocator (http://paperswelove.org/2015/video/ryan-zezeski-memory-by-the-slab/) Feedback/Questions Lars - Poudriere (http://pastebin.com/HRRyfxev) Warren - .NET (http://pastebin.com/fESV1egk) Eddy - Sys Init (http://pastebin.com/kQecpA1X) Tim - ZFS Resources (http://pastebin.com/5096cGXr) Morgan - Ports and Kernel (http://pastebin.com/rYr1CDcV) ***