Created by three guys who love BSD, we cover the latest news and have an extensive series of tutorials, as well as interviews with various people from all areas of the BSD community. It also serves as a platform for support and questions. We love and advocate FreeBSD, OpenBSD, NetBSD, DragonFlyBSD and TrueOS. Our show aims to be helpful and informative for new users that want to learn about them, but still be entertaining for the people who are already pros. The show airs on Wednesdays at 2:00PM (US Eastern time) and the edited version is usually up the following day.
Similar Podcasts
Elixir Outlaws
Elixir Outlaws is an informal discussion about interesting things happening in Elixir. Our goal is to capture the spirit of a conference hallway discussion in a podcast.
The Cynical Developer
A UK based Technology and Software Developer Podcast that helps you to improve your development knowledge and career,
through explaining the latest and greatest in development technology and providing you with what you need to succeed as a developer.
Programming Throwdown
Programming Throwdown educates Computer Scientists and Software Engineers on a cavalcade of programming and tech topics. Every show will cover a new programming language, so listeners will be able to speak intelligently about any programming language.
209: Signals: gotta catch ‘em all
We read a trip report about FreeBSD in China, look at how Unix deals with Signals, a stats collector in DragonFlyBSD & much more! This episode was brought to you by Headlines Trip Report: FreeBSD in China at COPU and LinuxCon (https://www.freebsdfoundation.org/blog/trip-report-freebsd-in-china-at-copu-and-linuxcon/) This trip report is from Deb Goodkin, the Executive Director of the FreeBSD Foundation. She travelled to China in May 2017 to promote FreeBSD, meet with companies, and participate in discussions around Open Source. > In May of 2017, we were invited to give a talk about FreeBSD at COPU’s (China Open Source Promotional Unit) Open Source China, Open Source World Summit, which took place June 21-22, in Beijing. This was a tremendous opportunity to talk about the advantages of FreeBSD to the open source leaders and organizations interested in open source. I was honored to represent the Project and Foundation and give the presentation “FreeBSD Advantages and Applications”. > Since I was already going to be in Beijing, and LinuxCon China was being held right before the COPU event, Microsoft invited me to be part of a women-in-tech panel they were sponsoring. There were six of us on the panel including two from Microsoft, one from the Linux Foundation, one from Accenture of China, and one from Women Who Code. Two of us spoke in English, with everyone else speaking Chinese. It was disappointing that we didn’t have translators, because I would have loved hearing everyone’s answers. We had excellent questions from the audience at the end. I also had a chance to talk with a journalist from Beijing, where I emphasized how contributing to an open source project, like FreeBSD, is a wonderful way to get experience to boost your resume for a job. > The first day of LinuxCon also happened to be FreeBSD Day. I had my posters with me and was thrilled to have the Honorary Chairman of COPU (also known as the “Father of Open Source in China”) hold one up for a photo op. Unfortunately, I haven’t been able to get a copy of that photo for proof (I’m still working on it!). We spent a long time discussing the strengths of FreeBSD. He believes there are many applications in China that could benefit from FreeBSD, especially for embedded devices, university research, and open source education. We had more time throughout the week to discuss FreeBSD in more detail. > Since I was at LinuxCon, I had a chance to meet with people from the Linux Foundation, other open source projects, and some of our donors. With LinuxCon changing its name to Open Source Summit, I discussed how important it is to include minority voices like ours to contribute to improving the open source ecosystem. The people I talked to within the Linux Foundation agreed and suggested that we get someone from the Project to give a talk at the Open Source Summit in Prague this October. Jim Zemlin, the Linux Foundation Executive Director, suggested having a BSD track at the summits. We did miss the call for proposals for that conference, but we need to get people to consider submitting proposals for the Open Source Summits in 2018. > I talked to a CTO from a company that donates to us and he brought up his belief that FreeBSD is much easier to get started on as a contributor. He talked about the steep path in Linux to getting contributions accepted due to having over 10,000 developers and the hierarchy of decision makers, from Linus to his main lieutenants to the layers beneath him. It can take 6 months to get your changes in! > On Tuesday, Kylie and I met with a representative from Huawei, who we’ve been meeting over the phone with over the past few months. Huawei has a FreeBSD contributor and is looking to add more. We were thrilled to hear they decided to donate this year. We look forward to helping them get up to speed with FreeBSD and collaborate with the Project. > Wednesday marked the beginning of COPU and the reason I flew all the way to Beijing! We started the summit with having a group photo of all the speakers:The honorary chairman, Professor Lu in the front middle. > My presentation was called “FreeBSD Advantages and Applications”. A lot of the material came from Foundation Board President, George-Neville-Neil’s presentation, “FreeBSD is not a Linux Distribution”, which is a wonderful introduction to FreeBSD and includes the history of FreeBSD, who uses it and why, and which features stand out. My presentation went well, with Professor Lu and others engaged through the translators. Afterwards, I was invited to a VIP dinner, which I was thrilled about. > The only hitch was that Kylie and I were running a FreeBSD meetup that evening, and both were important! Beijing during rush hour is crazy, even trying to go only a couple of miles is challenging. We made plans that I would go to the meetup and give the same presentation, and then head back to the dinner. Amazingly, it worked out. Check out the rest of her trip report and stay tuned for more news from the region as this is one of the focus areas of the Foundation. *** Unix: Dealing with signals (http://www.networkworld.com/article/3211296/linux/unix-dealing-with-signals.html) Signals on Unix systems are critical to the way processes live and die. This article looks at how they're generated, how they work, and how processes receive or block them On Unix systems, there are several ways to send signals to processes—with a kill command, with a keyboard sequence (like control-C), or through a program Signals are also generated by hardware exceptions such as segmentation faults and illegal instructions, timers and child process termination. But how do you know what signals a process will react to? After all, what a process is programmed to do and able to ignore is another issue. Fortunately, the /proc file system makes information about how processes handle signals (and which they block or ignore) accessible with commands like the one shown below. In this command, we’re looking at information related to the login shell for the current user, the "$$" representing the current process. On FreeBSD, you can use procstat -i PID to get that and even more information, and easier to digest form P if signal is pending in the global process queue I if signal delivery disposition is SIGIGN C if signal delivery is to catch it Catching a signal requires that a signal handling function exists in the process to handle a given signal. The SIGKILL (9) and SIGSTOP (#) signals cannot be ignored or caught. For example, if you wanted to tell the kernel that ctrl-C's are to be ignored, you would include something like this in your source code: signal(SIGINT, SIGIGN); To ensure that the default action for a signal is taken, you would do something like this instead: signal(SIGSEGV, SIGDFL); + The article then shows some ways to send signals from the command line, for example to send SIGHUP to a process with pid 1234: kill -HUP 1234 + You can get a list of the different signals by running kill -l On Unix systems, signals are used to send all kinds of information to running processes, and they come from user commands, other processes, and the kernel itself. Through /proc, information about how processes are handling signals is now easily accessible and, with just a little manipulation of the data, easy to understand. links owned by NGZ erroneously marked as on loan (https://smartos.org/bugview/OS-6274) NGZ (Non-Global Zone), is IllumOS speak for their equivalent to a jail > As reported by user brianewell in smartos-live#737, NGZ ip tunnels stopped persisting across zone reboot. This behavior appeared in the 20170202 PI and was not present in previous releases. After much spelunking I determined that this was caused by a regression introduced in commit 33df115 (part of the OS-5363 work). The regression was a one-line change to link_activate() which marks NGZ links as on loan when they are in fact not loaned because the NGZ created and owns the link. “On loan” means the interface belongs to the host (GZ, Global Zone), and has been loaned to the NGZ (Jail) This regression was easy to introduce because of the subtle nature of this code and lack of comments. I'm going to remove the regressive line, add clarifying comments, and also add some asserts. The following is a detailed analysis of the issue, how I debugged it, and why my one-line change caused the regression: To start I verified that PI 20170119 work as expected: booted 20170119 created iptun (named v4sys76) inside of a native NGZ (names sos-zone) performed a reboot of sos-zone zlogin to sos-zone and verify iptun still exists after reboot Then I booted the GZ into PI 20170202 and verified the iptun did not show up booted 20170202 started sos-zone zlogin and verified the iptun was missing At this point I thought I would recreate the iptun and see if I could monitor the zone halt/boot process for the culprit, but instead I received an error from dladm: "object already exists". I didn't expect this. So I used mdb to inspect the dlmgmtd state. Sure enough the iptun exists in dlmgmtd. Okay, so if the link already exists, why doesn't it show up (in either the GZ or the NGZ)? If a link is not marked as active then it won't show up when you query dladm. When booting the zone on 20170119 the llflags for the iptun contained the value 0x3. So the problem is the link is not marked as active on the 20170202 PI. The linkactivate() function is responsible for marking a link as active. I used dtrace to verify this function was called on the 20170202 PI and that the dlmgmtlinkt had the correct llflags value. So the iptun link structure has the correct llflags when linkactivate() returns but when I inspect the same structure with mdb afterwards the value has changed. Sometime after linkactivate() completes some other process changed the llflags value. My next question was: where is linkactivate() called and what comes after it that might affect the llflags? I did another trace and got this stack. The dlmgmtupid() function calls dlmgmtwritedbentry() after linkactivate() and that can change the flags. But dtrace proved the llflags value was still 0x3 after returning from this function. With no obvious questions left I then asked cscope to show me all places where llflags is modified. As I walked through the list I used dtrace to eliminate candidates one at a time -- until I reached dlmgmtdestroycommon(). I would not have expected this function to show up during zone boot but sure enough it was being called somehow, and by someone. Who? Since there is no easy way to track door calls it was at this point I decided to go nuclear and use the dtrace stop action to stop dlmgmtd when it hits dlmgmtdestroycommon(). Then I used mdb -k to inspect the door info for the dlmgmtd threads and look for my culprit. The culprit is doupiptun() caused by the dladm up-iptun call. Using ptree I then realized this was happening as part of the zone boot under the network/iptun svc startup. At this point it was a matter of doing a zlogin to sos-zone and running truss on dladm up-iptun to find the real reason why dladmdestroydatalinkid() is called. So the link is marked as inactive because dladmgetsnapconf() fails with DLADMSTATUSDENIED which is mapped to EACCESS. Looking at the dladmgetsnapconf() code I see the following “The caller is in a non-global zone and the persistent configuration belongs to the global zone.” What this is saying is that if a link is marked "on loan" (meaning it's technically owned/created by the GZ but assigned/loaned to the NGZ) and the zone calling dladmgetsnapconf() is an NGZ then return EACCESS because the configuration of the link is up to the GZ, not the NGZ. This code is correct and should be enforced, but why is it tripping in PI 20170202 and not 20170119? It comes back to my earlier observation that in the 20170202 PI we marked the iptun as "on loan" but not in the older one. Why? Well as it turns out while fixing OS-5363 I fixed what I thought was a bug in linkactivate() When I first read this code it was my understanding that anytime we added a link to a zone's datalink list, by calling zoneadddatalink(), that link was then considered "on loan". My understanding was incorrect. The linkactivate() code has a subtleness that eluded me. There are two cases in linkactivate(): 1. The link is under an NGZ's datalink list but it's lllinkid doesn't reflect that (e.g., the link is found under zoneid 3 but lllinkid is 0). In this case the link is owned by the GZ but is being loaned to an NGZ and the link state should be updated accordingly. We get in this situation when dlmgmtd is restated for some reason (it must resync it's in-memory state with the state of the system). 2. The link is NOT under any NGZ's (zonecheckdatalink() is only concerned with NGZs) datalink list but its llzoneid holds the value of an NGZ. This indicates that the link is owned by an NGZ but for whatever reason is not currently under the NGZ's datalink list (e.g., because we are booting the zone and we now need to assign the link to its list). So the fix is to revert that one line change as well as add some clarifying comments and also some asserts to prevent further confusion in the future. + A nice breakdown by Ryan Zezeski of how he accidently introduced a regression, and how he tracked it down using dtrace and mdb New experimental statistics collector in master (http://dpaste.com/2YP0X9C) Master now has an in-kernel statistics collector which is enabled by default, and a (still primitive) user land program to access it. This recorder samples the state of the machine once every 10 seconds and records it in a large FIFO, all in-kernel. The FIFO typically contains 8192 entries, or around the last 23 hours worth of data. Statistics recorded include current load, user/sys/idle cpu use, swap use, VM fault rate, VM memory statistics, and counters for syscalls, path lookups, and various interrupt types. A few more useful counters will probably be added... I'd like to tie cpu temperature, fork rate, and exec rate in at some point, as well as network and disk traffic. The statistics gathering takes essentially no real overhead and is always on, so any user at the spur of the moment with no prior intent can query the last 23 hours worth of data. There is a user frontend to the data called 'kcollect' (its tied into the buildworld now). Currently still primitive. Ultimately my intention is to integrate it with a dbm database for long-term statistical data retention (if desired) using an occasional (like once-an-hour) cron-job to soak up anything new, with plenty of wiggle room due to the amount of time the kernel keeps itself. This is better and less invasive than having a userland statistics gathering script running every few minutes from cron and has the advantage of giving you a lot of data on the spur of the moment without having to ask for it before-hand. If you have gnuplot installed (pkg install gnuplot), kcollect can generate some useful graphs based on the in-kernel data. Well, it will be boring if the machine isn't doing anything :-). There are options to use gnuplot to generate a plot window in X or a .jpg or .png file, and other options to set the width and height and such. At the moment the gnuplot output uses a subset of statically defined fields to plot but ultimately the field list it uses will be specifiable. Sample image generated during a synth run (http://apollo.backplane.com/DFlyMisc/kcollect03.jpg) News Roundup openbsd changes of note 626 (https://www.tedunangst.com/flak/post/openbsd-changes-of-note-626) Hackerthon is imminent. There are two signals one can receive after accessing invalid memory, SIGBUS and SIGSEGV. Nobody seems to know what the difference is or should be, although some theories have been unearthed. Make some attempt to be slightly more consistent and predictable in OpenBSD. Introduces jiffies in an effort to appease our penguin oppressors. Clarify that IP.OF.UPSTREAM.RESOLVER is not actually the hostname of a server you can use. Switch acpibat to use _BIX before _BIF, which means you might see discharge cycle counts, too. Assorted clang compatibility. clang uses -Oz to mean optimize for size and -Os for something else, so make gcc accept -Oz so all makefiles can be the same. Adjust some hardlinks. Make sure we build gcc with gcc. The SSLcheckprivate_key function is a lie. Switch the amd64 and i386 compiler to clang and see what happens. We are moving towards using wscons (wstpad) as the driver for touchpads. Dancing with the stars, er, NET_LOCK(). clang emits lots of warnings. Fix some of them. Turn off a bunch of clang builtins because we have a strong preference that code use our libc versions. Some other changes because clang is not gcc. Among other curiosities, static variables in the special .openbsd.randomdata are sometimes assumed to be all zero, leading the clang optimizer to eliminate reads of such variables. Some more pledge rules for sed. If the script doesn’t require opening new files, don’t let it. Backport a bajillion fixes to stable. Release errata. RFC 1885 was obsoleted nearly 20 years ago by RFC 2463 which was obsoleted over 10 years ago by RFC 4443. We are probably not going back. Update libexpat to 2.2.3. vmm: support more than 3855MB guest memory. Merge libdrm 2.4.82. Disable SSE optimizations on i386/amd64 for SlowBcopy. It is supposed to be slow. Prevents crashes when talking to memory mapped video memory in a hypervisor. The $25 “FREEDOM Laptop!” (https://functionallyparanoid.com/2017/08/08/the-25-freedom-laptop/) Time to get back to the original intent of this blog – talking about my paranoid obsession with information security! So break out your tinfoil hats my friends because this will be a fun ride. I’m looking for the most open source / freedom respecting portable computing experience I can possibly find and I’m going to document my work in real-time so you will get to experience the ups (and possibly the downs) of that path through the universe. With that said, let’s get rolling. When I built my OpenBSD router using the APU2 board, I discovered that there are some amd64 systems that use open source BIOS. This one used Coreboot and after some investigation I discovered that there was an even more paranoid open source BIOS called Libreboot out there. That started to feel like it might scratch my itch. Well, after playing around with some lower-powered systems like my APU2 board, my Thinkpad x230 and my SPARC64 boxes, I thought, if it runs amd64 code and I can run an open source operating system on it, the thing should be powerful enough for me to do most (if not all) of what I need it to do. At this point, I started looking for a viable machine. From a performance perspective, it looked like the Thinkpad x200, T400, T500 and W500 were all viable candidates. After paying attention on eBay for a while, I saw something that was either going to be a sweet deal, or a throwaway piece of garbage! I found a listing for a Thinkpad T500 that said it didn’t come with a power adapter and was 100% untested. From looking at the photos, it seemed like there was nothing that had been molested about it. Obviously, nobody was jumping on something this risky so I thought, “what the heck” and dropped a bit at the opening price of $24.99. Well, guess what. I won the auction. Now to see what I got. When the laptop showed up, I discovered it was minus its hard drive (but the outside plastic cover was still in place). I plugged in my x230’s power adapter and hit the button. I got lights and was dropped to the BIOS screen. To my eternal joy, I discovered that the machine I had purchased for $25 was 100% functional and included the T9400 2.54 GHz Core 2 Duo CPU and the 1680×1050 display panel. W00t! First things first, I need to get this machine a hard drive and get the RAM upgraded from the 2GB that it showed up with to 8GB. Good news is that these two purchases only totaled $50 for the pair. An aftermarket 9-cell replacement battery was another $20. Throw in a supported WiFi card that doesn’t require a non-free blob from Libreboot at $5.99 off of eBay and $5 for a hard drive caddy and I’m looking at about $65 in additional parts bringing the total cost of the laptop, fully loaded up at just over $100. Not bad at all… Once all of the parts arrived and were installed, now for the fun part. Disassembling the entire thing down to the motherboard so we can re-flash the BIOS with Libreboot. The guide looks particularly challenging for this but hey, I have a nice set of screwdrivers from iFixit and a remarkable lack of fear when it comes to disassembling things. Should be fun! Well, fun didn’t even come close. I wish I had shot some pictures along the way because at one point I had a heap of parts in one corner of my “workbench” (the dining room table) and just the bare motherboard, minus the CPU sitting in front of me. With the help of a clip and a bunch of whoops wires (patch cables), I connected my Beaglebone Black to the BIOS chip on the bare motherboard and attempted to read the chip. #fail I figured out after doing some more digging that you need to use the connector on the left side of the BBB if you hold it with the power connector facing away from you. In addition, you should probably read the entire process through instead of stopping at the exciting pinout connector diagram because I missed the bit about the 3.3v power supply need to have ground connected to pin 2 of the BIOS chip. Speaking of that infamous 3.3v power supply, I managed to bend a paperclip into a U shape and jam it into the connector of an old ATX power supply I had in a closet and source power from that. I felt like MacGyver for that one! I was able to successfully read the original Thinkpad BIOS and then flash the Libreboot + Grub2 VESA framebuffer image onto the laptop! I gulped loudly and started the reassembly process. Other than having some cable routing difficulties because the replacement WiFi card didn’t have a 5Ghz antenna, it all went back together. Now for the moment of truth! I hit the power button and everything worked!!! At this point I happily scurried to download the latest snapshot of OpenBSD – current and install it. Well, things got a little weird here. Looks like I have to use GRUB to boot this machine now and GRUB won’t boot an OpenBSD machine with Full Disk Encryption. That was a bit of a bummer for me. I tilted against that windmill for several days and then finally admitted defeat. So now what to do? Install Arch? Well, here’s where I think the crazy caught up to me. I decided to be an utter sell out and install Ubuntu Gnome Edition 17.04 (since that will be the default DE going forward) with full disk encryption. I figured I could have fun playing around in a foreign land and try to harden the heck out of that operating system. I called Ubuntu “grandma’s Linux” because a friend of mine installed it on his mom’s laptop for her but I figured what the heck – let’s see how the other half live! At this point, while I didn’t have what I originally set out to do – build a laptop with Libreboot and OpenBSD, I did have a nice compromise that is as well hardened as I can possibly make it and very functional in terms of being able to do what I need to do on a day to day basis. Do I wish it was more portable? Of course. This thing is like a six or seven pounder. However, I feel much more secure in knowing that the vast majority of the code running on this machine is open source and has all the eyes of the community on it, versus something that comes from a vendor that we cannot inspect. My hope is that someone with the talent (unfortunately I lack those skills) takes an interest in getting FDE working with Libreboot on OpenBSD and I will most happily nuke and repave this “ancient of days” machine to run that! FreeBSD Programmers Report Ryzen SMT Bug That Hangs Or Resets Machines (https://hothardware.com/news/freebsd-programmers-report-ryzen-smt-bug-that-hangs-or-resets-machines) It's starting to look like there's an inherent bug with AMD's Zen-based chips that is causing issues on Unix-based operating systems, with both Linux and FreeBSD confirmed. The bug doesn't just affect Ryzen desktop chips, but also AMD's enterprise EPYC chips. It seems safe to assume that Threadripper will bundle it in, as well. It's not entirely clear what is causing the issue, but it's related to the CPU being maxed out in operations, thus causing data to get shifted around in memory, ultimately resulting in unstable software. If the bug is exercised a certain way, it can even cause machines to reset. The revelation about the issue on FreeBSD was posted to the official repository, where the issue is said to happen when threads can lock up, and then cause the system to become unstable. Getting rid of the issue seems as simple as disabling SMT, but that would then negate the benefits provided by having so many threads at-the-ready. On the Linux side of the Unix fence, Phoronix reports on similar issues, where stressing Zen chips with intensive benchmarks can cause one segmentation fault after another. The issue is so profound, that Phoronix Test Suite developer Michael Larabel introduced a special test that can be run to act as a bit of a proof-of-concept. To test another way, PTS can be run with this command: PTS_CONCURRENT_TEST_RUNS=4 TOTAL_LOOP_TIME=60 phoronix-test-suite stress-run build-linux-kernel build-php build-apache build-imagemagick Running this command will compile four different software projects at once, over and over, for an hour. Before long, segfaults should begin to appear (as seen in the shot above). It's not entirely clear if both sets of issues here are related, but seeing as both involve stressing the CPU to its limit, it seems likely. Whether or not this could be patched on a kernel or EFI level is something yet to be seen. TrueOS - UNSTABLE update: 8/7/17 (https://www.trueos.org/blog/unstable-update-8717/) A new UNSTABLE update for TrueOS is available! Released regularly, UNSTABLE updates are the full “rolling release” of TrueOS. UNSTABLE includes experimental features, bugfixes, and other CURRENT FreeBSD work. It is meant to be used by those users interested in using the latest TrueOS and FreeBSD developments to help test and improve these projects. WARNING: UNSTABLE updates are released primarily for TrueOS and FreeBSD testing/experimentation purposes. Update and run UNSTABLE “at your own risk”. Note: There was a CDN issue over the weekend that caused issues for early updaters. Everything appears to be resolved and the update is fully available again. If you encountered instability or package issues from updating on 8/6 or 8/5, roll back to a previous boot environment and run the update again. Changes: UNSTABLE .iso and .img files beginning with TrueOS-2017-08-3-x64 will be available to download from http://download.trueos.org/unstable/amd64/. Due to CDN issues, these are not quite available, look for them later today or tomorrow (8/8/17). This update resyncs all ports with FreeBSD as of 8.1.2017. This includes: New/updated FreeBSD Kernel and World & New DRM (Direct Rendering Manager) next. Experimental patch for libhyve-remote: (From htps://github.com/trueos/freebsd/commit/a67a73e49538448629ea27, thanks araujobsd) The libhyve-remote aims to abstract functionalities from other third party libraries like libvncserver, freerdp, and spice to be used in hypervisor implementation. With a basic data structure it is easy to implement any remote desktop protocol without digging into the protocol specification or third part libraries – check some of our examples.We don’t statically link any third party library, instead we use a dynamic linker and load only the functionality necessary to launch the service.Our target is to abstract functionalities from libvncserver, freerdp and spice. Right now, libhyve-remote only supports libvncserver. It is possible to launch a VNC server with different screen resolution as well as with authentication.With this patch we implement support for bhyve to use libhyve-remote that basically abstract some functionalities from libvncserver. We can: Enable wait state, Enable authentication, Enable different resolutions< Have a better compression. Also, we add a new -s flag for vncserver, if the libhyve-remote library is not present in the system, we fallback to bhyve RFB implementation. For example: -s 2,fbuf,tcp=0.0.0.0:5937,w=800,h=600,password=1234567,vncserver,wait New SysAdm Client pages under the System Management category: System Control: This is an interface to browse all the sysctl’s on the system. Devices: This lists all known information about devices on the designated system. Lumina Theming: Lumina is testing new theming functionality! By default (in UNSTABLE), a heavily customized version of the Qt5ct engine is included and enabled. This is intended to allow users to quickly adjust themes/icon packs without needing to log out and back in. This also fixes a bug in Insight with different icons loading for the side and primary windows. Look for more information about this new functionality to be discussed on the Lumina Website. Update to Iridium Web Browser: Iridium is a Chromium based browser built with user privacy and security as the primary concern, but still maintaining the speed and usability of Chromium. It is now up to date – give it a try and let us know what you think (search for iridium-browser in AppCafe). Beastie Bits GhostBSD 11.1 Alpha1 is ready (http://www.ghostbsd.org/11.1-ALPHA1) A Special CharmBUG announcement (https://www.meetup.com/CharmBUG/events/242563414/) Byhve Obfuscation Part 1 of Many (https://github.com/HardenedBSD/hardenedBSD/commit/59eabffdca53275086493836f732f24195f3a91d) New BSDMag is out (https://bsdmag.org/download/bsd-magazine-overriding-libc-functions/) git: kernel - Lower VMMAXUSER_ADDRESS to finalize work-around for Ryzen bug (http://lists.dragonflybsd.org/pipermail/commits/2017-August/626190.html) Ken Thompson corrects one of his biggest regrets (https://twitter.com/_rsc/status/897555509141794817) *** Feedback/Questions Hans - zxfer (http://dpaste.com/2SQYQV2) Harza - Google Summer of Code (http://dpaste.com/2175GEB) tadslot - Microphones, Proprietary software, and feedback (http://dpaste.com/154MY1H) Florian - ZFS/Jail (http://dpaste.com/2V9VFAC) Modifying a ZFS root system to a beadm layout (http://dan.langille.org/2015/03/11/modifying-a-zfs-root-system-to-a-beadm-layout/) ***
208: Faces of Open Source
DragonflyBSD 4.8.1 has been released, we explore how the X11 clipboard works, and look at OpenBSD gaming resources. This episode was brought to you by Headlines LLVM, Clang and compiler-rt support enhancements (https://blog.netbsd.org/tnf/entry/llvm_clang_and_compiler_rt) In the last month I started with upstream of the code for sanitizers: the common layer and ubsan. I worked also on the elimination of unexpected failures in LLVM and Clang. I've managed to achieve, with a pile of local patches, the number of 0 unexpected bugs within LLVM (check-llvm) and 3 unexpected bugs within Clang (check-clang) (however these ones were caused by hardcoded environment -lstdc++ vs -lc++). The number of failures in sanitizers (check-sanitizer) is also low, it's close to zero. LLVM In order to achieve the goals of testability concerning the LLVM projects, I had to prepare a new pkgsrc-wip package called llvm-all-in-one that contains 12 active LLVM projects within one tree. The set of these projects is composed of: llvm, clang, compiler-rt, libcxx, libcxxabi, libunwind, test-suite, openmp, llgo, lld, lldb, clang-tools-extra. These were required to build and execute test-suites in the LLVM's projects. Ideally the tests should work in standalone packages - built out-of-LLVM-sources - and with GCC/Clang, however the real life is less bright and this forced me to use Clang as the system compiler an all-in-one package in order to develop the work environment with the ability to build and execute unit tests. There were four threads within LLVM: Broken std::callonce with libstdc++. This is an old and well-known bug, which was usually worked around with a homegrown implementation llvm::callonce. I've discovered that the llvm::callonce workaround isn't sufficient for the whole LLVM functionality, as std::callonce can be called internally inside the libstdc++ libraries - like within the C++11 futures interface. This bug has been solved by Joerg Sonnenberger in the ELF dynamic linker. Unportable shell construct hardcoded in tests ">&". This has been fixed upstream. LLVM JIT. The LLVM Memory generic allocator (or page mapper) was designed to freely map pages with any combination of the protection bits: R,W,X. This approach breaks on NetBSD with PaX MPROTECT and requires redesign of the interfaces. This is the continuation of the past month AllocateRWX and ReleaseRWX compatibility with NetBSD improvements. I've prepared few variations of local patches addressing these issues and it's still open for discussion with upstream. My personal preference is to remove the current API entirely and introduce a newer one with narrowed down functionality to swap between readable (R--), writable (RW-) and executable (R-X) memory pages. This would effectively enforce W^X. Sanitizers support. Right now, I keep the patches locally in order to upstream the common sanitizer code in compiler-rt. The LLVM JIT API is the last cause of unexpected failures in check-llvm. This breaks MCJIT, ORCJIT and ExecutionEngine libraries and causes around 200 unexpected failures within tests. Clang I've upstreamed a patch that enables ubsan and asan on Clang's frontend for NetBSD/amd64. This support isn't complete, and requires sanitizers' support code upstreamed to compiler-rt. compiler-rt The current compiler-rt tasks can be divided into: upstream sanitizer common code shared with POSIX platforms upstream sanitizer common code shared with Linux and FreeBSD upstream sanitizer common code shared with FreeBSD upstream sanitizer common code specific to NetBSD build, execute and pass tests for sanitizer common code in check-santizer This means that ubsan, asan and the rest of the specific sanitizers wait in queue. All the mentioned tasks are being worked on simultaneously, with a soft goal to finish them one after another from the first to the last one. The last point with check-sanitizer unveiled so far two generic bugs on NetBSD: Return errno EFAULT instead of EACCES on memory fault with read(2)/write(2)-like syscalls. Honor PTHREADDESTRUCTORITERATIONS in libpthread. These bugs are not strictly real bugs, but they were introducing needless differences with other modern POSIX systems. The fixes were introduced by Christos Zoulas and backported to NetBSD-8. Plan for the next milestone I have decided not to open new issues in with the coming month and focus on upstreaming the remaining LLVM code. The roadmap for the next month is to continue working on the goals of the previous months. std::call_once is an example that every delayed bug keeps biting again and again in future. LLVM 5.0.0 is planned to be released this month (August) and there is a joint motivation with the upstream maintainer to push compatibility fixes for LLVM JIT. There is an option to submit a workaround now and introduce refactoring for the trunk and next version (6.0.0). This work was sponsored by The NetBSD Foundation. The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can: http://netbsd.org/donations/#how-to-donate *** DragonFly BSD 4.8.1 released (http://lists.dragonflybsd.org/pipermail/commits/2017-August/626150.html) +Updates by dev: + Antonio Huete Jimenez (1): + libc/gmon: Replace sbrk() with mmap() + Francois Tigeot (3): + drm: bring in Linux compability changes from master + drm/linux: make flushwork() more robust + drm/i915: Update to Linux 4.7.10 + Imre Vadász (4): + drm - Fix hrtimer, don't reset timer->function to NULL in timeout handler. + sound - Delete devfs clone handler for /dev/dsp and /dev/mixer on unload. + ifvtnet - Allocate struct vtnettxheader entries from a queue. + Make sure that cam(4)'s dashutdown handler runs before DEVICESHUTDOWN(). + Matthew Dillon (24): + kernel - MFC b48dd28447fc (sigtramp workaround) + kernel - Fix deadlock in sound system + kernel - Fix broken wakeup in crypto code + kernel - Add KERNPROCSIGTRAMP + gcc - Adjust the unwind code to use the new sigtramp probe sysctl + kernel - Implement NX + kernel - Implement NX (2) + kernel - Implement machdep.pmapnxenable TUNABLE + kernel - Implement NX (3) - cleanup + kernel - Temporarily set the default machdep.pmapnxenable to 0 + param - Change _DragonFlyversion to 400801 + kernel - Fix i915 deadlock + pthreads - Change PTHREADSTACKMIN + libc - Fix bug in rcmdsh() + ppp - Fix minor overflow in protocol search + libtelnet - Fix improper statement construction (not a bug in the binary) + libdevstat - Limit sscanf field, fix redundant condition + openssh - Fix a broken assignment + window - Fix Graphics capability enable test + kernel - Fix event preset + mfiutil - Fix static buffer overflow + mixer - Fix sscanf() overflow + gcore - fix overflow in sscanf + kernel - Fix improper parens + Sascha Wildner (17): + libkvm: Fix char pointer dereference. + Fix some cases where an index was used before its limits check. + Really ensure that our world/kernel are built under POSIX locale ("C"). + zoneinfo: Create a /usr/share/zoneinfo/UTC link. + kernel/cam: Add CAMSCSIITNEXUSLOST (in preparation for virtioscsi(4)). + kernel: Add FreeBSD's virtioscsi(4) driver. + ccdconfig(8): Add missing free(). + libpuffs: Fix two asserts. + kernel/acpi: Untangle the wakecode generation during buildkernel. + kernel/acpica: Better check AcpiOsPredefinedOverride()'s InitVal argument + kernel/acpica: ACPITHREADID is unsigned. + kernel/acpica: Return curthread as thread id from AcpiOsGetThreadId(). + kernel/acpica: Remove no longer needed #include. + kernel/acpi: Call AcpiInitializeSubsystem() before AcpiInitializeTables(). + kernel/urtwn: Add missing braces. + kernel/ieee80211: Add missing braces. + libthreadxu: Fix checking of pthreadbarrierinit()'s count argument. + Sepherosa Ziehau (7): + sound/hda: Sync device ID table with FreeBSD + inet6: Restore mbuf hash after defragmentation. + pf: Normalized, i.e. defragged, packets requiring rehash. + em: Enable MSI by default on devices has PCI advanced features capability. + sched: Change CPU_SETSIZE to signed int, same as FreeBSD/Linux. + usched: Allow process to change self cpu affinity + ix: Fixup TX/RX ring settings for X550, which supports 64/64 TX/RX rings. + zrj (1): + Revert "Always use unix line endings" Porting Unix to the 386: A Practical Approach (http://www.informatica.co.cr/unix-source-code/research/1991/0101.html) The University of California's Berkeley Software Distribution (BSD) has been the catalyst for much of the innovative work done with the UNIX operating system in both the research and commercial sectors. Encompassing over 150 Mbytes (and growing) of cutting-edge operating systems, networking, and applications software, BSD is a fully functional and nonproprietary complete operating systems software distribution (see Figure 1). In fact, every version of UNIX available from every vendor contains at least some Berkeley UNIX code, particularly in the areas of filesystems and networking technologies. However, unless one could pay the high cost of site licenses and equipment, access to this software was simply not within the means of most individual programmers and smaller research groups. The 386BSD project was established in the summer of 1989 for the specific purpose of porting BSD to the Intel 80386 microprocessor platform so that the tools this software offers can be made available to any programmer or research group with a 386 PC. In coordination with the Computer Systems Research Group (CSRG) at the University of California at Berkeley, we successively ported a basic research system to a common AT class machine (see, Figure 2), with the result that approximately 65 percent of all 32-bit systems could immediately make use of this new definition of UNIX. We have been refining and improving this base port ever since. By providing the base 386BSD port to CSRG, our hope is to foster new interest in Berkeley UNIX technology and to speed its acceptance and use worldwide. We hope to see those interested in this technology build on it in both commercial and noncommercial ventures. In this and following articles, we will examine the key aspects of software, strategy, and experience that encompassed a project of this magnitude. We intend to explore the process of the 386BSD port, while learning to effectively exploit features of the 386 architecture for use with an advanced operating system. We also intend to outline some of the tradeoffs in implementation goals which must be periodically reexamined. Finally, we will highlight extensions which remain for future work, perhaps to be done by some of you reading this article today. Note that we are assuming familiarity with UNIX, its concepts and structures, and the basic functions of the 386, so we will not present exhaustive coverage of these areas. In this installment, we discuss the beginning of our project and the initial framework that guided our efforts, in particular, the development of the 386BSD specification. Future articles will address specific topics of interest and actual nonproprietary code fragments used in 386BSD. Among the future areas to be covered are: 386BSD process context switching Executing the first 386BSD process on the PC 386BSD kernel interrupt and exception handling 386BSD INTERNET networking ISA device drivers and system support 386BSD bootstrap process *** X11: How does “the” clipboard work (https://www.uninformativ.de/blog/postings/2017-04-02/0/POSTING-en.html) > If you have used another operating system before you switched to something that runs X11, you will have noticed that there is more than one clipboard: > Sometimes, you can use the mouse to select some text, switch to another window, and then hit the middle mouse button to paste text. > Sometimes, you can select text, then hit some hotkey, e.g. Ctrl+C, switch to another window, hit another hotkey, e.g. Ctrl+V, and paste said text. > Sometimes, you can do both. > Selections as a form of IPC First things first, in X11 land, “clipboards” are called “selections”. Yes, there is more than one selection and they all work independently. In fact, you can use as many selections as you wish. In theory, that is. When using selections, you make different clients communicate with each other. This means that those clients have to agree on which selections to use. You can’t just invent your own selection and then expect Firefox to be compatible with it. How are selections identified? There are three “standard” selection names: PRIMARY: The “middle mouse clipboard” SECONDARY: Virtually unused these days CLIPBOARD: The “Ctrl+C clipboard” Program 1: Query selection owners Content type and conversion Program 2: Get clipboard as UTF-8 Program 3: Owning a selection Program 4: Content type TARGETS Handling binary data using xclip Large amounts of data Clipboard managers Summary News Roundup TrueOS Documentation: A great way to give back! (https://www.trueos.org/blog/trueos-documentation-great-way-give-back/) The TrueOS project is always looking for community contribution. Documentation changes are a great way for users to not only make a solid contribution to the project, but learn more about it too! Over the last few months, many users have asked for both simple and detailed instructions on making documentation changes. These are now added to the TrueOS handbook in the Contributing to TrueOS section. If interested in making a small alteration to the TrueOS handbook, here are some instructions for submitting a patch through the GitHub website. These instructions are also applicable to the Lumina and SysAdm handbooks. Lumina documentation is in the the lumina-docs repository, and SysAdm guides are in sysadm-docs. Make a Doc change! A GitHub account is required to submit patches to the TrueOS docs. Open a web browser and sign in to GitHub or make a new account. When making a new account, be sure to use an often checked email address, as all communication regarding patches and pull requests are sent to this address. Navigate to the trueos-docs GitHub repository. Click on the trueos-handbook directory to view all the documentation files. Open the .rst file corresponding to the chapter needing an update. The chapter names are reflected in the title of the .rst files. For example, open install.rst to fix an error spotted in handbook chapter 3: “Install”. This first image shows the trueos-docs repository and the contents of the trueos-handbook directory Open the desired chapter file by clicking its entry in the list. The trueos.rst file is an index file and should be ignored. Begin editing the file by clicking the Pencil icon in the upper right corner above the file’s text. The file moves to edit mode, where it is now possible to make changes, as the next image shows. Editing install.rst with GitHub When making a simple change, it is recommended to avoid adjusting the specific formatting elements and instead work within or around them. Once satisfied, scroll to the bottom of the page and write a detailed commit summary of the new changes. Click Propose file change (green button), then Create pull request to submit the changes to the project. GitHub then does an automated merge check. Click Create pull request again to submit the change to the repository. In the final step, a developer or project committer reviews the changes, merging them into the project or asking for more changes as necessary. Learn more about TrueOS documentation To learn more about the underlying structure of TrueOS documentation like the Sphinx Documentation Generator and reStructuredText markup, browse the Advanced Documentation Changes section of the TrueOS handbook. This section also contains instructions for forking the repository and configuring a local clone, build testing, updating the translation files, and other useful information. The Sphinx website is also a valuable resource. libHijack Revival (https://www.soldierx.com/news/Hijack-Revival) Over a decade ago, while standing naked and vulnerable in the comfort of my steaming hot shower, I gathered my thoughts as humans typically attempt to do in the wee hours of the morning. Thoughts of a post-exploitation exercise raced in my mind, the same thoughts that made sleeping the night before difficult. If only I could inject into Apache some code that would allow me to hook into its parsing engine without requiring persistance. Putting a file-backed entry into /proc/pid/maps would tip off the security team to a compromise. The end-goal was to be able to send Apache a special string and have Apache perform a unique action based on the special string. FelineMenace's Binary Protection Schemes whitepaper provided inspiration. Silvio Cesare paved the way into PLT/GOT redirection attacks. Various Phrack articles selflessly contributed to the direction I was to head. Alas, in the aforementioned shower, an epiphany struck me. I jumped as an awkward stereotypical geek does: like an elaborate Elaine Benes dance rehearsal in the air. If I used PTrace, ELF, and the PLT/GOT to my advantage, I could cause the victim application to allocate anonymous memory mappings arbitrarily. In the newly-created memory mapping, I could inject arbitrary code. Since a typical operating system treats debuggers as God-like applications, the memory mapping could be mapped without write access, but as read and execute only. Thus enabling the stealth that I sought. The project took a few years to develop in my spare time. I ended up creating several iterations, taking a rough draft/Proof-of-Concept style code and rewriting it to be more efficient and effective. I had toyed with FreeBSD off-and-on for over a decade by this point, but by-and-large I was still mostly using Linux. FreeBSD gained DTrace and ZFS support, winning me over from the Linux camp. I ported libhijack to FreeBSD, giving it support for both Linux and FreeBSD simultaneously. In 2013, I started work on helping Oliver Pinter with his ASLR implementation, which was originally destined to be upstreamed to FreeBSD. It took a lot of work, and my interest in libhijack faded. As a natural consequence, I handed libhijack over to SoldierX, asking the community to take it and enhance it. Over four years went by without a single commit. The project was essentially abandoned. My little baby was dead. This past week, I wondered if libhijack could even compile on FreeBSD anymore. Given that four years have passed by and major changes have happened in those four years, I thought libhijack would need a major overhaul just to compile, let alone function. Imagine my surprise when libhijack needed only a few fixups to account for changes in FreeBSD's RTLD. Today, I'm announcing the revival of libhijack. No longer is it dead, but very much alive. In order to develop the project faster, I've decided to remove support for Linux, focusing instead on FreeBSD. I've removed hundreds of lines of code over the past few days. Supporting both FreeBSD and Linux meant some code had to be ugly. Now the beautification process has begun. I'm announcing the availability of libhijack 0.7.0 today. The ABI and API should be considered unstable as they may change without notice. Note that HardenedBSD fully mitigates libhijack from working with two security features: setting security.bsd.unprivilegedprocdebug to 0 by default and the implementation of PaX NOEXEC. The security.bsd.unprivilegedprocdebug sysctl node prevents PTrace access for applications the debugger itself did not fork+execve for unprivileged (non-root) users. Privileged users (the root account) can use PTrace to its fullest extent. HardenedBSD's implementation of PaX NOEXEC prevents the creation of memory mappings that are both writable and executable. It also prevents using mprotect to toggle between writable and executable. In libhijack's case, FreeBSD grants libhijack the ability to write to memory mappings that are not marked writable. Debuggers do this to set breakpoints. HardenedBSD behaves differently due to PaX NOEXEC. Each memory mapping has a notion of a maximum protection level. When a memory mapping is created, if the write bit is set, then HardenedBSD drops the execute bit from the maximum protection level. When the execute bit is set at memory mapping creation time, then the write bit is dropped from the maximum protection level. If both the write and execute bits are set, then the execute bit is silently dropped from both the mapping creation request and the maximum protection level. The maximum protection level is always obeyed, even for debuggers. Thus we see that PaX NOEXEC is 100% effective in preventing libhijack from injecting code into a process. Here is a screenshot showing PaX NOEXEC preventing libhijack from injecting shellcode into a newly-created memory mapping. What's next for libhijack? Here's what we have planned, in no particular order: Python bindings Port to arm64 This requires logic for handling machine-dependent code. High priority. Finish anonymous shared object injection. This requires implementing a custom RTLD from within libhijack. More cleanups. Adhere to style(9). libhijack can be found on GitHub @ https://github.com/SoldierX/libhijack *** Contributing to FreeBSD (https://blather.michaelwlucas.com/archives/2988) I’ve talked to a whole bunch of folks who say things like “I’m a junior programmer. I’m looking for a way to help. I have no specific expertise, but I’m willing to learn.” Today, I present such junior programmers with an opportunity. An opportunity for you to learn skills that will be incredibly valuable to your career, and will simultaneously expand your career opportunities. For decades, FreeBSD has relied on its users for testing. They expect users to install pre-release versions of the OS and exercise them to identify regressions. That’s necessary, but it’s nowhere near enough. The FreeBSD Testing Project is building an automated test suite for the entire operating system. They have a whole mess of work to do. There’s only four people on the team, so each additional person that contributes can have a serious impact. They have tutorials on how to write tests, and sample tests. There’s a whole bunch of tests left to be written. You have an almost open field. They need tests for everything from ls(1) to bhyve. (Yes, ls(1) broke at one point in the last few years.) Everything needs testing. Learning to write, submit, and commit small tests is valuable experience for developing the big tests. What’s more, learning to write tests for a system means learning the system. Developing tests will transform you into a FreeBSD expert. Once you’ve demonstrated your competence, worth, and ability to work within the project, other FreeBSD teams will solicit your help and advice. The Project will suck you in. Testing is perhaps the most valuable contribution anyone can make to an open source project. And this door into the FreeBSD Project is standing wide, wide open. OpenBSD Gaming Resource (https://mrsatterly.com/openbsd_games.html) > What isn't there to love about playing video games on your favorite operating system? OpenBSD and video games feels like a natural combination to me. My resource has software lists, links to free games not in ports, lists of nonfree games, and recommendations. The Table of Contents has these high-level items for you: > General Resources > OpenBSD Exclusive > Ports > Network Clients > Browser Games > Game Engines > Multiple Game Engines > Multiple System Emulation > Computer Emulation > Game Console Emulation > Live Media Emulation > Operating System Emulation > Games in Other Software Have fun with these games! *** Beastie Bits Dragonfly introduces kcollect(8) (https://www.dragonflydigest.com/2017/08/07/20061.html) The Faces of Open Source (http://facesofopensource.com/unix/) Edgemesh CEO, Jake Loveless and Joyent CTO, Bryan Cantrill join together for a fireside chat to discuss distributed caching at scale, Docker, Node.js, Mystery Science Theater 3000, and more! (https://www.joyent.com/blog/joyent-edgemesh-cache-me-if-you-can) UFS: Place the information needed to find alternate superblocks to the end of the area reserved for the boot block (https://svnweb.freebsd.org/base?view=revision&revision=322297) Let ‘localhost’ be localhost (https://tools.ietf.org/html/draft-west-let-localhost-be-localhost-04) Hurry up and register for vBSDCon September 7-9 (http://www.verisign.com/en_US/internet-technology-news/verisign-events/vbsdcon/index.xhtml?dmn=vBSDcon.com) and EuroBSDCon September 21-24 (https://2017.eurobsdcon.org/) *** Feedback/Questions Morgan - btrfs deprecated (http://dpaste.com/0JEYE1K) Ben - UEFI, GELI, BEADM, and more (http://dpaste.com/2TP90HD) Brad - Hostname Clarification (http://dpaste.com/1MQH1BD) M Rod - BSD Laptop (http://dpaste.com/39C6PGN) Jeremy - Contributing to BSDs (http://dpaste.com/3SVP5SF) ***
207: Bridge over the river Cam
We recap our devsummit experiences at BSDCambridge, share why memcmp is more complicated than expected, explore Docker on FreeBSD, and we look at a retro terminal. This episode was brought to you by Headlines BSDCam recap (https://wiki.freebsd.org/DevSummit/201708) The 2017 Cambridge DevSummit took place from 2-4 August 2017. The event took place over three days including a formal dinner at St John's College, and was attended by 55 registered developers and guests. Prior to the start of the conference, we had a doc hacking lounge, the computer lab provided a room where we could meet and try to spend some time on documentation. Sevan walked two interested people through the process of creating a documentation patch and submitting it for the first time. In the process, found ways to improve the documentation on how to write documentation. The event is run "un-conference style" in that we brainstorm the actual session schedule on the first morning, with a focus on interactive topics that reflect the interests and exploit the knowledge of the attendees. The idea is to maximize the amount of discussion and decisions that can be made while we are all in the same room The first morning, we all gather in the slightly too small, and even more slightly under air conditioned FW11 classroom. We go around the room introducing ourselves, and listing a few topics we would be interested in discussing. Eventually the whiteboard is full of topics, with various numbers of ticks beside them to indicate the number of interested people There are breakout rooms of all sizes, so even topics with only a small group of interested folks can get a lot accomplished The most difficult is trying to schedule the sessions, as there is much overlap and people usually want to be in concurrent sessions, or someone's schedule means they won’t be available that day, etc. This years working groups: Toolchain (Compilers, Linkers, External Toolchain, Static analysis and sanitizers) Virtualization (bhyve, xen, jails, docker) Transport (TCP) and Network Performance Security and mitigations (W^X, noexec stack, CFI, ASLR, KASLR, Safe Stack, etc) Testing (Status, What to test, How to test, QA for releases) Capsicum (Automation with LLVM etc, Casper, Namespacing, “Services”, capsh) Desktop / WiFi (drm-next, drivers, resume, power, installer, desktop, OOB Experience) Tracing (Blackbox, DTrace, KTR, ptrace, truss, hardware tracing) Packaging and Packaged Base (Sets, Kernels, Ports & flavours, sub-packages, privlib) Architectural Security Features (CPU Features: SGX, PXN/PAN, Pointer Authentication, AMD Memory Encryption, Libcrunch, RISC-V, CheriABI) Architectures and Embedded systems (RISC-V, ARM, ARM64, MIPS(64), SPARC64) Teaching (Audiences, Objectives, Targets, Material, future directions) Provisioning and Management Tools (CfgMgmt tools, Image building, VM/bhyve orchestration, Preconfigured VMs for testing, Wishlist) Storage (ZFS status update, ZFS encryption infrastructure, ZFS Zero Copy / Sendfile, Acceleration of checksums and raidz parity calculations, sesutil, mpsutil) And that wasn’t everything. We then had a series of short talklets: Enhancing and replacing mmap() SDIO support eBPF support for FreeBSD Tracing + Virtualization Practical DMA Attack Protection On Thursday night there was a special dinner at St John's College Overall it was a great DevSummit, and I even managed to get some of the work assigned to me finished. Shortly I will commit an update to the boot loader menu that will automatically populate the kernel selection menu with the automatically detected list of installed kernels. The list is also properly refreshed when you switch boot environments. *** Hosts/BSD – for when you need to run your BSD inside a penguin (https://wiki.qemu.org/index.php/Hosts/BSD) This wiki provides details on how to run each of the various BSDs under QEMU The target audience is Linux developers looking to test their apps etc under BSD The wiki is in need of some love, there are some option questions, and it lacks some polish There are instructions on building qemu from source, but it should likely mention the qemu-devel port There should probably also be instructions on using other architectures, like ARM/MIPS etc If you have used QEMU, or would like to spend the time to learn how, please help update this wiki *** memcmp -- more complicated than you might expect (http://trust-in-soft.com/memcmp-requires-pointers-to-fully-valid-buffers/) “A suspicious pattern in open-source software” One bug recently found by John using tis-interpreter on a widely used open-source library involved the comparison of strings with memcmp. The unexpected condition was that memcmp was, in one case, called with a pointer to a buffer shorter than the length passed as third argument, breaking one of the two symmetrical pre-conditions in the function’s ACSL contract A reason that may have made this use of memcmp look okay to the developer is that the buffers being passed to it always differed before the end of the buffers were reached. a memcmp implementation based on stopping as soon as a difference is found, would not have caused any out-of-bounds read access The first question raised was whether the pattern memcmp("a", "bc", 3) was problematic according to the letter of the C standard. If it was, the second question was whether the busy maintainer of one of the Open Source packages that make the Internet tick should be bothered with a bug report. I would like to be able to say that memcmp’s ACSL contract was the product of careful deliberation, but unfortunately this is not the case: many standard function contracts were written quickly in order to get most of the standard library covered, and have not been tested by time. Anyway, upon proofreading the relevant clause in the C11 standard, my feeling was that the ACSL formalization was, in this particular case, right, and that it was undefined behavior to pass as memcmp argument a buffer that wasn’t fully valid, even if the implementation sort-of needs to read the buffer’s characters in order for the purpose of finding the first mismatch. The post then goes on to look at the memcmp code in glibc There are two distinct optimizations for long buffers, one that applies when both buffers start at the same offset modulo the word size, memcmpcommonalignment, and one that applies when they don’t, memcmpnotcommonalignment. The function memcmpcommonalignment is relatively well-behaved: it reads from the two buffers aligned word by aligned word, and thus reads the entire words that contain differing bytes. If the caller passed buffers that aren’t valid after the differing byte, this amounts to reading out of bounds, but this sort of out-of-bounds access is not detected by the typical MMU, which works at the scale of the page. The “notcommon_alignment” case, however, tells a different story. When passed the carefully (mis-)aligned buffers t1 and (char*)t2+1, although these buffers differ in the 8th byte, Glibc’s implementation of memcmp reads 8 bytes beyond the end of t1. By making the 16th byte differ instead of the 8th one, it is also possible to make Glibc’s implementation of memcmp read 16 bytes beyond the end of t1. In conclusion, yes, some implementations of memcmp will crash when invoked with buffers that aren’t valid for the full length, even if they differ early. The circumstances are rare (probably the reason this bug was still there to be found in a library that had already been tested with all the available techniques) but outside the programmer’s control. The pattern described in this post should be reported as a bug when found. It is interesting to read the detailed analysis of a bug in such a basic libc feature *** News Roundup Docker on FreeBSD (http://daemon-notes.com/articles/network/docker) There are two approaches to running Docker on FreeBSD. First one was created back in 2015 and it was a native port of Docker engine to FreeBSD. It was an ambitious project but nobody stepped forward to continuously port the never-ending flow of upstream code to FreeBSD. So the port still exists (sysutils/docker-freebsd) but it wasn't updated since 2015 and it is Docker v1 (it is v17 as of 2017). The other approach is to use official way of running Docker on platforms other than Linux. Well, somewhat official as Docker still does not support FreeBSD as a host officially. This is docker-machine tool which in turn will use VirtualBox to run a virtual machine with Linux and Docker engine. docker utility on the host will communicate with the engine inside VB where all the work will be done. This article describes what needs to be done to start using it. Before we begin you need VirtualBox installed. Do not skip adding /boot/loader.conf and /etc/rc.conf lines mentioned on that page. You won't need user inteface or anything, docker-machine will do all the work, just make sure VirtualBox is present and ready to be used. `pkg install docker docker-machine docker-compose’ Docker will store its stuff in ~/.docker. You might not want the virtual machine image files to live in your home, in this case just create a symlink: mkdir ~/.docker ln -s /storage/docker ~/.docker/machine docker-machine create --driver virtualbox \ --virtualbox-memory 2048 \ --virtualbox-cpu-count 2 \ --virtualbox-disk-size 102400 \ --virtualbox-hostonly-cidr "10.2.1.1/24" \ docker1 Here's the example. We are creating machine named docker1. It is using VirtualBox driver, the vm has 2G of memory, 2 cores and 100G of disk space. docker-machine setups VirtualBox to use host-only network adapter (it will create vboxnet0 interface on the host automatically) and we are instructing it to use 10.2.1.1/24 as the address of this adapter — change it to what suits your needs or omit this flag (default is 192.168.99.1/24). And basically that is all. Check if it is running: docker-machine ls If you do open VirtualBox interface you will find a virtual machine named docker1 running. You can start/stop/whatever your machine using docker-machine utility. Here’s how you can connect to the machine: docker utility by default tries to talk to Docker engine running on the same host. However with specific environment variables you can instruct it to talk to other host. docker-machine can export these variables for you. eval docker-machine env docker1 docker run hello-world There was quite a bit of discussion about docker at the FreeBSD developers summit in Cambridge during the first week of August. Two docker developers who had worked on the Mac OS X port, one of whom is an OpenBSD advocate, explained how docker has evolved, and the linux-isms have been abstracted away such that a truly native docker solution for FreeBSD can be built and maintained with a lot less headache than before I look forward to seeing if we can’t make that happen *** The POSIX Shell And Utilities (http://shellhaters.org/) The POSIX Shell And Utilities Compiled for The Shell Hater's Handbook *** PostgreSQL – logging to a file (http://dan.langille.org/2017/07/31/postgresql-logging-to-a-file/) These steps were carried out on FreeBSD 11.0 with PostgreSQL 9.6 (two of my favorite tools). I like logging. I like logging PostgreSQL. With logs, you can see what happened. Without, you can only guess. Setting up logging for PostgreSQL involves several parts, each of which must be completed or else I don’t get what I want. This is not a criticism of PostgreSQL. It’s a feature. I am documenting this because each time I configure a new PostgreSQL instance, it takes me more than one iteration to get it working. The goal: this post lets both you and me get it right the first time. The parts include: + Telling PostgreSQL to log via syslog + Telling FreeBSD to local postgres to /var/log/postgres.log (my preference). + Telling PostgreSQL the things you want logged. + Changes to postgresql.conf The file location varies with the version installed. For PostgreSQL 9.6 on FreeBSD, the file is /var/db/postgres/data96/postgresql.conf (adjust 96 according to the version installed). I made these changes to that file. log_destination = 'syslog' log_min_messages = notice log_min_error_statement = notice log_checkpoints = on log_lock_waits = on log_timezone = 'UTC' By default, PostgreSQL logs to the local0 facility and is controlled by the syslog_facility in postgresql.conf. This will be used in syslog.conf (see the next section of this post). The above mentioned changes require a reload: service postgresql reload Changes to /etc/syslog.conf Now that we have PostgreSQL logging to syslog, we want to tell syslog where to put those messages. I changed this line in /etc/syslog.conf:*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages With .notice pulling in some local0 messages, adding local0.none to the line will free the messages up for later use in the configuration file. Otherwise, the PostgreSQL messages will be in /var/log/messages. The changed line is: `.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err;local0.none /var/log/messages Then, to get the messages into my preferred location, I added this to the file: local0.* /var/log/postgresql.log` Log file rotation For rotating my log file, I added a new file: /usr/local/etc/newsyslog.conf.d/postgresql96 /var/log/postgresql.log pgsql:wheel 640 7 * $D0 GB /var/db/postgres/data96/postmaster.pid 30 Before restarting syslog, I did this, so the destination file existed. This isn’t always/strictly necessary, but because the ownership is not chown root:wheel, I do it to get that part set. touch /var/log/postgresql.log chown pgsql:wheel Restarting syslog: sudo kill -HUP `sudo cat /var/run/syslog.pid ` That’s it Now you should see PostgreSQL logging in /var/log/postgresql.log. mandoc-1.14.2 released (http://undeadly.org/cgi?action=article&sid=20170729122350) i just released portable mandoc-1.14.2. It is available now from http://mandoc.bsd.lv/ (http://mandoc.bsd.lv/). ```From: Ingo Schwarze schwarze@usta.de Date: Fri, 28 Jul 2017 20:12:44 +0200 To: discuss@mandoc.bsd.lv Subject: mandoc-1.14.2 released Hi, i just released portable mandoc-1.14.2. It is available now from http://mandoc.bsd.lv/ . All downstream maintainers are encouraged to update their ports and packages from 1.14.1 to 1.14.2. Mandoc 1.14.2 is a feature release introducing: a new -Tmarkdown output mode anchors for deep linking into -Thtml manual pages a superset of the functionality of the former mdoclint(1) utility a new -Wstyle message level with several new messages automatic line breaking inside individual tbl(7) cells a rewrite of the eqn(7) lexer, and some eqn(7) rendering improvements support for many additional low-level roff(7) features and various smaller features and bug fixes. For more details, see: http://mandoc.bsd.lv/NEWS With the improved mandoc features, only twenty-five out of the ten thousand software packages in the OpenBSD ports tree still need groff to format their manual pages. Since the project has been called "mandoc" rather than "mdocml" for several years now, the website, the distribution tarball, and the source extraction directory are now also called "mandoc" rather than "mdocml". The release was tested on the following systems: + OpenBSD-current and OpenBSD-stable + NetBSD-current + illumos + Debian Linux + Void Linux x86_64 glibc and musl + Crux Linux + SunOS 5.11.2, 5.10, and 5.9 As before, catman(8) and the regression suite cannot be used on SunOS 5.10 and SunOS 5.9. A big thanks to everybody who provided patches, bug reports, feature suggestions, advice, and help with testing! Yours, Ingo``` Beastie Bits A good looking terminal emulator which mimics the old cathode display. Available in x11/cool-retro-terminal (https://github.com/Swordfish90/cool-retro-term) Milestone Complete! OpenRC conversion (https://www.trueos.org/blog/milestone-complete-openrc-conversion/) Healthy developer interaction between FreeBSD and IllumOS re: mdb (https://illumos.topicbox.com/groups/developer/discussions/T5eae6079331c4df4) Large Batch of Kernel Errata Patches Released (http://undeadly.org/cgi?action=article&sid=20170804053102) opnsense 17.7 released (https://opnsense.org/opnsense-17-7-released/) Twitter Co-Founder and CEO states “FreeBSD rules them all” (https://twitter.com/jack/status/892605692317650944) Hurry up and register for vBSDCon September 7-9 (http://www.verisign.com/en_US/internet-technology-news/verisign-events/vbsdcon/index.xhtml?dmn=vBSDcon.com) and EuroBSDCon September 21-24 (https://2017.eurobsdcon.org/) *** Feedback/Questions Dominik - Monitoring Software (http://dpaste.com/08971FQ) Darren - Wonderful Awk (http://dpaste.com/0YCS4DN) Andrew - Thanks (http://dpaste.com/0ZREKTV) Jens - Migration Questions (http://dpaste.com/1GVZNWN) ***
206: To hier is UNIX
Lumina Desktop 1.3 is out, we show you a Plasma 5 on FreeBSD tutorial, explore randomness, and more. This episode was brought to you by Headlines Lumina Desktop v1.3 released (https://lumina-desktop.org/version-1-3-0-released/) Notable Changes: New Utility: lumina-mediaplayer. Lumina Media Player is a graphic interface for the Qt QMediaPlayer Class, with Pandora internet radio streaming integration. Lumina Media Player supports many audio formats, including .ogg, .mp3, .mp4, .flac, and .wmv. It is also possible to increase the number of playable formats by installing gstreamer-plugins. This utility is found in the Applications → Utilities section, or opened by typing lumina-mediaplayer in a command line. New Utility: lumina-xdg-entry. This is another simple utility designed to help users create .desktop entries and shortcuts. Find it in the Utilities application category, or open it by typing lumina-xdg-entry in a command line. Lumina Desktop: Desktop folders are integrated, and can now be manipulated directly from the desktop. Added the automatic settings migration of a desktop monitor (single monitor only, for now). Numerous speed and performance improvements with how icons load and the system interacts with the desktop. Lumina-FM: Now fully integrated with lumina-archiver. A “System directory” tree pane is available. Options to enable/disable it are being added later, as it is on by default. Numerous speed improvements with caching and loading icons. Lumina Texteditor: There is a new json manifest file format for syntax highlighting support. Users can open this file, customize their highlighting options, and immediately see their changes without rebuilding the utility. The text editor now supports more than 10 different file formats. Added options for file-wide preferences in syntax files. Options include: word wrap, character per line limits, excess whitespace highlighting, font style restrictions, and tab-width settings. LTE supports tabs with detach, drag’n’drop, and location customization with the View menu option. Add checkable menu option to show the “unsaved changes” dialogue box on close. Lumina Screenshot: Adjustments to the lumina-screenshot interface. Add an adjustable warning to lumina-screenshot when closing with an unsaved image. Add functionality to select a specific area of the screen for screenshots. Lumina Archiver: Functionality improvements. Bug fixes. Interface changes. General Improvements: Permission checks for settings files (all utilities). When launched with sudo, all tools use or create a root-permissioned copy of the user’s settings file. This prevents a settings file being locked by root. UI text reworks to help re-unify style. Add hooks to update the desktop with icons for the /media directory when a system uses USB automounting functionality. Fix Fluxbox bug with windows workspace assignments. Work on new utility lumina-notify (not fully functional yet). Fix panel reporting error crashing lumina-config. Bug fix for dbus-send calls for Gentoo. Clean up automatic DPI scaling support. Bug fix for the panel clock. Compton compositor is now disabled by default (but can be manually enabled). Translation file updates. Documentation updates. *** FreeBSD 11.0 and Plasma 5 HowTo (https://euroquis.nl/bobulate/?p=1609) Here’s a step-by-step guide to getting a machine with FreeBSD 11 in it, running X, and KDE Plasma 5 Desktop and KDE Applications. It’s the latest thing! (Except that 11-STABLE is in the middle of the pack of what’s supported .. but the KDE bits are fresh. I run 10.3 with KDE4 or Plasma 5 on my physical machines, myself, so the FreeBSD version isn’t that important except that packages are readily available for 11-STABLE, not for 10-STABLE.) We skip the part about installing FreeBSD (it’s in there if you need it) and get right to the important parts that you need: An X Server and a backup X11 environment (ancient): pkg install xorg xterm twm Desktop technologies (modern): pkg install hal dbus echo haldenable=YES >> /etc/rc.conf echo dbusenable=YES >> /etc/rc.conf Next up, test whether the X server works by running startx and exiting out of twm. If running with ZFS, it’s a good idea to snapshot now, just so you can easily roll back to the it-works-with-basic-X11 setup you have now. zfs snapshot -r zroot@x11 Now swap out the default FreeBSD package repository, for the KDE-FreeBSD community one. This is documented also on the Area51 page (https://community.kde.org/FreeBSD/Setup/Area51). mkdir -p /usr/local/etc/pkg/repos cd /usr/local/etc/pkg/repos cat > FreeBSD.conf < Area51.conf <> /boot/loader.conf echo webcamdenable=YES >> /etc/rc.conf Log in as your test user, and set up .xinitrc to start Plasma 5: cat > .xinitrc <
205: FreeBSD Turning it up to 11.1
FreeBSD 11.1-RELEASE is out, we look at building at BSD home router, how to be your own OpenBSD VPN provider, and find that glob matching can be simple and fast. This episode was brought to you by Headlines FreeBSD 11.1-RELEASE (https://www.freebsd.org/releases/11.1R/relnotes.html) FreeBSD 11.1 was released on July 26th (https://www.freebsd.org/releases/11.1R/announce.asc) You can download it as an ISO or USB image, a prebuilt VM Image (vmdk, vhd, qcow2, or raw), and it is available as a cloud image (Amazon EC2, Microsoft Azure, Google Compute Engine, Vagrant) Thanks to everyone, including the release engineering team who put so much time and effort into managing this release and making sure it came out on schedule, all of the FreeBSD developers who contributed the features, the companies that sponsored that development, and the users who tested the betas and release candidates. Support for blacklistd(8) has been added to OpenSSH The cron(8) utility has been updated to add support for including files within /etc/cron.d and /usr/local/etc/cron.d by default. The syslogd(8) utility has been updated to add the include keyword which allows specifying a directory containing configuration files to be included in addition to syslog.conf(5). The default syslog.conf(5) has been updated to include /etc/syslog.d and /usr/local/etc/syslog.d by default. The zfsbootcfg(8) utility has been added, providing one-time boot.config(5)-style options The efivar(8) utility has been added, providing an interface to manage UEFI variables. The ipsec and tcpmd5 kernel modules have been added, these can now be loaded without having to recompile the kernel A number of new IPFW modules including Network Prefix Translation for IPv6 as defined in RFC 6296, stateless and stateful NAT64, and a module to modify the TCP-MSS of packets A huge array of driver updates and additions The NFS client now supports the Amazon® Elastic File System™ (EFS) The new ZFS Compressed ARC feature was added, and is enabled by default The EFI loader has been updated to support TFTPFS, providing netboot support without requiring an NFS server For a complete list of new features and known problems, please see the online release notes and errata list, available at: FreeBSD 11.1-RELEASE Release Notes (https://www.freebsd.org/releases/11.1R/relnotes.html) FreeBSD 11.1-RELEASE Errata (https://www.freebsd.org/releases/11.1R/errata.html) For more information about FreeBSD release engineering activities, please see: Release Engineering Information (https://www.freebsd.org/releng/) Availability FreeBSD 11.1-RELEASE is now available for the amd64, i386, powerpc, powerpc64, sparc64, armv6, and aarch64 architectures. FreeBSD 11.1-RELEASE can be installed from bootable ISO images or over the network. Some architectures also support installing from a USB memory stick. The required files can be downloaded as described in the section below. SHA512 and SHA256 hashes for the release ISO, memory stick, and SD card images are included at the bottom of this message. PGP-signed checksums for the release images are also available at: FreeBSD 11.1 Release Checksum Signatures (https://www.freebsd.org/releases/11.1R/signatures.html) A PGP-signed version of this announcement is available at: FreeBSD 11.1-RELEASE Announcement (https://www.FreeBSD.org/releases/11.1R/announce.asc) *** Building a BSD home router - ZFS and Jails (https://eerielinux.wordpress.com/2017/07/15/building-a-bsd-home-router-pt-8-zfs-and-jails/) Part of a series of posts about building a router: Part 1 (https://eerielinux.wordpress.com/2017/05/30/building-a-bsd-home-router-pt-1-hardware-pc-engines-apu2/) -- discussing why you want to build your own router and how to assemble the APU2 Part 2 (https://eerielinux.wordpress.com/2017/06/03/building-a-bsd-home-router-pt-2-the-serial-console-excursion) -- some Unix history explanation of what a serial console is Part 3 (https://eerielinux.wordpress.com/2017/06/10/building-a-bsd-home-router-pt-3-serial-access-and-flashing-the-firmware/) -- demonstrating serial access to the APU and covering firmware update Part 4 (https://eerielinux.wordpress.com/2017/06/15/building-a-bsd-home-router-pt-4-installing-pfsense/) -- installing pfSense Part 5 (https://eerielinux.wordpress.com/2017/06/20/building-a-bsd-home-router-pt-5-installing-opnsense/) -- installing OPNsense instead Part 6 (https://eerielinux.wordpress.com/2017/06/30/building-a-bsd-home-router-pt-7-advanced-opnsense-setup/) -- Comparison of pfSense and OPNsense Part 7 (https://eerielinux.wordpress.com/2017/06/30/building-a-bsd-home-router-pt-7-advanced-opnsense-installation/) -- Advanced installation of OPNsense After the advanced installation in part 7, the tutorials covers converting an unused partition into swap space, and converting the system to ZFS After creating a new pool using the set aside partition, some datasets are created, and the log files, ports, and obj ZFS datasets are mounted The tutorial then goes on to cover how to download the ports tree, and install additional software on the router I wonder what part 9 will be about. *** Be your own VPN provider with OpenBSD (v2) (https://networkfilter.blogspot.com/2017/04/be-your-own-vpn-provider-with-openbsd-v2.htm) This article covers how to build your own VPN server with some advanced features including: Full Disk Encryption (FDE) Separate CA/signing machine (optional) Multiple DNSCrypt proxy instances for failover OpenVPN: Certificate Revocation List/CRL (optional) OpenVPN: TLS 1.2 only OpenVPN: TLS cipher based on AES-256-GCM only OpenVPN: HMAC-SHA512 instead of HMAC-SHA1 OpenVPN: TLS encryption of control channel (makes it harder to identify OpenVPN traffic) The article starts with an explanation of the differences between OpenVPN and IPSEC. In the end the author chose OpenVPN because you can select the port it runs on, and it has a better chance of working from hotel or coffee shop WiFi. The guide them walks through doing an installation on an encrypted disk, with a caution about the limitations of encrypted disk with virtual machines hosted by other parties. The guide then locks down the newly installed system, configuring SSH for keys only, adding some PF rules, and configuring doas Then networking is configured, including enabling IP forwarding since this machine is going to act as the VPN gateway Then a large set of firewall rules are created that NAT the VPN traffic out of the gateway, except for DNS requests that are redirected to the gateways local unbound Then some python scripts are provided to block brute force attempts We will use DNSCrypt to make our DNS requests encrypted, and Unbound to have a local DNS cache. This will allow us to avoid using our VPS provider DNS servers, and will also be useful to your future VPN clients which will be able to use your VPN server as their DNS server too Before configuring Unbound, which is the local DNS cache which will make requests to dnscrypt_proxy, we can configure an additional dnscrypt instance, as explained in the pkg readme. Indeed, dnscrypt DNS servers being public ones, they often goes into maintenance, become offline or temporarily unreachable. To address this issue, it is possible to setup multiple dnscrypt instances. Below are the steps to follow to add one, but you can add more if you wish Then a CA and Certificate are created for OpenVPN OpenVPN is installed and configured as a server Configuration is also provided for a client, and a mobile client Thanks to the author for this great tutorial You might also want to check out this section from their 2015 version of this post: Security vs Anonymity (https://networkfilter.blogspot.nl/2015/01/be-your-own-vpn-provider-with-openbsd.html#security_anonymity) *** Essen Hackathon Trip - Benedict Reuschling (https://www.freebsdfoundation.org/blog/2017-essen-hackathon-trip-report-benedict-reuschling/) Over on the FreeBSD Foundation Blog, Benedict provides a detailed overview of the Essen Hackathon we were at a few weeks ago. Head over there and give it a read, and get a feel for what these smaller type of community events are like. Hopefully you can attend, or better yet, organize, a similar event in your area. News Roundup Blog about my self-hosted httpd blog (https://reykfloeter.com/posts/blog-about-my-blog) I really like Twitter because it allows me to share short messages, we have a great community, and 140 characters are enough for everybody. And this statement was exactly 140 characters, but sometimes I want to say more than that. And that's why I finally created this new blog. I was never really into blogging because I barely had time or the audience to write long articles. I sometimes wrote short stories for sites like undeadly.org, I collected some of them here, but my own blog was hosted on tumblr and never saw any activity. I want to try it again, and this time I decided to create a self-hosted blog. Something that runs on my own server and with httpd, the web server that I wrote for OpenBSD. So I was looking for potential blogging tools that I could use to run my own blog. Besides the popular and heavyweight ones such as WordPress, there are countless other options: I looked at blogs from fellow developers, such as Ted Unangst's flak (I like the fact that it is written in Lua but the implementation is a bit over my head), or Pelican that is used by Peter Hessler for bad.network (but, sorry, I don't like Python), and finally Kristaps Dzonsons' sblg that is used for all of his projects and blogs. I decided to use sblg. Kristaps keeps on releasing very useful free software. Most well-known is mandoc, at least everyone is using it for manpages these days, but there is is also his BCHS (beaches) web stack which strongly advertises OpenBSD's httpd. Great. I also use kcgi whenever I have to write small CGIs. So sblg seemed like the right choice to me. Let me quickly iterate over my current Makefile. I keep on tweaking this file, so it might have been changed by the time you are reading this article. Please note that the Makefile is written for OpenBSD's make, a distant derivative of pmake which is not like GNU make. I'm not a designer or web developer, but I appreciate good looking web pages. I wanted to have something that is responsive, works on desktops and mobiles, looks somewhat modern, works without JavaScript, but doesn't disqualify me for all the eye candy from a geek point of view. I bootstrapped the theme by creating a simple grid layout with a fairly typical blog style: banner, top menu, middle text, sidebar. In 2017, bootstrap is probably a vintage (or retro) framework but it makes it very easy to create responsive pages with a proper layout and without caring about all the CSS and HTML5 madness too much. I also use Font Awesome because it is awesome, provides some fancy icons, and was suggested in sblg's example templates (let's blame Kristaps for it). I do not include any JavaScript which prevents me from using bootstrap's responsive hamburger menu. I have to admit that "reykfloeter" is not an ideal name for a blog. My actual name is "Reyk Flöter", and I normally just use my first name "reyk" as a user- and nickname, but it was taken when I registered my Twitter account and the related domain. So I picked reykfloeter in a few places. I'm aware that my German last name is nearly unpronounceable for others, so "reykfloeter" appears like a random concatenation of letters. As most of us, I own a number of domains and maybe I should move the blog to bsd.plumbing (which is used as a home for relayd and httpd), arc4random.com (but I intended to use it as a fine OpenBSD-powered Entropy-as-a-Service for poor Linuxers), or even copper.coffee? In addition to the domain, I also need a good blog name or tag line. A very memorable example in the BSD world is Peter Hansteen's THAT GRUMPY BSD GUY blog. So what should I use? Reyk Flöter's blog OpenBSD hacker. Coffee nerd. Founder. Ask Reyk (imaginary how-tos and 10 step guides) Sewage, Drainage and BSD Plumbing (bsd.plumbing/blog) A Replacement Call for Random (arc4random.com) Coffee with Reyk (copper.coffee) For now it will just be reykfloeter - blog iXsystems releases the X10 (https://www.ixsystems.com/blog/serverenvy-truenas-x10/) TrueNAS X10 is the the 3rd generation of the TrueNAS unified storage line. The X10 is the first of a new TrueNAS series, and will be expandable to up to 360TB with the TrueNAS ES12 expansion shelf. The X10 is cost effective, at a 30% lower price point than the Z20, making it an effective addition to your backup/DR infrastructure. The street price of a 20TB non-HA model falls under $10K. It’s designed to move with six predefined configurations that match common use cases. The dual controllers for high availability are an optional upgrade to ensure business continuity and avoid downtime. The X10 boasts 36 hot swap SAS using two expansion shelves, for up to 360TB of storage, allowing you to backup thousands of VMs or share tens of thousands of files. One of the use cases for TrueNAS X10 is for backup, so users can upgrade the X10 to two ports of blazing 10GigE connectivity. The 20TB non-HA model enables you to backup over 7,000 VDI VMs for under $3.00 per VM. Overall, the X10 is a greener solution than the TrueNAS Z product line, with the non-HA version boasting only 138 watts of power and taking up only 2U of space. Best of all, the TrueNAS X10 starts at $5,500 street. You can purchase a 120TB configuration today for under $20K street. Glob Matching Can Be Simple And Fast Too (https://research.swtch.com/glob) Here’s a straightforward benchmark. Time how long it takes to run ls (a)nb in a directory with a single file named a100, compared to running ls | grep (a.)nb. Superscripts denote string repetition and parentheses are for grouping only, so that when n is 3, we’re running ls aaab in a directory containing the single file aaa…aaa (100 a’s), compared against ls | grep a.a.a.b in the same directory. The exception seems to be the original Berkeley csh, which runs in linear time (more precisely, time linear in n). Looking at the source code, it doesn’t attempt to perform glob expansion itself. Instead it calls the C library implementation glob(3), which runs in linear time, at least on this Linux system. So maybe we should look at programming language implementations too. Most programming languages provide some kind of glob expansion, like C’s glob. Let’s repeat the experiment in a variety of different programming languages: Perhaps the most interesting fact evident in the graph is that GNU glibc, the C library used on Linux systems, has a linear-time glob implementation, but BSD libc, the C library used on BSD and macOS systems, has an exponential-time implementation. PHP is not shown in the graph, because its glob function simply invokes the host C library’s glob(3), so that it runs in linear time on Linux and in exponential time on non-Linux systems. (I have not tested what happens on Windows.) All the languages shown in the graph, however, implement glob matching without using the host C library, so the results should not vary by host operating system. The netkit ftpd runs quickly on Linux because it relies on the host C library’s glob function. If run on BSD, the netkit ftpd would take exponential time. ProFTPD ships a copy of the glibc glob, so it should run quickly even on BSD systems. Ironically, Pure-FTPd and tnftpd take exponential time on Linux because they ship a copy of the BSD glob function. Presumably they do this to avoid assuming that the host C library is bug-free, but, at least in this one case, the host C library is better than the one they ship. Additional Reading This post is an elaboration of an informal 2012 Google+ post showing that most shells used exponential-time glob expansion. At the time, Tom Duff, the author of Plan 9’s rc shell, commented that, “I can confirm that rc gets it wrong. My excuse, feeble as it is, is that doing it that way meant that the code took 10 minutes to write, but it took 20 years for someone to notice the problem. (That’s 10 ‘programmer minutes’, i.e. less than a day.)” I agree that’s a reasonable decision for a shell. In contrast, a language library routine, not to mention a network server, today needs to be robust against worst-case inputs that might be controlled by remote attackers, but nearly all of the code in question predates that kind of concern. I didn’t realize the connection to FTP servers until I started doing additional research for this post and came across a reference to CVE-2010-2632 in FreeBSD’s glob implementation. BSD VPS Providers Needed (https://torbsd.github.io/blog.html#bsd-vps) One of TDP’s recent projects is accumulating a list of virtual private server services (VPS) that provide a BSD option. VPS’s are generally inexpensive services that enable the user to only concern themselves with software configuration, and not be bothered with hardware or basic operating system setup. In the pre-Cloud era, VPS providers were the “other people’s computers” that users outsourced their systems to. The same shortcomings of cloud services apply to VPS providers. You don’t control the hardware. Your files are likely viewable by users up the directory hierarchy. The entropy source or pool is a single source for multiple systems. The same time drift applies to all time-keeping services. Nevertheless, VPS services are often cheap and provide a good spread in terms of geography. All a provider really needs is a few server-grade computers and a decent network connection. VPS’s are still a gateway drug to bare-metal servers, although it seems more and more of these gateway users stop at stage one. Cheap systems with a public IP are also a great way to tinker with a new operating system. For this reason, TDP created this list of BSD VPS providers. Some explicitly deny running Tor as a server. Some just reference vague “proxy services.” Others don’t mention Tor or proxies at all. The list is a start with currently just under 70 VPS providers listed. Input through various channels already started, and TDP intends to update the list over the coming months. A first draft email and open letter addressed to the providers were drafted, and we are looking to speak directly to at least some of the better-known BSD VPS providers. We may be able to convince a few to allow public Tor relays, or at least published bridges. These providers could be new BSD users’ gateway drug into the world of BSD Tor nodes. Running a Tor relay shouldn’t be considered a particularly risky activity. Maybe we can adjust that perception. Let us know any input via email or GitHub, and we’ll be glad to make updates. Beastie Bits Avoid OS Detection with OpenBSD (https://blog.cagedmonster.net/avoid-os-detection-openbsd/) TrueOS update to fix updating (https://www.trueos.org/blog/update-fix-updating/) MidnightBSD 0.8.5 VirtualBox Install (https://www.youtube.com/watch?v=I08__ZWaJ0w) BSD Pizza Night in Portland (http://calagator.org/events/tag/BSD) *** Feedback/Questions Andrew - BSDCan videos? (http://dpaste.com/08E90PX) Marc - The Rock64 Board (http://dpaste.com/08KE40G) Jason - Follow up on UEFI and Bhyve (http://dpaste.com/2EP7BFC) Patrick - EFI booting (http://dpaste.com/34Z9SFM) ***
204: WWF - Wayland, Weston, and FreeBSD
In this episode, we clear up the myth about scrub of death, look at Wayland and Weston on FreeBSD, Intel QuickAssist is here, and we check out OpenSMTP on OpenBSD. This episode was brought to you by Headlines Matt Ahrens answers questions about the “Scrub of Death” In working on the breakdown of that ZFS article last week, Matt Ahrens contacted me and provided some answers he has given to questions in the past, allowing me to answer them using HIS exact words. “ZFS has an operation, called SCRUB, that is used to check all data in the pool and recover any data that is incorrect. However, if a bug which make errors on the pool persist (for example, a system with bad non-ecc RAM) then SCRUB can cause damage to a pool instead of recover it. I heard it called the “SCRUB of death” somewhere. Therefore, as far as I understand, using SCRUB without ECC memory is dangerous.” > I don't believe that is accurate. What is the proposed mechanism by which scrub can corrupt a lot of data, with non-ECC memory? > ZFS repairs bad data by writing known good data to the bad location on disk. The checksum of the data has to verify correctly for it to be considered "good". An undetected memory error could change the in-memory checksum or data, causing ZFS to incorrectly think that the data on disk doesn’t match the checksum. In that case, ZFS would attempt to repair the data by first re-reading the same offset on disk, and then reading from any other available copies of the data (e.g. mirrors, ditto blocks, or RAIDZ reconstruction). If any of these attempts results in data that matches the checksum, then the data will be written on top of the (supposed) bad data. If the data was actually good, then overwriting it with the same good data doesn’t hurt anything. > Let's look at what will happen with 3 types of errors with non-ECC memory: > 1. Rare, random errors (e.g. particle strikes - say, less than one error per GB per second). If ZFS finds data that matches the checksum, then we know that we have the correct data (at least at that point in time, with probability 1-1/2^256). If there are a lot of memory errors happening at a high rate, or if the in-memory checksum was corrupt, then ZFS won’t be able to find a good copy of the data , so it won’t do a repair write. It’s possible that the correctly-checksummed data is later corrupted in memory, before the repair write. However, the window of vulnerability is very very small - on the order of milliseconds between when the checksum is verified, and when the write to disk completes. It is implausible that this tiny window of memory vulnerability would be hit repeatedly. > 2. Memory that pretty much never does the right thing. (e.g. huge rate of particle strikes, all memory always reads 0, etc). In this case, critical parts of kernel memory (e.g. instructions) will be immediately corrupted, causing the system to panic and not be able to boot again. > 3. One or a few memory locations have "stuck bits", which always read 0 (or always read 1). This is the scenario discussed in the message which (I believe) originally started the "Scrub of Death" myth: https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/ This assumes that we read in some data from disk to a memory location with a stuck bit, "correct" that same bad memory location by overwriting the memory with the correct data, and then we write the bad memory location to disk. However, ZFS doesn't do that. (It seems the author thinks that ZFS uses parity, which it only does when using RAID-Z. Even with RAID-Z, we also verify the checksum, and we don't overwrite the bad memory location.) > Here's what ZFS will actually do in this scenario: If ZFS reads data from disk into a memory location with a stuck bit, it will detect a checksum mismatch and try to find a good copy of the data to repair the "bad" disk. ZFS will allocate a new, different memory location to read a 2nd copy of the data, e.g. from the other side of a mirror (this happens near the end of dslscanscrub_cb()). If the new memory location also has a stuck bit, then its checksum will also fail, so we won't use it to repair the "bad" disk. If the checksum of the 2nd copy of the data is correct, then we will write it to the "bad" disk. This write is unnecessary, because the "bad" disk is not really bad, but it is overwriting the good data with the same good data. > I believe that this misunderstanding stems from the idea that ZFS fixes bad data by overwriting it in place with good data. In reality, ZFS overwrites the location on disk, using a different memory location for each read from disk. The "Scrub of Death" myth assumes that ZFS overwrites the location in memory, which it doesn't do. > In summary, there's no plausible scenario where ZFS would amplify a small number of memory errors, causing a "scrub of death". Additionally, compared to other filesystems, ZFS checksums provide some additional protection against bad memory. “Is it true that ZFS verifies the checksum of every block on every read from disk?” > Yes “And if that block is incorrect, that ZFS will repair it?” > Yes “If yes, is it possible set options or flag for change that behavior? For example, I would like for ZFS to verify checksums during any read, but not change anything and only report about issues if it appears. Is it possible?” > There isn't any built-in flag for doing that. It wouldn't be hard to add one though. If you just wanted to verify data, without attempting to correct it, you could read or scan the data with the pool was imported read-only “If using a mirror, when a file is read, is it fully read and verified from both sides of the mirror?” > No, for performance purposes, each block is read from only one side of the mirror (assuming there is no checksum error). “What is the difference between a scrub and copying every file to /dev/null?” > That won't check all copies of the file (e.g. it won't check both sides of the mirror). *** Wayland, and Weston, and FreeBSD - Oh My! (https://euroquis.nl/bobulate/?p=1617) KDE’s CI system for FreeBSD (that is, what upstream runs to continuously test KDE git code on the FreeBSD platform) is missing some bits and failing some tests because of Wayland. Or rather, because FreeBSD now has Wayland, but not Qt5-Wayland, and no Weston either (the reference implementation of a Wayland compositor). Today I went hunting for the bits and pieces needed to make that happen. Fortunately, all the heavy lifting has already been done: there is a Weston port prepared and there was a Qt5-Wayland port well-hidden in the Area51 plasma5/ branch. I have taken the liberty of pulling them into the Area51 repository as branch qtwayland. That way we can nudge Weston forward, and/or push Qt5-Wayland in separately. Nicest from a testing perspective is probably doing both at the same time. I picked a random “Hello World” Wayland tutorial and also built a minimal Qt program (using QMessageBox::question, my favorite function to hate right now, because of its i18n characteristics). Then, setting XDGRUNTIMEDIR to /tmp/xdg, I could start Weston (as an X11 client), wayland-hello (as a Wayland client, displaying in Weston) and qt-hello (as either an X11 client, or as a Wayland client). So this gives users of Area51 (while shuffling branches, granted) a modern desktop and modern display capabilities. Oh my! It will take a few days for this to trickle up and/or down so that the CI can benefit and we can make sure that KWin’s tests all work on FreeBSD, but it’s another good step towards tight CI and another small step towards KDE Plasma 5 on the desktop on FreeBSD. pkgsrcCon 2017 report (https://blog.netbsd.org/tnf/entry/pkgsrccon_2017_report) This years pkgsrcCon returned to London once again. It was last held in London back in 2014. The 2014 con was the first pkgsrcCon I attended, I had been working on Darwin/PowerPC fixes for some months and presented on the progress I'd made with a 12" G4 PowerBook. I took away a G4 Mac Mini that day to help spare the PowerBook for use and dedicate a machine for build and testing. The offer of PowerPC hardware donations was repeated at this years con, thanks to jperkin@ who showed up with a backpack full of Mac Minis (more on that later). Since 2014 we have held cons in Berlin (2015) & Krakow (2016). In Krakow we had talks about a wide range of projects over 2 days, from Haiku Ports to Common Lisp to midipix (building native PE binaries for Windows) and back to the BSDs. I was very pleased to continue the theme of a diverse program this year. Aside from pkgsrc and NetBSD, we had talks about FreeBSD, OpenBSD, Slackware Linux, and Plan 9. Things began with a pub gathering on the Friday for the pre-con social, we hung out and chatted till almost midnight on a wide range of topics, such as supporting a system using NFS on MS-DOS, the origins of pdksh, corporate IT, culture and many other topics. On parting I was asked about the starting time on Saturday as there was some conflicting information. I learnt that the registration email had stated a later start than I had scheduled for & advertised on the website, by 30 minutes. Lesson learnt: register for your own event! Not a problem, I still needed to setup a webpage for the live video stream, I could do both when I got back. With some trimming here and there I had a new schedule, I posted that to the pkgsrcCon website and moved to trying to setup a basic web page which contained a snippet of javascript to play a live video stream from Scale Engine. 2+ hours later, it was pointed out that the XSS protection headers on pkgsrc.org breaks the functionality. Thanks to jmcneill@ for debugging and providing a working page. Saturday started off with Giovanni Bechis speaking about pledge in OpenBSD and adding support to various packages in their ports tree, alnsn@ then spoke about installing packages from a repo hosted on the Tor network. After a quick coffee break we were back to hear Charles Forsyth speak about how Plan 9 and Inferno dealt with portability, building software and the problem which are avoided by the environment there. This was followed by a very energetic rant by David Spencer from the Slackbuilds project on packaging 3rd party software. Slackbuilds is a packaging system for Slackware Linux, which was inspired by FreeBSD ports. For the first slot after lunch, agc@ gave a talk on the early history of pkgsrc followed by Thomas Merkel on using vagrant to test pkgsrc changes with ease, locally, using vagrant. khorben@ covered his work on adding security to pkgsrc and bsiegert@ covered the benefits of performing our bulk builds in the cloud and the challenges we currently face. My talk was about some topics and ideas which had inspired me or caught my attention, and how it could maybe apply to my work.The title of the talk was taken from the name of Andrew Weatherall's Saint Etienne remix, possibly referring to two different styles of track (dub & vocal) merged into one or something else. I meant it in terms of applicability of thoughts and ideas. After me, agc@ gave a second talk on the evolution of the Netflix Open Connect appliance which runs FreeBSD and Vsevolod Stakhov wrapped up the day with a talk about the technical implementation details of the successor to pkgtools in FreeBSD, called pkg, and how it could be of benefit for pkgsrc. For day 2 we gathered for a hack day at the London Hack Space. I had burn't some some CD of the most recent macppc builds of NetBSD 8.0BETA and -current to install and upgrade Mac Minis. I setup the donated G4 minis for everyone in a dual-boot configuration and moved on to taking apart my MacBook Air to inspect the wifi adapter as I wanted to replace it with something which works on FreeBSD. It was not clear from the ifixit teardown photos of cards size, it seemed like a normal mini-PCIe card but it turned out to be far smaller. Thomas had also had the same card in his and we are not alone. Thomas has started putting together a driver for the Broadcom card, the project is still in its early days and lacks support for encrypted networks but hopefully it will appear on review.freebsd.org in the future. weidi@ worked on fixing SunOS bugs in various packages and later in the night we setup a NetBSD/macppc bulk build environment together on his Mac Mini. Thomas setup an OpenGrock instance to index the source code of all the software available for packaging in pkgsrc. This helps make the evaluation of changes easier and the scope of impact a little quicker without having to run through a potentially lengthy bulk build with a change in mind to realise the impact. bsiegert@ cleared his ticket and email backlog for pkgsrc and alnsn@ got NetBSD/evbmips64-eb booting on his EdgeRouter Lite. On Monday we reconvened at the Hack Space again and worked some more. I started putting together the talks page with the details from Saturday and the the slides which I had received, in preparation for the videos which would come later in the week. By 3pm pkgsrcCon was over. I was pretty exhausted but really pleased to have had a few days of techie fun. Many thanks to The NetBSD Foundation for purchasing a camera to use for streaming the event and a speedy response all round by the board. The Open Source Specialist Group at BCS, The Chartered Institute for IT and the London Hack Space for hosting us. Scale Engine for providing streaming facility. weidi@ for hosting the recorded videos. Allan Jude for pointers, Jared McNeill for debugging, NYCBUG and Patrick McEvoy for tips on streaming, the attendees and speakers. This year we had speakers from USA, Italy, Germany and London E2. Looking forward to pkgsrcCon 2018! The videos and slides are available here (http://www.pkgsrc.org/pkgsrcCon/2017/talks.html) and the Internet Archive (http://archive.org/details/pkgsrcCon-2017). News Roundup QuickAssist Driver for FreeBSD is here and pfSense Support Coming (https://www.servethehome.com/quickassist-driver-freebsd-pfsupport-coming/) This week we have something that STH readers will be excited about. Before I started writing for STH, I was a reader and had been longing for QuickAssist support ever since STH’s first Rangeley article over three and a half years ago. It was clear from the get-go that Rangeley was going to be the preeminent firewall appliance platform of its day. The scope of products that were impacted by the Intel Atom C2000 series bug showed us it was indeed. For my personal firewalls, I use pfSense on that Rangeley platform so I have been waiting to use QuickAssist with my hardware for almost an entire product generation. + New Hardware and QuickAssist Incoming to pfSense (Finally) pfSense (and a few other firewalls) are based on FreeBSD. FreeBSD tends to lag driver support behind mainstream Linux but it is popular for embedded security appliances. While STH is the only site to have done QuickAssist benchmarks for OpenSSL and IPSec VPNs pre-Skylake, we expect more platforms to use it now that the new Intel Xeon Scalable Processor Family is out. With the Xeon Scalable platforms, the “Lewisburg” PCH has QuickAssist options of up to 100Gbps, or 2.5x faster than the previous generation add-in cards we tested (40Gbps.) We now have more and better hardware for QAT, but we were still devoid of a viable FreeBSD QAT driver from Intel. That has changed. Our Intel Xeon Scalable Processor Family (Skylake-SP) Launch Coverage Central has been the focus of the STH team’s attention this week. There was another important update from Intel that got buried, a publicly available Intel QuickAssist driver for FreeBSD. You can find the driver on 01.org here dated July 12, 2017. Drivers are great, but we still need support to be enabled in the OS and at the application layer. Patrick forwarded me this tweet from Jim Thompson (lead at Netgate the company behind pfSense): The Netgate team has been a key company pushing QuickAssist appliances in the market, usually based on Linux. To see that QAT is coming to FreeBSD and that they were working to integrate into “pfSense soon” is more than welcome. For STH readers, get ready. It appears to be actually and finally happening. QuickAssist on FreeBSD and pfSense OpenBSD on the Huawei MateBook X (https://jcs.org/2017/07/14/matebook) The Huawei MateBook X is a high-quality 13" ultra-thin laptop with a fanless Core i5 processor. It is obviously biting the design of the Apple 12" MacBook, but it does have some notable improvements such as a slightly larger screen, a more usable keyboard with adequate key travel, and 2 USB-C ports. It also uses more standard PC components than the MacBook, such as a PS/2-connected keyboard, removable m.2 WiFi card, etc., so its OpenBSD compatibility is quite good. In contrast to the Xiaomi Mi Air, the MateBook is actually sold (2) in the US and comes with a full warranty and much higher build quality (though at twice the price). It is offered in the US in a "space gray" color for the Core i5 model and a gold color for the Core i7. The fanless Core i5 processor feels snappy and doesn't get warm during normal usage on OpenBSD. Doing a make -j4 build at full CPU speed does cause the laptop to get warm, though the palmrest maintains a usable temperature. The chassis is all aluminum and has excellent rigidity in the keyboard area. The 13.0" 2160x1440 glossy IPS "Gorilla glass" screen has a very small bezel and its hinge is properly weighted to allow opening the lid with one hand. There is no wobble in the screen when open, even when jostling the desk that the laptop sits on. It has a reported brightness of 350 nits. I did not experience any of the UEFI boot variable problems that I did with the Xiaomi, and the MateBook booted quickly into OpenBSD after re-initializing the GPT table during installation. OpenSMTPD under OpenBSD with SSL/VirtualUsers/Dovecot (https://blog.cagedmonster.net/opensmtpd-under-openbsd-with-ssl-virtualusers-dovecot/) During the 2013 AsiaBSDCon, the team of OpenBSD presented its mail solution named OpenSMTPD. Developed by the OpenBSD team, we find the so much appreciated philosophy of its developers : security, simplicity / clarity and advanced features. Basic configuration : OpenSMTPD is installed by default, we can immediately start with a simple configuration. > We listen on our interfaces, we specify the path of our aliases file so we can manage redirections. > Mails will be delivered for the domain cagedmonster.net to mbox (the local users mailbox), same for the aliases. > Finally, we accept to relay local mails exclusively. > We can now enable smtpd at system startup and start the daemon. Advanced configuration including TLS : You can use SSL with : A self-signed certificate (which will not be trusted) or a certificate generated by a trusted authority. LetsEncrypt uses Certbot to generated your certificate. You can check this page for further informations. Let's focus on the first. Generation of the certificate : We fix the permissions : We edit the config file : > We have a mail server with SSL, it's time to configure our IMAP server, Dovecot, and manage the creation of virtual users. Dovecot setup, and creation of Virtual Users : We will use the package system of OpenBSD, so please check the configuration of your /etc/pkg.conf file. Enable the service at system startup : Setup the Virtual Users structure : Adding the passwd table for smtpd : Modification of the OpenSMTPD configuration : We declare the files used for our Virtual Accounts, we include SSL, and we configure mails delivery via the Dovecot lmtp socket. We'll create our user lina@cagedmonster.net and set its password. Configure SSL Configure dovecot.conf Configure mail.con Configure login.conf : Make sure that the value of openfiles-cur in /etc/login.conf is equal or superior of 1000 ! Starting Dovecot *** OpenSMTPD and Dovecot under OpenBSD with MySQL support and SPAMD (https://blog.cagedmonster.net/opensmtpd-and-dovecot-under-openbsd-with-mysql-support-and-spamd/) This article is the continuation of my previous tutorial OpenSMTPD under OpenBSD with SSL/VirtualUsers/Dovecot. We'll use the same configuration and add some features so we can : Use our domains, aliases, virtual users with a MySQL database (MariaDB under OpenBSD). Deploy SPAMD with OpenSMTPD for a strong antispam solution. + Setup of the MySQL support for OpenSMTPD & Dovecot + We create our SQL database named « smtpd » + We create our SQL user « opensmtpd » we give him the privileges on our SQL database and we set its password + We create the structure of our SQL database + We generate our password with Blowfish (remember it's OpenBSD !) for our users + We create our tables and we include our datas + We push everything to our database + Time to configure OpenSMTPD + We create our mysql.conf file and configure it + Configuration of Dovecot.conf + Configuration of auth-sql.conf.ext + Configuration of dovecot-sql.conf.ext + Restart our services OpenSMTPD & SPAMD : SPAMD is a service simulating a fake SMTP server and relying on strict compliance with RFC to determine whether the server delivering a mail is a spammer or not. + Configuration of SPAMD : + Enable SPAMD & SPAMLOGD at system startup : + Configuration of SPAMD flags + Configuration of PacketFilter + Configuration of SPAMD + Start SPAMD & SPAMLOGD Running a TOR relay on FreeBSD (https://networkingbsdblog.wordpress.com/2017/07/14/freebsd-tor-relay-using-priveledge-seperation/) There are 2 main steps to getting a TOR relay working on FreeBSD: Installing and configuring Tor Using an edge router to do port translation In my case I wanted TOR to run it’s services on ports 80 and 443 but any port under 1024 requires root access in UNIX systems. +So I used port mapping on my router to map the ports. +Begin by installing TOR and ARM from: /usr/ports/security/tor/ /usr/ports/security/arm/ Arm is the Anonymizing Relay Monitor: https://www.torproject.org/projects/arm.html.en It provides useful monitoring graph and can be used to configure the torrc file. Next step edit the torrc file (see Blog article for the edit) It is handy to add the following lines to /etc/services so you can more easily modify your pf configuration. torproxy 9050/tcp #torsocks torOR 9090/tcp #torOR torDIR 9099/tcp #torDIR To allow TOR services my pf.conf has the following lines: # interfaces lan_if=”re0″ wifi_if=”wlan0″ interfaces=”{wlan0,re0}” tcp_services = “{ ssh torproxy torOR torDIR }” # options set block-policy drop set loginterface $lan_if # pass on lo set skip on lo scrub in on $lan_if all fragment reassemble # NAT nat on $lan_if from $wifi_if:network to !($lan_if) -> ($lan_if) block all antispoof for $interfaces #In NAT pass in log on $wifi_if inet pass out all keep state #ICMP pass out log inet proto icmp from any to any keep state pass in log quick inet proto icmp from any to any keep state #SSH pass in inet proto tcp to $lan_if port ssh pass in inet proto tcp to $wifi_if port ssh #TCP Services on Server pass in inet proto tcp to $interfaces port $tcp_services keep state The finally part is mapping the ports as follows: TOR directory port: LANIP:9099 —> WANIP:80 TOR router port: LANIP:9090 —-> WANIP:443 Now enable TOR: $ sudo echo “tor_enable=YES” >> /etc/rc.conf Start TOR: $ sudo service tor start *** Beastie Bits OpenBSD as a “Desktop” (Laptop) (http://unixseclab.com/index.php/2017/06/12/openbsd-as-a-desktop-laptop/) Sascha Wildner has updated ACPICA in DragonFly to Intel’s version 20170629 (http://lists.dragonflybsd.org/pipermail/commits/2017-July/625997.html) Dport, Rust, and updates for DragonFlyBSD (https://www.dragonflydigest.com/2017/07/18/19991.html) OPNsense 17.7 RC1 released (https://opnsense.org/opnsense-17-7-rc1/) Unix’s mysterious && and || (http://www.networkworld.com/article/3205148/linux/unix-s-mysterious-andand-and.html#tk.rss_unixasasecondlanguage) The Commute Deck : A Homebrew Unix terminal for tight places (http://boingboing.net/2017/06/16/cyberspace-is-everting.html) FreeBSD 11.1-RC3 now available (https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087407.html) Installing DragonFlyBSD with ORCA when you’re totally blind (http://lists.dragonflybsd.org/pipermail/users/2017-July/313528.html) Who says FreeBSD can’t look good (http://imgur.com/gallery/dc1pu) Pratik Vyas adds the ability to do paused VM migrations for VMM (http://undeadly.org/cgi?action=article&sid=20170716160129) Feedback/Questions Hrvoje - OpenBSD MP Networking (http://dpaste.com/0EXV173#wrap) Goran - debuggers (http://dpaste.com/1N853NG#wrap) Abhinav - man-k (http://dpaste.com/1JXQY5E#wrap) Liam - university setup (http://dpaste.com/01ERMEQ#wrap)
203: For the love of ZFS
This week on BSD Now, we clear up some ZFS FUD, show you how to write a NetBSD kernel module, and cover DragonflyBSD on the desktop. This episode was brought to you by Headlines ZFS is the best file system (for now) (http://blog.fosketts.net/2017/07/10/zfs-best-filesystem-now/) In my ongoing effort to fight misinformation and FUD about ZFS, I would like to go through this post in detail and share my thoughts on the current state and future of OpenZFS. The post starts with: ZFS should have been great, but I kind of hate it: ZFS seems to be trapped in the past, before it was sidelined it as the cool storage project of choice; it’s inflexible; it lacks modern flash integration; and it’s not directly supported by most operating systems. But I put all my valuable data on ZFS because it simply offers the best level of data protection in a small office/home office (SOHO) environment. Here’s why. When ZFS first appeared in 2005, it was absolutely with the times, but it’s remained stuck there ever since. The ZFS engineers did a lot right when they combined the best features of a volume manager with a “zettabyte-scale” filesystem in Solaris 10 The skies first darkened in 2007, as NetApp sued Sun, claiming that their WAFL patents were infringed by ZFS. Sun counter-sued later that year, and the legal issues dragged on. The lawsuit was resolved, and it didn’t really impede ZFS. Some say it is the reason that Apple didn’t go with ZFS, but there are other theories too. By then, Sun was hitting hard times and Oracle swooped in to purchase the company. This sowed further doubt about the future of ZFS, since Oracle did not enjoy wide support from open source advocates. Yes, Oracle taking over Sun and closing the source for ZFS definitely seemed like a setback at the time, but the OpenZFS project was started and active development has continued as an ever increasing pace. As of today, more than half of the code in OpenZFS has been written since the fork from the last open version of Oracle ZFS. the CDDL license Sun applied to the ZFS code was https://sfconservancy.org/blog/2016/feb/25/zfs-and-linux/ (judged incompatible) with the GPLv2 that covers Linux, making it a non-starter for inclusion in the world’s server operating system. That hasn’t stopped the ZFS-on-Linux project, or Ubuntu… Although OpenSolaris continued after the Oracle acquisition, and FreeBSD embraced ZFS, this was pretty much the extent of its impact outside the enterprise. Sure, NexentaStor and http://blog.fosketts.net/2008/09/15/greenbytes-embraces-extends-zfs/ (GreenBytes) helped push ZFS forward in the enterprise, but Oracle’s lackluster commitment to Sun in the datacenter started having an impact. Lots of companies have adopted OpenZFS for their products. Before OpenZFS, there were very few non-Sun appliances that used ZFS, now there are plenty. OpenZFS Wiki: Companies with products based on OpenZFS (http://open-zfs.org/wiki/Companies) OpenZFS remains little-changed from what we had a decade ago. Other than the fact that half of the current code did not exist a decade ago… Many remain skeptical of deduplication, which hogs expensive RAM in the best-case scenario. This is one of the weaker points in ZFS. As it turns out, the demand for deduplication is actually not that strong. Most of the win can be had with transparent compression. However, there are a number of suggested designs to work around the dedup problems: Dedup Ceiling: Set a limit on the side of the DDT and just stop deduping new unique blocks when this limit is reached. Allocation Classes: A feature being developed by Intel for a supercomputer, will allow different types of data to be classified, and dedicated vdevs (or even metaslabs within a vdev), to be dedicated to that class of data. This could be extended to having the DDT live on a fast device like an PCIe NVMe, combined with the Dedup Ceiling when the device is full. DDT Pruning: Matt Ahrens described a design where items in the DDT with only a single reference, would be expired in an LRU type fashion, to allow newer blocks to live in the DDT in hopes that they would end up with more than a single reference. This doesn’t cause bookkeeping problems since when a block is about to be freed, if it is NOT listed in the DDT, ZFS knows it was never deduplicated, so the current block must be the only reference, and it can safely be freed. This provides a best case scenario compared to Dedup Ceiling, since blocks that will deduplicate well, are likely to be written relatively close together, whereas the chance to a dedup match on a very old block is much lower. And I do mean expensive: Pretty much every ZFS FAQ flatly declares that ECC RAM is a must-have and 8 GB is the bare minimum. In my own experience with FreeNAS, 32 GB is a nice amount for an active small ZFS server, and this costs $200-$300 even at today’s prices. As we talked about a few weeks ago, ECC is best, but it is not required. If you want your server to stay up for a long time, to be highly available, you’ll put ECC in it. Don’t let a lack of ECC stop you from using ZFS, you are just putting your data at more risk. The scrub of death is a myth. ZFS does not ‘require’ lots of ram. Your NAS will work happily with 8 GB instead of 32 GB of RAM. Its cache hit ratio will be much lower, so performance will be worse. It won’t be able to buffer as many writes, so performance will be worse. Copy-on-Write has some drawbacks, data tends to get scattered and fragmented across the drives when it is written gradually. The ARC (RAM Cache) lessens the pain of this, and allows ZFS to batch incoming writes up into nice contiguous writes. ZFS purposely alternates between reading and writing, since both are faster when the other is not happening. So writes are batched up until there is too much dirty data, or the timeout expires. Then reads are held off while the bulk linear write finishes as quickly as possible, and reads are resumed. Obviously all of this works better and more efficiently in larger batches, which you can do if you have more RAM. ZFS can be tuned to use less RAM, and if you do not have a lot of RAM, or you have a lot of other demand on your RAM, you should do that tuning. And ZFS never really adapted to today’s world of widely-available flash storage: Although flash can be used to support the ZIL and L2ARC caches, these are of dubious value in a system with sufficient RAM, and ZFS has no true hybrid storage capability. It’s laughable that the ZFS documentation obsesses over a few GB of SLC flash when multi-TB 3D NAND drives are on the market. And no one is talking about NVMe even though it’s everywhere in performance PC’s. Make up your mind, is 32GB of ram too expensive or not… the L2ARC exists specifically for the case where it is not possible to just install more RAM. Be it because there are no more slots, of limits of the processor, or limits of your budget. The SLOG is optional, but it never needs to be very big. A number of GBs of SLC flash is all you need, it is only holding writes that have not been flushed to the regular storage devices yet. The reason the documentation talks about SLC specifically is because your SLOG needs a very high write endurance, something never the newest NVMe devices cannot yet provide. Of course you can use NVMe devices with ZFS, lots of people do. All flash ZFS arrays are for sale right now. Other than maybe a little tuning of the device queue depths, ZFS just works and there is nothing to think about. However, to say there is nothing happening in this space is woefully inaccurate. The previously mentioned allocation classes code can be used to allocate metadata (4 KB blocks) on SSD or NVMe, while allocating bulk storage data (up to 16 MB blocks) on spinning disks. Extended a bit beyond what Intel is building for their super computer, this will basically create hybrid storage for ZFS. With the metaslab classes feature, it will even be possible to mix classes on the same device, grouping small allocations and large allocations in different areas, decreasing fragmentation. Then there’s the question of flexibility, or lack thereof. Once you build a ZFS volume, it’s pretty much fixed for life. There are only three ways to expand a storage pool: Replace each and every drive in the pool with a larger one (which is great but limiting and expensive) It depends on your pool layout. If you design with this in mind using ZFS Mirrors, it can be quite useful Add a stripe on another set of drives (which can lead to imbalanced performance and redundancy and a whole world of potential stupid stuff) The unbalanced LUNs performance issues were sorted out in 2013-2016. 2014: OpenZFS Allocation Performance (http://open-zfs.org/w/images/3/31/Performance-George_Wilson.pdf) 2016: OpenZFS space allocation: doubling performance on large and fragmented pools (http://www.bsdcan.org/2016/schedule/events/710.en.html) These also mostly solved the performance issues when a pool gets full, you can run a lot closer to the edge now Build a new pool and “zfs send” your datasets to it (which is what I do, even though it’s kind of tricky) This is one way to do it, yes. There is another way coming, but I can’t talk about it just yet. Look for big news later this year. Apart from option 3 above, you can’t shrink a ZFS pool. Device removal is arriving now. It will not work for RAIDZ*, but for Mirrors and Stripes you will be able to remove a device. I’ve probably made ZFS sound pretty unappealing right about now. It was revolutionary but now it’s startlingly limiting and out of touch with the present solid-state-dominated storage world. I don’t feel like ZFS is out of touch with solid state. Lots of people are running SSD only pools. I will admit the tiered storage options in ZFS are a bit limited still, but there is a lot of work being done to overcome this. After all, reliably storing data is the only thing a storage system really has to do. All my important data goes on ZFS, from photos to music and movies to office files. It’s going to be a long time before I trust anything other than ZFS! + I agree. + ZFS has a great track record of doing its most important job, keeping your data safe. + Work is ongoing to make ZFS more performance, and more flexible. The import thing is that this work is never allowed to compromise job #1, keeping your data safe. + Hybrid/tiered storage features, re-RAID-ing, are coming + There is a lot going on with OpenZFS, check out the notes from the last two OpenZFS Developer Summits just to get an idea of what some of those things are: 2015 (http://open-zfs.org/wiki/OpenZFS_Developer_Summit_2015) & 2016 (http://open-zfs.org/wiki/OpenZFS_Developer_Summit_2016) Some highlights: Compressed ARC Compressed send/recv ABD (arc buf scatter/gather) ZFS Native Encryption (scrub/resilver, send/recv, etc without encryption keys loaded) Channel Programs (do many administrative operations as one atomic transaction) Device Removal Redacted send/recv ZStandard Compression TRIM Support (FreeBSD has its own, but this will be more performant and universal) Faster Scrub/Resilver (https://youtu.be/SZFwv8BdBj4) Declustered RAID (https://youtu.be/MxKohtFSB4M) Allocation Classes (https://youtu.be/28fKiTWb2oM) Multi-mount protection (for Active/Passive failover) Zpool Checkpoint (undo almost anything) Even more Improved Allocator Performance vdev spacemap log ZIL performance improvements (w/ or w/o SLOG) Persistent L2ARC What I don’t think the author of this article understands is how far behind every other filesystem is. 100s of Engineer years have gone into OpenZFS, and the pace is accelerating. I don’t see how BtrFS can ever catch up, without a huge cash infusion. Writing a NetBSD kernel module (https://saurvs.github.io/post/writing-netbsd-kern-mod/) Kernel modules are object files used to extend an operating system’s kernel functionality at run time. In this post, we’ll look at implementing a simple character device driver as a kernel module in NetBSD. Once it is loaded, userspace processes will be able to write an arbitrary byte string to the device, and on every successive read expect a cryptographically-secure pseudorandom permutation of the original byte string. You will need the NetBSD Source Code. This doc (https://www.netbsd.org/docs/guide/en/chap-fetch.html) will explain how you can get it. The article gives an easy line by line walkthrough which is easy to follow and understand. The driver implements the bare minimum: open, close, read, and write, plus the module initialization function It explains the differences in how memory is allocated and freed in the kernel It also describes the process of using UIO to copy data back and forth between userspace and the kernel Create a Makefile, and compile the kernel module Then, create a simple userspace program to use the character device that the kernel module creates All the code is available here (https://github.com/saurvs/rperm-netbsd) *** DragonFlyBSD Desktop! (https://functionallyparanoid.com/2017/07/11/dragonflybsd-desktop/) If you read my last post (https://functionallyparanoid.com/2017/06/30/boot-all-the-things/), you know that I set up a machine (Thinkpad x230) with UEFI and four operating systems on it. One, I had no experience with – DragonFlyBSD (other than using Matthew Dillon’s C compiler for the Amiga back in the day!) and so it was uncharted territory for me. After getting the install working, I started playing around inside of DragonFlyBSD and discovered to my delight that it was a great operating system with some really unique features – all with that BSD commitment to good documentation and a solid coupling of kernel and userland that doesn’t exist (by design) in Linux. So my goal for my DragonFlyBSD desktop experience was to be as BSD as I possibly could. Given that (and since I’m the maintainer of the port on OpenBSD ), I went with Lumina as the desktop environment and XDM as the graphical login manager. I have to confess that I really like the xfce terminal application so I wanted to make sure I had that as well. Toss in Firefox, libreOffice and ownCloud sync client and I’m good to go! OK. So where to start. First, we need to get WiFi and wired networking happening for the console at login. To do that, I added the following to /etc/rc.conf: wlans_iwn0=”wlan0″ ifconfig_wlan0=”WPA DHCP” ifconfig_em0=”DHCP” I then edited /etc/wpa_supplicant.conf to put in the details of my WiFi network: network={ ssid=”MY-NETWORK-NAME” psk=”my-super-secret-password” } A quick reboot showed that both wired and wireless networking were functional and automatically were assigned IP addresses via DHCP. Next up is to try getting into X with whatever DragonFlyBSD uses for its default window manager. A straight up “startx” met with, shall we say, less than stellar results. Therefore, I used the following command to generate a simple /etc/X11/xorg.conf file: # Xorg -configure # cp /root/xorg.conf.new /etc/X11/xorg.conf With that file in place, I could get into the default window manager, but I had no mouse. After some searching and pinging folks on the mailing list, I was able to figure out what I needed to do. I added the following to my /etc/rc.conf file: moused_enable=”YES” moused_type=”auto” moused_port=”/dev/psm0″ I rebooted (I’m sure there is an easier way to get the changes but I don’t know it… yet) and was able to get into a basic X session and have a functional mouse. Next up, installing and configuring Lumina! To do that, I went through the incredibly torturous process of installing Lumina: # pkg install lumina Wow! That was really, really hard. I might need to pause here to catch my breath. 🙂 Next up, jumping into Lumina from the console. To do that, I created a .xinitrc file in my home directory with the following: exec start-lumina-desktop From there, I could “startx” until my heart was content and bounce into Lumina. That wasn’t good enough though! I want a graphical login (specifically xdm). To do that, I had to do a little research. The trick on DragonFlyBSD is not to add anything to /etc/rc.conf like you do in other BSDs, it’s a bit more old school. Basically you need to edit the /etc/ttys file and update ttyv8 to turn on the xdm daemon: ttyv8 “/usr/X11R6/bin/xdm -nodaemon” xterm on secure The other thing you need to do is set it up to use your desktop environment of choice. In my case, that’s Lumina. To do that, I needed to edit /usr/local/lib/X11/xdm/Xsession and change the next to the last line in the file to launch Lumina: exec /usr/local/bin/start-lumina-desktop I then crossed my fingers, rebooted and lo and behold had a graphical login that, when I actually didn’t fat finger my password from excitement, put me into the Lumina desktop environment! Next up – I need a cool desktop wallpaper. Of course that’s way more important that installing application or other stuff! After some searching, I found this one that met my needs. I downloaded it to a local ~/Pictures directory and then used the Lumina wallpaper preference application to add the directory containing the picture and set it to automatic layout. Voila! I had a cool DragonFlyBSD wallpaper. Next I installed the xfce4 terminal program by doing: # pkg install xfce4-terminal I then went into the Lumina “All Desktop Settings” preferences, found the applet for the “Menu” under “Interface Configuration” and swapped out “terminal” for “Xfce Terminal”. I then configured Lumina further to have a 26 pixel thick, 99% length bottom center panel with the following gadgets in it (in this order): Start Menu Task Manager (No Groups) Spacer System Tray Time/Date Battery Monitor I then went into my Appearance | Window Manager gadget and set my Window Theme to “bora_blue” (my favorite out of the defaults supplied). I then installed my remaining applications that I needed in order to have a functioning desktop: # pkg install owncloudclient qtkeychain evolution evolution-ews firefox libreoffice After that, I really had a nicely functioning desktop environment! By the way, the performance of DragonFlyBSD is pretty impressive in terms of its day to day usage. Keep in mind I’m not doing any official benchmarking or anything, but it sure feels to me to be just as fast (if not faster) than OpenBSD and FreeBSD. I know that the kernel team has done a lot to unlock things (which FreeBSD has done and we are starting to do on OpenBSD) so perhaps I can attribute the “snappiness” to that? As you can see, although there isn’t as much documentation on the Internet for this BSD, you can get a really nice, functional desktop out of it with some simple (and intuitive) configuration. I’m really looking forward to living in this system for a while and learning about it. Probably the first thing I’ll do is ring up the port maintainer for Lumina and see if we can’t collaborate on getting Lumina 1.3 moved over to it! Give this one a try – I think you’ll find that its a very nice operating system with some very cool features (the HAMMER filesystem for one!). News Roundup Porting NetBSD to Alwinner H3 SoCs (http://blog.netbsd.org/tnf/entry/porting_netbsd_to_allwinner_h3) Jared McNeill writes on the the NetBSD blog: A new SUNXI evbarm kernel has appeared recently in NetBSD -current with support for boards based on the Allwinner H3 system on a chip (SoC). The H3 SoC is a quad-core Cortex-A7 SoC designed primarily for set-top boxes, but has managed to find its way into many single-board computers (SBC). This is one of the first evbarm ports built from the ground up with device tree support, which helps us to use a single kernel config to support many different boards. To get these boards up and running, first we need to deal with low-level startup code. For the SUNXI kernel this currently lives in sys/arch/evbarm/sunxi/. The purpose of this code is fairly simple; initialize the boot CPU and initialize the MMU so we can jump to the kernel. The initial MMU configuration needs to cover a few things -- early on we need to be able to access the kernel, UART debug console, and the device tree blob (DTB) passed in from U-Boot. We wrap the kernel in a U-Boot header that claims to be a Linux kernel; this is no accident! This tells U-Boot to use the Linux boot protocol when loading the kernel, which ensures that the DTB (loaded by U-Boot) is processed and passed to us in r2. Once the CPU and MMU are ready, we jump to the generic ARM FDT implementation of initarm in sys/arch/evbarm/fdt/fdtmachdep.c. The first thing this code does is validate and relocate the DTB data. After it has been relocated, we compare the compatible property of the root node in the device tree with the list of ARM platforms compiled into the kernel. The Allwinner sunxi platform code lives in sys/arch/arm/sunxi/sunxiplatform.c. The sunxi platform code provides SoC-specific versions of code needed early at boot. We need to know how to initialize the debug console, spin up application CPUs, reset the board, etc. With a bit of luck, we're now booting and enumerating devices. Apart from a few devices, almost nothing works yet as we are missing a driver for the CCU. The CCU in the Allwinner H3 SoC controls PLLs and most of the clock generation, division, muxing, and gating. Since there are many similarities between Allwinner SoCs, I opted to write generic CCU code and then SoC-specific frontends. The resulting code lives in sys/arch/arm/sunxi/; generic code as sunxiccu.c and H3-specific code in sun8ih3_ccu.c. Jared has more information about porting and also provides a boot log. *** TrueOS Community Spotlight: Reset ZFS Replication using the command line (https://www.trueos.org/blog/community-spotlight/) We’d like to spotlight TrueOS community member Brad Alexander for documenting his experience repairing ZFS dataset replication with TrueOS. Thank you! His notes are posted here, and they’ve been added to the TrueOS handbook Troubleshooting section for later reference. Original indications The SysAdm Client tray icon was pulsing red. Right-clicking on the icon and clicking Messages would show the message: FAILED replication task on NCC74602 -> 192.168.47.20: LOGFILE: /var/log/lpreserver/lpreserver_failed.log /var/log/lpreserver/lastrep-send.log shows very little information: send from @auto-2017-07-12-01-00-00 to NCC74602/ROOT/12.0-CURRENT-up-20170623_120331@auto-2017-07-14-01-00-00 total estimated size is 0 TIME SENT SNAPSHOT And no useful errors were being written to the lpreserver_failed.log. Repairing replication First attempt: The first approach I tried was to use the Sysadm Client: I clicked on the dataset in question, then clicked Initialize. After waiting a few minutes, I clicked Start. I was immediately rewarded with a pulsing red icon in the system tray and received the same messages as above. Second attempt: I was working with, and want to specially thank @RodMyers and @NorwegianRockCat. They suggested I use the lpreserver command line. So I issued these commands: sudo lpreserver replicate init NCC74602 192.168.47.20 sudo lpreserver replicate run NCC74602 192.168.47.20 Unfortunately, the replication failed again. I got these messages in the logs: Fri Jul 14 09:03:34 EDT 2017: Removing NX80101/archive/yukon.sonsofthunder.nanobit.org/ROOT - re-created locally cannot unmount '/mnt/NX80101/archive/yukon.sonsofthunder.nanobit.org/ROOT': Operation not permitted Failed creating remote dataset! cannot create 'NX80101/archive/yukon.sonsofthunder.nanobit.org/ROOT': dataset already exists It turned out there were a number of children. I logged into luna (the FreeNAS) and issued this command as root: zfs destroy -r NX80101/archive/defiant.sonsofthunder.nanobit.org I then ran the replicate init and replicate run commands again from the TrueOS host, and replication worked! It has continued to work too, at least until the next fiddly bit breaks. Kernel relinking status from Theo de Raadt (http://undeadly.org/cgi?action=article&sid=20170701170044) As you may have heard (and as was mentioned in an earlier article), on recent OpenBSD snapshots we have KARL, which means that the kernel is relinked so each boot comes with a new kernel where all .o files are linked in random order and with random offsets. Theo de Raadt summarized the status in a message to the tech@ mailing list, subject kernel relinking as follows: 5 weeks ago at d2k17 I started work on randomized kernels. I've been having conversations with other developers for nearly 5 years on the topic... but never got off to a good start, probably because I was trying to pawn the work off on others. Having done this, I really had no idea we'd end up where we are today. Here are the behaviours: The base set has grown, it now includes a link-kit At install time, a unique kernel is built from this link-kit At upgrade time, a unique kernel is built At boot time, a unique kernel is built and installed for the next boot If someone compiles their own kernel and uses 'make install', the link-kit is also updated, so that the next boot can do the work If a developer cp's a kernel to /, the mechanism dis-engages until a 'make install" or upgrade is performed in the future. That may help debugging. A unique kernel is linked such that the startup assembly code is kept in the same place, followed by randomly-sized gapping, followed by all the other .o files randomly re-organized. As a result the distances between functions and variables are entirely new. An info leak of a pointer will not disclose other pointers or objects. This may also help reduce gadgets on variable-sized architectures, because polymorphism in the instruction stream is damaged by nested offsets changing. At runtime, the kernel can unmap it's startup code. On architectures where an unmap isn't possible due to large-PTE use, the code can be trashed instead. I did most of the kernel work on amd64 first, then i386. I explained what needed to be done to visa and patrick, who handled the arm and mips platforms. Then I completed all the rest of the architectures. Some architecture are missing the unmap/trashing of startup code, that's a step we need to get back to. The next part was tricky and I received assistance from tb and rpe. We had to script the behaviours at build time, snapshot time, relink time, boot time, and later on install/upgrade time also. While they helped, I also worked to create the "gap.o" file without use of a C compiler so that relinks can occur without the "comp" set installed. This uses the linkscript feature of ld. I don't think anyone has done this before. It is little bit fragile, and the various linkers may need to receive some fixes due to what I've encountered. To ensure this security feature works great in the 6.2 release, please test snapshots. By working well, I mean it should work invisibly, without any glitch such as a broken kernel or anything. If there are ugly glitches we would like to know before the release. You heard the man, now is the time to test and get a preview of the new awsomeness that will be in OpenBSD 6.2. Beastie Bits Beta Undeadly call for testing (http://undeadly.org/cgi?action=article&sid=20170704122507) Absolute FreeBSD 3rd Edition Update (https://blather.michaelwlucas.com/archives/2972) New home for the NetBSD repository conversion (http://mail-index.netbsd.org/tech-repository/2017/06/10/msg000637.html) TrueOS unstable Update: 7/14/17 (https://www.trueos.org/blog/trueos-unstable-update-71417/) Interview with George Neville-Neil - President of the FreeBSD Foundation (https://www.mappingthejourney.com/single-post/2017/07/06/Episode-4-Interview-with-George-Neville-Neil-President-of-FreeBSD) LibreSSL 2.5.5, 2.6.0 released (https://marc.info/?l=openbsd-announce&m=149993703415746) *** Feedback/Questions Jason - Byhve VM UEFI woes (http://dpaste.com/30EY7GZ#wrap) Donald - Several Questions (http://dpaste.com/39X6YSQ#wrap) Dan - Several Questions (http://dpaste.com/3B50ZRV#wrap) Bryson - Jails (http://dpaste.com/08C43XN#wrap) *** Final Note We’ve decided to do something different for a change of pace and let you the audience interview Allan and Benedict. Getting our entire audience live when we record would be a challenge, so instead we want you to send in your questions for Allan and Benedict. This interview is not going to be like our typical support feedback questions. This is a chance for you, our audience, to ask Allan and Benedict any questions that you’ve been wondering over the years. Questions like… “Of all the conferences you've gone to, what was your favorite and why?” We will answer the questions during a random week during the month of September. Send all your questions to feedback@bsdnow.tv with the subject of “viewer interview questions” We reserve the right to not answer questions which we feel are inappropriate or trolling. ***
202: Brokering Bind
We look at an OpenBSD setup on a new laptop, revel in BSDCan trip reports, and visit daemons and friendly ninjas. This episode was brought to you by Headlines OpenBSD and the modern laptop (http://bsdly.blogspot.de/2017/07/openbsd-and-modern-laptop.html) Peter Hansteen has a new blog post about OpenBSD (http://www.openbsd.org/) on laptops: Did you think that OpenBSD is suitable only for firewalls and high-security servers? Think again. Here are my steps to transform a modern mid to high range laptop into a useful Unix workstation with OpenBSD. One thing that never ceases to amaze me is that whenever I'm out and about with my primary laptop at conferences and elsewhere geeks gather, a significant subset of the people I meet have a hard time believing that my laptop runs OpenBSD, and that it's the only system installed. and then it takes a bit of demonstrating that yes, the graphics runs with the best available resolution the hardware can offer, the wireless network is functional, suspend and resume does work, and so forth. And of course, yes, I do use that system when writing books and articles too. Apparently heavy users of other free operating systems do not always run them on their primary workstations. Peter goes on to describe the laptops he’s had over the years (all running OpenBSD) and after BSDCan 2017, he needed a new one due to cracks in the display. So the time came to shop around for a replacement. After a bit of shopping around I came back to Multicom, a small computers and parts supplier outfit in rural Åmli in southern Norway, the same place I had sourced the previous one. One of the things that attracted me to that particular shop and their own-branded offerings is that they will let you buy those computers with no operating system installed. That is of course what you want to do when you source your operating system separately, as we OpenBSD users tend to do. The last time around I had gone for a "Thin and lightweight" 14 inch model (Thickness 20mm, weight 2.0kg) with 16GB RAM, 240GB SSD for system disk and 1TB HD for /home (since swapped out for a same-size SSD, as the dmesg will show). Three years later, the rough equivalent with some added oomph for me to stay comfortable for some years to come ended me with a 13.3 inch model, 18mm and advertised as 1.3kg (but actually weighing in at 1.5kg, possibly due to extra components), 32GB RAM, 512GB SSD and 2TB harddisk. For now the specification can be viewed online here (https://www.multicom.no/systemconfigurator.aspx?q=st:10637291;c:100559;fl:0#4091-10500502-1;4086-10637290-1;4087-8562157-2;4088-9101982-1;4089-9101991-1) (the site language is Norwegian, but product names and units of measure are not in fact different). The OpenBSD installer is a wonder of straightforward, no-nonsense simplicity that simply gets the job done. Even so, if you are not yet familiar with OpenBSD, it is worth spending some time reading the OpenBSD FAQ's installation guidelines and the INSTALL.platform file (in our case, INSTALL.amd64) to familiarize yourself with the procedure. If you're following this article to the letter and will be installing a snapshot, it is worth reading the notes on following -current too. The main hurdle back when I was installing the 2014-vintage 14" model was getting the system to consider the SSD which showed up as sd1 the automatic choice for booting (I solved that by removing the MBR, setting the size of the MBR on the hard drive that showed up as sd0 to 0 and enlarging the OpenBSD part to fill the entire drive). + He goes on to explain the choices he made in the installer and settings made after the reboot to set up his work environment. Peter closes with: If you have any questions on running OpenBSD as a primary working environment, I'm generally happy to answer but in almost all cases I would prefer that you use the mailing lists such as misc@openbsd.org or the OpenBSD Facebook (https://www.facebook.com/groups/2210554563/) group so the question and hopefully useful answers become available to the general public. Browsing the slides for my recent OpenBSD and you (https://home.nuug.no/~peter/openbsd_and_you/) user group talk might be beneficial if you're not yet familiar with the system. And of course, comments on this article are welcome. BSDCan 2017 Trip Report: Roller Angel (https://www.freebsdfoundation.org/blog/2017-bsdcan-trip-report-roller-angel/) We could put this into next week’s show, because we have another trip report already that’s quite long. After dropping off my luggage, I headed straight over to the Goat BoF which took place at The Royal Oak. There were already a number of people there engaged in conversation with food and drink. I sat down at a table and was delighted that the people sitting with me were also into the BSD’s and were happy to talk about it the whole time. I felt right at home from the start as people were very nice to me, and were interested in what I was working on. I honestly didn’t know that I would fit in so well. I had a preconceived notion that people may be a bit hard to approach as they are famous and so technically advanced. At first, people seemed to only be working in smaller circles. Once you get more familiar with the faces, you realize that these circles don’t always contain the same people and that they are just people talking about specific topics. I found that it was easy to participate in the conversation and also found out that people are happy to get your feedback on the subject as well. I was actually surprised how easily I got along with everyone and how included I felt in the activities. I volunteered to help wherever possible and got to work on the video crew that recorded the audio and slides of the talks. The people at BSDCan are incredibly easy to talk to, are actually interested in what you’re doing with BSD, and what they can do to help. It’s nice to feel welcome in the community. It’s like going home. Dan mentioned in his welcome on the first day of BSDCan that the conference is like home for many in the community. The trip report is very detailed and chronicles the two days of the developer summit, and the two days of the conference There was some discussion about a new code of conduct by Benno Rice who mentioned that people are welcome to join a body of people that is forming that helps work out issues related to code of conduct and forwards their recommendations on to core. Next, Allan introduced the idea of creating a process for formally discussing big project changes or similar discussions that is going to be known as FCP or FreeBSD Community Proposal. In Python we have the Python Enhancement Proposal or PEP which is very similar to the idea of FCP. I thought this idea is a great step for FreeBSD to be implementing as it has been a great thing for Python to have. There was some discussion about taking non-code contributions from people and how to recognize those people in the project. There was a suggestion to have a FreeBSD Member status created that can be given to people whose non-code contributions are valuable to the project. This idea seemed to be on a lot of people’s minds as something that should be in place soon. The junior jobs on the FreeBSD Wiki were also brought up as a great place to look for ideas on how to get involved in contributing to FreeBSD. Roller wasted no time, and started contributing to EdgeBSD at the conference. On the first day of BSDCan I arrived at the conference early to coordinate with the team that records the talks. We selected the rooms that each of us would be in to do the recording and set up a group chat via WhatsApp for coordination. Thanks to Roller, Patrick McAvoy, Calvin Hendryx-Parker, and all of the others who volunteered their time to run the video and streaming production at BSDCan, as well as all others who volunteered, even if it was just to carry a box. BSDCan couldn’t happen without the army of volunteers. After the doc lounge, I visited the Hacker Lounge. There were already several tables full of people talking and working on various projects. In fact, there was a larger group of people who were collaborating on the new libtrue library that seemed to be having a great time. I did a little socializing and then got on my laptop and did some more work on the documentation using my new skills. I really enjoyed having a hacker lounge to go to at night. I want to give a big thank you to the FreeBSD Foundation for approving my travel grant. It was a great experience to meet the community and participate in discussions. I’m very grateful that I was able to attend my first BSDCan. After visiting the doc lounge a few times, I managed to get comfortable using the tools required to edit the documentation. By the end of the conference, I had submitted two documentation patches to the FreeBSD Bugzilla with several patches still in progress. Prior to the conference I expected that I would be spending a lot of time working on my Onion Omega and Edge Router Lite projects that I had with me, but I actually found that there was always something fun going on that I would rather do or work on. I can always work on those projects at home anyway. I had a good time working with the FreeBSD community and will continue working with them by editing the documentation and working with Bugzilla. One of the things I enjoy about these trip reports is when they help convince other people to make the trip to their first conference. Hopefully by sharing their experience, it will convince you to come to the next conference: vBSDCon in Virginia, USA: Sept 7-9 EuroBSDCon in Paris, France: Sept 21-24 BSDTW in Taipei, Taiwan: November 11-12 (CFP ends July 31st) *** BSDCan 2017 - Trip report double-p (http://undeadly.org/cgi?action=article&sid=20170629150641) Prologue Most overheard in Tokyo was "see you in Ottawaaaaah", so with additional "personal item" being Groff I returned home to plan the trip to BSDCan. Dan was very helpful with getting all the preparations (immigration handling), thanks for that. Before I could start, I had to fix something: the handling of the goat. With a nicely created harness, I could just hang it along my backpack. Done that it went to the airport of Hamburg and check-in for an itinerary of HAM-MUC-YUL. While the feeder leg was a common thing, boarding to YUL was great - cabin-crew likes Groff :) Arriving in Montreal was like entering a Monsoon zone or something, sad! After the night the weather was still rain-ish but improving and i shuttled to Dorval VIARail station to take me to Ottawa (ever avoid AirCanada, right?). Train was late, but the conductor (or so) was nice to talk to - and wanted to know about Groff's facebook page :-P. Picking a cab in Ottawa to take me to "Residence" was easy at first - just that it was the wrong one. Actually my fault and so I had a "nice, short" walk to the actual one in the rain with wrong directions. Eventually I made it and after unpacking, refreshment it was time to hit the Goat BOF! Day 1 Since this was my first BSDCan I didnt exactly knew what to expect from this BOF. But it was like, we (Keeper, Dan, Allan, ..) would talk about "who's next" and things like that. How mistaken I was :). Besides the sheer amount of BSD people entering the not-so-yuuge Oak some Dexter sneaked in camouflage. The name-giver got a proper position to oversee the mess and I was glad I did not leave him behind after almost too many Creemores. Day 2 Something happened it's crystal blue on the "roof" and sun is trying its best to wake me up. To start the day, I pick breakfast at 'Father+Sons' - I can really recommend that. Very nice home made fries (almost hashbrowns) and fast delivery! Stuffed up I trott along to get to phessler's tutorial about BGP-for-sysadmins-and-developers. Peter did a great job, but the "lab" couldn't happen, since - oh surprise - the wifi was sluggish as hell. Must love the first day on a conference every time. Went to Hackroom in U90 afterwards, just to fix stuff "at home". IPsec giving pains again. Time to pick food+beer afterwards and since it's so easy to reach, we went to the Oak again. Having a nice backyard patio experience it was about time to meet new people. Cheers to Tom, Aaron, Nick, Philip and some more, we'd an awesome night there. I also invited some not-really-computer local I know by other means who was completly overwhelmed by what kind of "nerds" gather around BSD. He planned to stay "a beer" - and it was rather some more and six hours. Looks like "we" made some impression on him :). Day 3 Easy day, no tutorials at hand, so first picking up breakfast at F+S again and moving to hackroom in U90. Since I promised phessler to help with an localized lab-setup, I started to hack on a quick vagrant/ansible setup to mimic his BGP-lab and went quickly through most of it. Plus some more IPsec debugging and finally fixing it, we went early in the general direction of the Red Lion to pick our registration pack. But before that could happen it was called to have shawarma at 3brothers along. Given a tight hangover it wasn't the brightest idea to order a poutine m-(. Might be great the other day, it wasn't for me at the very time and had to throw away most of it :(. Eventually passing on to the Red Lion I made the next failure with just running into the pub - please stay at the front desk until "seated". I never get used to this concept. So after being "properly" seated, we take our beers and the registration can commence after we had half of it. So I register myself; btw it's a great idea to grant "not needed" stuff to charity. So dont pick "just because", think about it if you really need this or that gadget. Then I register Groff - he really needs badges - just to have Dru coming back to me some minutes later one to hand me the badge for Henning. That's just "amazing"; I dont know IF i want to break this vicious circle the other day, since it's so funny. Talked to Theo about the ongoing IPsec problems and he taught me about utrace(2) which looks "complicated" but might be an end of the story the other day. Also had a nice talk to Peter (H.) about some other ideas along books. BTW, did I pay for ongoing beers? I think Tom did - what a guy :). Arriving at the Residence, I had to find my bathroom door locked (special thing).. crazy thing is they dont have a master key at the venue, but to have to call in one from elsewhere. Short night shortened by another 30minutes :(. Day 4 Weather is improving into beach+sun levels - and it's Conference Day! The opening keynote from Geist was very interesting ("citation needed"). Afterwards I went to zfs-over-ssh, nothing really new (sorry Allan). But then Jason had a super interesting talk on how about to apply BSD for the health-care system in Australia. I hope I can help him with the last bits (rdomain!) in the end. While lunch I tried to recall my memories about utrace(2) while talking to Theo. Then it was about to present my talk and I think it was well perceipted. One "not so good" feedback was about not taking the audience more into account. I think I was asking every other five slides or so - but, well. The general feedback (in spoken terms) was quite good. I was a bit "confused" and I did likely a better job in Tokyo, but well. Happened we ended up in the Oak again.. thanks to mwl, shirkdog, sng, pitrh, kurtm for having me there :) Day 5 While the weather had to decide "what next", I rushed to the venue just to gather Reyk's talk about vmd(8). Afterwards it was MSTP from Paeps which was very interesting and we (OpenBSD) should look into it. Then happened BUG BOF and I invite all "coastal Germans" to cbug.de :) I had to run off for other reasons and came back to Dave's talk which was AWESOME. Following was Rod's talk.. well. While I see his case, that was very poor. The auction into closing was awesome again, and I spend $50 on a Tshirt. :) + Epilogue I totally got the exit dates wrong. So first cancel a booking of an Hotel and then rebook the train to YUL. So I have plenty of time "in the morning" to get breakfast with the local guy. After that he drives me to VIARail station and I dig into "business" cussions. Well, see you in Ottawa - or how about Paris, Taipei? Bind Broker (http://www.tedunangst.com/flak/post/bind-broker) Ted Unangst writes about an interesting idea he has He has a single big server, and lots of users who would like to share it, many want to run web servers. This would be great, but alas, archaic decisions made long ago mean that network sockets aren’t really files and there’s this weird concept of privileged ports. Maybe we could assign each user a virtual machine and let them do whatever they want, but that seems wasteful. Think of the megabytes! Maybe we could setup nginx.conf to proxy all incoming connections to a process of the user’s choosing, but that only works for web sites and we want to be protocol neutral. Maybe we could use iptables, but nobody wants to do that. What we need is a bind broker. At some level, there needs to be some kind of broker that assigns IPs to users and resolves conflicts. It should be possible to build something of this nature given just the existing unix tools we have, instead of changing system design. Then we can deploy our broker to existing systems without upgrading or disrupting their ongoing operation. The bind broker watches a directory for the creation, by users, of unix domain sockets. Then it binds to the TCP port of the same name, and transfers traffic between them. A more complete problem specification is as follows. A top level directory, which contains subdirectories named after IP addresses. Each user is assigned a subdirectory, which they have write permission to. Inside each subdirectory, the user may create unix sockets named according to the port they wish to bind to. We might assign user alice the IP 10.0.0.5 and the user bob the IP 10.0.0.10. Then alice could run a webserver by binding to net/10.0.0.5/80 and bob could run a mail server by binding to net/10.0.0.10/25. This maps IP ownership (which doesn’t really exist in unix) to the filesystem namespace (which does have working permissions). So this will be a bit different than jails. The idea is to use filesystem permissions to control which users can bind to which IP addresses and ports The broker is responsible for watching each directory. As new sockets are created, it should respond by binding to the appropriate port. When a socket is deleted, the network side socket should be closed as well. Whenever a connection is accepted on the network side, a matching connection is made on the unix side, and then traffic is copied across. A full set of example code is provided There’s no completely portable way to watch a directory for changes. I’m using a kevent extension. Otherwise we might consider a timeout and polling with fstat, or another system specific interface (or an abstraction layer over such an interface). Otherwise, if one of our mappings is ready to read (accept), we have a new connection to handle. The first half is straightforward. We accept the connection and make a matching connect call to the unix side. Then I broke out the big cheat stick and just spliced the sockets together. In reality, we’d have to set up a read/copy/write loop for each end to copy traffic between them. That’s not very interesting to read though. The full code, below, comes in at 232 lines according to wc. Minus includes, blank lines, and lines consisting of nothing but braces, it’s 148 lines of stuff that actually gets executed by the computer. Add some error handling, and working read/write code, and 200 lines seems about right. A very interesting idea. I wonder about creating a virtual file system that would implement this and maybe do a bit more to fully flesh out this idea. What do you think? *** News Roundup Daemons and friendly Ninjas (https://euroquis.nl/bobulate/?p=1600) There’s quite a lot of software that uses CMake as a (meta-)buildsystem. A quick count in the FreeBSD ports tree shows me 1110 ports (over a thousand) that use it. CMake generates buildsystem files which then direct the actual build — it doesn’t do building itself. There are multiple buildsystem-backends available: in regular usage, CMake generates Makefiles (and does a reasonable job of producing Makefiles that work for GNU Make and for BSD Make). But it can generate Ninja, or Visual Studio, and other buildsystem files. It’s quite flexible in this regard. Recently, the KDE-FreeBSD team has been working on Qt WebEngine, which is horrible. It contains a complete Chromium and who knows what else. Rebuilding it takes forever. But Tobias (KDE-FreeBSD) and Koos (GNOME-FreeBSD) noticed that building things with the Ninja backend was considerably faster for some packages (e.g. Qt WebEngine, and Evolution data-thingy). Tobias wanted to try to extend the build-time improvements to all of the CMake-based ports in FreeBSD, and over the past few days, this has been a success. Ports builds using CMake now default to using Ninja as buildsystem-backend. Here’s a bitty table of build-times. These are one-off build times, so hardly scientifically accurate — but suggestive of a slight improvement in build time. Name Size GMake Ninja liblxt 50kB 0:32 0:31 llvm38 1655kB * 19:43 musescore 47590kB 4:00 3:54 webkit2-gtk3 14652kB 44:29 37:40 Or here’s a much more thorough table of results from tcberner@, who did 5 builds of each with and without ninja. I’ve cut out the raw data, here are just the average-of-five results, showing usually a slight improvement in build time with Ninja. Name av make av ninj Delta D/Awo compiler-rt 00:08 00:07 -00:01 -14% openjpeg 00:06 00:07 +00:01 +17% marble 01:57 01:43 -00:14 -11% uhd 01:49 01:34 -00:15 -13% opencacscade 04:08 03:23 -00:45 -18% avidemux 03:01 02:49 -00:12 – 6% kdevelop 01:43 01:33 -00:10 – 9% ring-libclient 00:58 00:53 -00:05 – 8% Not everything builds properly with Ninja. This is usually due to missing dependencies that CMake does not discover; this shows up when foo depends on bar but no rule is generated for it. Depending on build order and speed, bar may be there already by the time foo gets around to being built. Doxygen showed this, where builds on 1 CPU core were all fine, but 8 cores would blow up occasionally. In many cases, we’ve gone and fixed the missing implicit dependencies in ports and upstreams. But some things are intractable, or just really need GNU Make. For this, the FreeBSD ports infrastructure now has a knob attached to CMake for switching a port build to GNU Make. Normal: USES=cmake Out-of-source: USES=cmake:outsource GNU Make: USES=cmake:noninja gmake OoS, GMake: USES=cmake:outsource,noninja gmake Bad: USES=cmake gmake For the majority of users, this has no effect, but for our package-building clusters, and for KDE-FreeBSD developers who build a lot of CMake-buildsystem software in a day it may add up to an extra coffee break. So I’ll raise a shot of espresso to friendship between daemons and ninjas. Announcing the pkgsrc-2017Q2 release (http://mail-index.netbsd.org/pkgsrc-users/2017/07/10/msg025237.html) For the 2017Q2 release we welcome the following notable package additions and changes to the pkgsrc collection: Firefox 54 GCC 7.1 MATE 1.18 Ruby 2.4 Ruby on Rails 4.2 TeX Live 2017 Thunderbird 52.1 Xen 4.8 We say goodbye to: Ruby 1.8 Ruby 2.1 The following infrastructure changes were introduced: Implement optional new pkgtasks and init infrastructure for pkginstall. Various enhancements and fixes for building with ccache. Add support to USE_LANGUAGES for newer C++ standards. Enhanced support for SSP, FORTIFY, and RELRO. The GitHub mirror has migrated to https://github.com/NetBSD/pkgsrc In total, 210 packages were added, 43 packages were removed, and 1,780 package updates were processed since the pkgsrc-2017Q1 release. *** OpenBSD changes of note 624 (http://www.tedunangst.com/flak/post/openbsd-changes-of-note-624) There are a bunch, but here are a few that jump out: Start plugging some leaks. Compile kernels with umask 007. Install them minus read permissions. Pure preprocessor implementation of the roff .ec and .eo requests, though you are warned that very bad things will happen to anybody trying to use these macros in OpenBSD manuals. Random linking for arm64. And octeon. And alpha. And hppa. There’s some variation by platform, because every architecture has the kernel loaded with different flavors of initial physical and virtual mappings. And landisk. And loongson. And sgi. And macppc. And a gap file for sparc64, but nobody yet dares split locore. And arm7. Errata for perl File::Path race condition. Some fixes for potential link attacks against cron. Add pledge violations to acct reporting. Take random linking to the next stage. More about KARL - kernel address randomized link. As noted, a few difficulties with hibernate and such, but the plan is coming together. Add a new function reorder_kernel() that relinks and installs the new kernel in the background on system startup. Add support for the bootblocks to detect hibernate and boot the previous kernel. Remove the poorly described “stuff” from ksh. Replace usage of TIOCSTI in csh using a more common IO loop. Kind of like the stuff in ksh, but part of the default command line editing and parsing code, csh would read too many characters, then send the ones it didn’t like back into the terminal. Which is weird, right? Also, more importantly, eliminating the code that uses TIOCSTI to inject characters into ttys means that maybe TIOCSTI can be removed. Revamp some of the authentication logging in ssh. Add a verbose flag to rm so you can panic immediately upon seeing it delete the wrong file instead of waiting to discover your mistake after the fact. Update libexpat to version 2.2.1 which has some security fixes. Never trust an expat, that’s my motto. Update inteldrm to code based on Linux 4.4.70. This brings us support for Skylake and Cherryview and better support for Broadwell and Valleyview. Also adds MST support. Fun times for people with newish laptops. *** OPNsense 17.1.9 released (https://opnsense.org/opnsense-17-1-9-released/) firewall: move gateway switching from system to firewall advanced settings firewall: keep category selection when changing tabs firewall: do not skip gateway switch parsing too early (contributed by Stephane Lesimple) interfaces: show VLAN description during edit firmware: opnsense-revert can now handle multiple packages at once firmware: opnsense-patch can now handle permission changes from patches dnsmasq: use canned –bogus-priv for noprivatereverse dnsmasq: separate log file, ACL and menu entries dynamic dns: fix update for IPv6 (contributed by Alexander Leisentritt) dynamic dns: remove usage of CURLAUTH_ANY (contributed by Alexander Leisentritt) intrusion detection: suppress “fast mode available” boot warning in PCAP mode openvpn: plugin framework adaption unbound: add local-zone type transparent for PTR zone (contributed by Davide Gerhard) unbound: separate log file, ACL and menu entries wizard: remove HTML from description strings mvc: group relation to something other than uuid if needed mvc: rework “item in” for our Volt templates lang: Czech to 100% translated (contributed by Pavel Borecki) plugins: zabbix-agent 1.1 (contributed by Frank Wall) plugins: haproxy 1.16 (contributed by Frank Wall) plugins: acme-client 1.8 (contributed by Frank Wall) plugins: tinc fix for switch mode (contributed by Johan Grip) plugins: monit 1.3 (contributed by Frank Brendel) src: support dhclient supersede statement for option 54 (contributed by Fabian Kurtz) src: add Intel Atom Cherryview SOC HSUART support src: add the ID for the Huawei ME909S LTE modem src: HardenedBSD Stack Clash mitigations[1] ports: sqlite 3.19.3[2] ports: openvpn 2.4.3[3] ports: sudo 1.8.20p2[4] ports: dnsmasq 2.77[5] ports: openldap 2.4.45[6] ports: php 7.0.20[7] ports: suricata 3.2.2[8] ports: squid 3.5.26[9] ports: carootnss 3.31 ports: bind 9.11.1-P2[10] ports: unbound 1.6.3[11] ports: curl 7.54.1[12] *** Beastie Bits Thinkpad x230 - trying to get TrackPoint / Touchpad working in X (http://lists.dragonflybsd.org/pipermail/users/2017-July/313519.html) FreeBSD deprecates all r-cmds (rcp, rlogin, etc.) (http://marc.info/?l=freebsd-commits-all&m=149918307723723&w=2) Bashfill - art for your terminal (https://max.io/bash.html) Go 1.9 release notes: NetBSD support is broken, please help (https://github.com/golang/go/commit/32002079083e533e11209824bd9e3a797169d1c4) Jest, A ReST api for creating and managing FreeBSD jails written in Go (https://github.com/altsrc-io/Jest) *** Feedback/Questions John - zfs send/receive (http://dpaste.com/3ANETHW#wrap) Callum - laptops (http://dpaste.com/11TV0BJ) & An update (http://dpaste.com/3A14BQ6#wrap) Lars - Snapshot of VM datadisk (http://dpaste.com/0MM37NA#wrap) Daryl - Jail managers (http://dpaste.com/0CDQ9EK#wrap) ***
201: Skip grep, use awk
In which we interview a unicorn, FreeNAS 11.0 is out, show you how to run Nextcloud in a FreeBSD jail, and talk about the connection between oil changes and software patches. This episode was brought to you by Headlines FreeNAS 11.0 is Now Here (http://www.freenas.org/blog/freenas-11-0/) The FreeNAS blog informs us: After several FreeNAS Release Candidates, FreeNAS 11.0 was released today. This version brings new virtualization and object storage features to the World’s Most Popular Open Source Storage Operating System. FreeNAS 11.0 adds bhyve virtual machines to its popular SAN/NAS, jails, and plugins, letting you use host web-scale VMs on your FreeNAS box. It also gives users S3-compatible object storage services, which turns your FreeNAS box into an S3-compatible server, letting you avoid reliance on the cloud. FreeNAS 11.0 also introduces the beta version of a new administration GUI. The new GUI is based on the popular Angular framework and the FreeNAS team expects the GUI to be themeable and feature complete by 11.1. The new GUI follows the same flow as the existing GUI, but looks better. For now, the FreeNAS team has released it in beta form to get input from the FreeNAS community. The new GUI, as well as the classic GUI, are selectable from the login screen. Also new in FreeNAS 11 is an Alert Service page which configures the system to send critical alerts from FreeNAS to other applications and services such as Slack, PagerDuty, AWS, Hipchat, InfluxDB, Mattermost, OpsGenie, and VictorOps. FreeNAS 11.0 has an improved Services menu that adds the ability to manage which services and applications are started at boot. The FreeNAS community is large and vibrant. We invite you to join us on the FreeNAS forum (https://forums.freenas.org/index.php) and the #freenas IRC channel on Freenode. To download FreeNAS and sign-up for the FreeNAS Newsletter, visit freenas.org/download (http://www.freenas.org/download/). Building an IPsec Gateway With OpenBSD (https://www.exoscale.ch/syslog/2017/06/26/building-an-ipsec-gateway-with-openbsd/) Pierre-Yves Ritschard wrote the following blog article: With private networks just released on Exoscale, there are now more options to implement secure access to Exoscale cloud infrastructure. While we still recommend the bastion approach, as detailed in this article (https://www.exoscale.ch/syslog/2016/01/15/secure-your-cloud-computing-architecture-with-a-bastion/), there are applications or systems which do not lend themselves well to working this way. In these cases, the next best thing is building IPsec gateways. IPsec is a protocol which works directly at layer 3. It uses its configuration to determine which network flows should be sent encrypted on the wire. Once IPsec is correctly configured, selected network flows are transparently encrypted and applications do not need to modify anything to benefit from secured traffic. In addition to encryption, IPSec also authenticates the end points, so you can be sure you are exchanging packets with a trusted host For the purposes of this article we will work under the following assumptions: We want a host to network setup, providing access to cloud-hosted infrastructure from a desktop environment. Only stock tooling should be used on desktop environment, no additional VPN client should be needed. In this case, to ensure no additional software is needed on the client, we will configure an L2TP/IPsec gateway. This article will use OpenBSD as the operating system to implement the gateway. While this choice may sound surprising, OpenBSD excels at building gateways of all sorts thanks to its simple configuration formats and inclusion of all necessary software and documentation to do so in the base system. The tutorial assumes you have setup a local network between the hosts in the cloud, and walks through the configuration of an OpenBSD host as a IPsec gateway On the OpenBSD host, all necessary software is already installed. We will configure the system, as well as pf, npppd, and ipsec + Configure L2TP + Configure IPsec + Configure NAT + Enabled services: ipsec isakmpd npppd The tutorial then walks through configuring a OS X client, but other desktops will be very similar *** Running Nextcloud in a jail on FreeBSD (https://ramsdenj.com/2017/06/05/nextcloud-in-a-jail-on-freebsd.html) I recently setup Nextcloud 12 inside a FreeBSD jail in order to allow me access to files i might need while at University. I figured this would be a optimal solution for files that I might need access to unexpectedly, on computers where I am not in complete control. My Nextcloud instance is externally accessible, and yet if someone were to get inside my Jail, I could rest easy knowing they still didn’t have access to the rest of my host server. I chronicled the setup process including jail setup using iocage, https with Lets Encrypt, and full setup of the web stack. Nextcloud has a variety of features such as calendar synchronization, email, collaborative editing, and even video conferencing. I haven’t had time to play with all these different offerings and have only utilized the file synchronization, but even if file sync is not needed, Nextcloud has many offerings that make it worth setting up. MariaDB, PHP 7.0, and Apache 2.4 To manage my jails I’m using iocage. In terms of jail managers it’s a fairly new player in the game of jail management and is being very actively developed. It just had a full rewrite in Python, and while the code in the background might be different, the actual user interface has stayed the same. Iocage makes use of ZFS clones in order to create “base jails”, which allow for sharing of one set of system packages between multiple jails, reducing the amount of resources necessary. Alternatively, jails can be completely independent from each other; however, using a base jail makes it easier to update multiple jails as well. + pkg install iocage + sysrc iocageenable=YES + iocage fetch -r 11.0-RELEASE + iocage create tag="stratus" jailzfs=on vnet=off boot=on ip4_addr="sge0|172.20.0.100/32" -r 11.0-RELEASE + iocage start stratus + iocage console stratus I have chosen to provide storage to the Nextcloud Jail by mounting a dataset over NFS on my host box. This means my server can focus on serving Nextcloud and my storage box can focus on housing the data. The Nextcloud Jail is not even aware of this since the NFS Mount is simply mounted by the host server into the jail. The other benefit of this is the Nextcloud jail doesn’t need to be able to see my storage server, nor the ability to mount the NFS share itself. Using a separate server for storage isn’t necessary and if the storage for my Nextcloud server was being stored on the same server I would have created a ZFS dataset on the host and mounted it into the jail. Next I set up a dataset for the database and delegated it into the jail. Using a separate dataset allows me to specify certain properties that are better for a database, it also makes migration easier in case I ever need to move or backup the database. With most of the requirements in place it was time to start setting up Nextcloud. The requirements for Nextcloud include your basic web stack of a web server, database, and PHP. Also covers the setup of acme.sh for LetsEncrypt. This is now available as a package, and doesn’t need to be manually fetched Install a few more packages, and do a bit of configuration, and you have a NextCloud server *** Historical: My first OpenBSD Hackathon (http://bad.network/historical-my-first-openbsd-hackathon.html) This is a blog post by our friend, and OpenBSD developer: Peter Hessler This is a story about encouragement. Every time I use the word "I", you should think "I as in me, not I as in the author". In 2003, I was invited to my first OpenBSD Hackathon. Way before I was into networking, I was porting software to my favourite OS. Specifically, I was porting games. On the first night most of the hackathon attendees end up at the bar for food and beer, and I'm sitting next to Theo de Raadt, the founder of OpenBSD. At some point during the evening, he's telling me about all of these "crazy" ideas he has about randomizing libraries, and protections that can be done in ld.so. (ld.so is the part of the OS that loads the libraries your program needs. It's, uh, kinda important.) Theo is encouraging me to help implement some of these ideas! At some point I tell Theo "I'm just a porter, I don't know C." Theo responds with "It isn't hard, I'll have Dale (Rahn) show you how ld.so works, and you can do it." I was hoping that all of this would be forgotten by the next day, but sure enough Dale comes by. "Hey, are you Peter? Theo wanted me to show you how ld.so works" Dale spends an hour or two showing me how it works, the code structure, and how to recover in case of failure. At first I had lots of failures. Then more failures. And even more failures. Once, I broke my machine so badly I had to reinstall it. I learned a lot about how an OS works during this. But, I eventually started doing changes without it breaking. And some even did what I wanted! By the end of the hackathon I had came up with a useful patch, that was committed as part of a larger change. I was a nobody. With some encouragement, enough liquid courage to override my imposter syndrome, and a few hours of mentoring, I'm now doing big projects. The next time you're sitting at a table with someone new to your field, ask yourself: how can you encourage them? You just might make the world better. Thank you Dale. And thank you Theo. Everyone has to start somewhere. One of the things that sets the BSDs apart from certain other open source operating systems, is the welcoming community, and the tradition of mentorship. Sure, someone else in the OpenBSD project could have done the bits that Peter did, likely a lot more quickly, but then OpenBSD wouldn’t have gained a new committer. So, if you are interested in working on one of the BSDs, reach out, and we’ll try to help you find a mentor. What part of the system do you want to work on? *** Interview - Dan McDonald - allcoms@gmail.com (mailto:allcoms@gmail.com) (danboid) News Roundup FreeBSD 11.1-RC1 Available (https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087340.html) 11.1-RC1 Installation images are available for: amd64, i386 powerpc, powerpc64 sparc64 armv6 BANANAPI, BEAGLEBONE, CUBIEBOARD, CUBIEBOARD2, CUBOX-HUMMINGBOARD, GUMSTIX, RPI-B, RPI2, PANDABOARD, WANDBOARD aarch64 (aka arm64), including the RPI3, Pine64, OverDrive 1000, and Cavium Server A summary of changes since BETA3 includes: Several build toolchain related fixes. A use-after-free in RPC client code has been corrected. The ntpd(8) leap-seconds file has been updated. Various VM subsystem fixes. The '_' character is now allowed in newfs(8) labels. A potential sleep while holding a mutex has been corrected in the sa(4) driver. A memory leak in an ioctl handler has been fixed in the ses(4) driver. Virtual Machine Disk Images are available for the amd64 and i386 architectures. Amazon EC2 AMI Images of FreeBSD/amd64 EC2 AMIs are available The freebsd-update(8) utility supports binary upgrades of amd64 and i386 systems running earlier FreeBSD releases. Systems running earlier FreeBSD releases can upgrade as follows: freebsd-update upgrade -r 11.1-RC1 During this process, freebsd-update(8) may ask the user to help by merging some configuration files or by confirming that the automatically performed merging was done correctly. freebsd-update install The system must be rebooted with the newly installed kernel before continuing. shutdown -r now After rebooting, freebsd-update needs to be run again to install the new userland components: freebsd-update install It is recommended to rebuild and install all applications if possible, especially if upgrading from an earlier FreeBSD release, for example, FreeBSD 10.x. Alternatively, the user can install misc/compat10x and other compatibility libraries, afterwards the system must be rebooted into the new userland: shutdown -r now Finally, after rebooting, freebsd-update needs to be run again to remove stale files: freebsd-update install Oil changes, safety recalls, and software patches (http://www.daemonology.net/blog/2017-06-14-oil-changes-safety-recalls-software-patches.html) Every few months I get an email from my local mechanic reminding me that it's time to get my car's oil changed. I generally ignore these emails; it costs time and money to get this done (I'm sure I could do it myself, but the time it would cost is worth more than the money it would save) and I drive little enough — about 2000 km/year — that I'm not too worried about the consequences of going for a bit longer than nominally advised between oil changes. I do get oil changes done... but typically once every 8-12 months, rather than the recommended 4-6 months. From what I've seen, I don't think I'm alone in taking a somewhat lackadaisical approach to routine oil changes. On the other hand, there's another type of notification which elicits more prompt attention: Safety recalls. There are two good reasons for this: First, whether for vehicles, food, or other products, the risk of ignoring a safety recall is not merely that the product will break, but rather that the product will be actively unsafe; and second, when there's a safety recall you don't have to pay for the replacement or fix — the cost is covered by the manufacturer. I started thinking about this distinction — and more specifically the difference in user behaviour — in the aftermath of the "WannaCry" malware. While WannaCry attracted widespread attention for its "ransomware" nature, the more concerning aspect of this incident is how it propagated: By exploiting a vulnerability in SMB for which Microsoft issued patches two months earlier. As someone who works in computer security, I find this horrifying — and I was particularly concerned when I heard that the NHS was postponing surgeries because they couldn't access patient records. Think about it: If the NHS couldn't access patient records due to WannaCry, it suggests WannaCry infiltrated systems used to access patient records — meaning that someone else exploiting the same vulnerabilities could have accessed those records. The SMB subsystem in Windows was not merely broken; until patches were applied, it was actively unsafe. I imagine that most people in my industry would agree that security patches should be treated in the same vein as safety recalls — unless you're certain that you're not affected, take care of them as a matter of urgency — but it seems that far more users instead treat security patches more like oil changes: something to be taken care of when convenient... or not at all, if not convenient. It's easy to say that such users are wrong; but as an industry it's time that we think about why they are wrong rather than merely blaming them for their problems. There are a few factors which I think are major contributors to this problem. First, the number of updates: When critical patches occur frequently enough to become routine, alarm fatigue sets in and people cease to give the attention updates deserve, even if on a conscious level they still recognize the importance of applying updates. Colin also talks about his time as the FreeBSD Security Officer, and the problems in ensuring the patches are correct and do not break the system when installed He also points out the problem of systems like Windows Update, the combines optional updates, and things like its license checking tool, in the same interface that delivers important updates. Or my recent machines, that gets constant popups about how some security updates will not be delivered because my processor is too new. My bank sends me special offers in the mail but phones if my credit card usage trips fraud alarms; this is the sort of distinction in intrusiveness we should see for different types of software updates Finally, I think there is a problem with the mental model most people have of computer security. Movies portray attackers as geniuses who can break into any system in minutes; journalists routinely warn people that "nobody is safe"; and insurance companies offer insurance against "cyberattacks" in much the same way as they offer insurance against tornados. Faced with this wall of misinformation, it's not surprising that people get confused between 400 pound hackers sitting on beds and actual advanced persistent threats. Yes, if the NSA wants to break into your computer, they can probably do it — but most attackers are not the NSA, just like most burglars are not Ethan Hunt. You lock your front door, not because you think it will protect you from the most determined thieves, but because it's an easy step which dramatically reduces your risk from opportunistic attack; but users don't see applying security updates as the equivalent of locking their front door when they leave home. SKIP grep, use AWK (http://blog.jpalardy.com/posts/skip-grep-use-awk/) This is a tip from Jonathan Palardy in a series of blog posts about awk. It is especially helpful for people who write a lot of shell scripts or are using a lot of pipes with awk and grep. Over the years, I’ve seen many people use this pattern (filter-map): $ [data is generated] | grep something | awk '{print $2}' but it can be shortened to: $ [data is generated] | awk '/something/ {print $2}' AWK can take a regular expression (the part between the slashes) and matches that to the input. Anything that matches is being passed to the print $2 action (to print the second column). Why would I do this? I can think of 4 reasons: *it’s shorter to type *it spawns one less process *awk uses modern (read “Perl”) regular expressions, by default – like grep -E *it’s ready to “augment” with more awk How about matching the inverse (search for patterns that do NOT match)? But “grep -v” is OK… Many people have pointed out that “grep -v” can be done more concisely with: $ [data is generated] | awk '! /something/' See if you have such combinations of grep piped to awk and fix those in your shell scripts. It saves you one process and makes your scripts much more readable. Also, check out the other intro links on the blog if you are new to awk. *** vim Adventures (https://vim-adventures.com) This website, created by Doron Linder, will playfully teach you how to use vim. Hit any key to get started and follow the instructions on the playing field by moving the cursor around. There is also a menu in the bottom left corner to save your game. Try it out, increase your vim-fu, and learn how to use a powerful text editor more efficiently. *** Beastie Bits Slides from PkgSrcCon (http://pkgsrc.org/pkgsrcCon/2017/talks.html) OpenBSD’s doas adds systemd compat shim (http://marc.info/?l=openbsd-tech&m=149902196520920&w=2) Deadlock Empire -- “Each challenge below is a computer program of two or more threads. You take the role of the Scheduler - and a cunning one! Your objective is to exploit flaws in the programs to make them crash or otherwise malfunction.” (https://deadlockempire.github.io/) EuroBSDcon 2017 Travel Grant Application Now Open (https://www.freebsdfoundation.org/blog/eurobsdcon-2017-travel-grant-application-now-open/) Registration for vBSDCon is open (http://www.vbsdcon.com/) - Registration is only $100 if you register before July 31. Discount hotel rooms arranged at the Hyatt for only $100/night while supplies last. BSD Taiwan call for papers opens, closes July 31st (https://bsdtw.org/)Windows Application Versand *** Feedback/Questions Joseph - Server Monitoring (http://dpaste.com/2AM6C2H#wrap) Paulo - Updating Jails (http://dpaste.com/1Z4FBE2#wrap) Kevin - openvpn server (http://dpaste.com/2MNM9GJ#wrap) Todd - several questions (http://dpaste.com/17BVBJ3#wrap) ***
200: Getting Scrubbed to Death
The NetBSD 8.0 release process is underway, we try to measure the weight of an electron, and look at stack clashing. This episode was brought to you by Headlines NetBSD 8.0 release process underway (https://mail-index.netbsd.org/netbsd-announce/2017/06/06/msg000267.html) Soren Jacobsen writes on NetBSD-announce: If you've been reading source-changes@, you likely noticed the recent creation of the netbsd-8 branch. If you haven't been reading source-changes@, here's some news: the netbsd-8 branch has been created, signaling the beginning of the release process for NetBSD 8.0. We don't have a strict timeline for the 8.0 release, but things are looking pretty good at the moment, and we expect this release to happen in a shorter amount of time than the last couple major releases did. At this point, we would love for folks to test out netbsd-8 and let us know how it goes. A couple of major improvements since 7.0 are the addition of USB 3 support and an overhaul of the audio subsystem, including an in-kernel mixer. Feedback about these areas is particularly desired. To download the latest binaries built from the netbsd-8 branch, head to [http://daily-builds.NetBSD.org/pub/NetBSD-daily/netbsd-8/(]http://daily-builds.NetBSD.org/pub/NetBSD-daily/netbsd-8/) Thanks in advance for helping make NetBSD 8.0 a stellar release! OpenIndiana Hipster 2017.04 is here (https://www.openindiana.org/2017/05/03/openindiana-hipster-2017-04-is-here/) Desktop software and libraries Xorg was updated to 1.18.4, xorg libraries and drivers were updated. Mate was updated to 1.16 Intel video driver was updated, the list of supported hardware has significantly extended (https://wiki.openindiana.org/oi/Intel+KMS+driver) libsmb was updated to 4.4.6 gvfs was updated to 1.26.0 gtk3 was updated to 3.18.9 Major text editors were updated (we ship vim 8.0.104, joe 4.4, emacs 25.2, nano 2.7.5 pulseaudio was updated to 10.0 firefox was updated to 45.9.0 thunderbird was updated to 45.8.0 critical issue in enlightenment was fixed, now it's operational again privoxy was updated to 3.0.26 Mesa was updated to 13.0.6 Nvidia driver was updated to 340.102 Development tools and libraries GCC 6 was added. Patches necessary to compile illumos-gate with GCC 6 were added (note, compiling illumos-gate with version other than illumos-gcc-4.4.4 is not supported) GCC 7.1 added to Hipster (https://www.openindiana.org/2017/05/05/gcc-7-1-added-the-hipster-and-rolling-forward/) Bison was updated to 3.0.4 Groovy 2.4 was added Ruby 1.9 was removed, Ruby 2.3 is the default Ruby now Perl 5.16 was removed. 64-bit Perl 5.24 is shipped. 64-bit OpenJDK 8 is the default OpenJDK version now. Mercurial was updated to 4.1.3 Git was updated to 2.12.2 ccache was updated to 3.3.3 QT 5.8.0 was added Valgrind was updated to 3.12.0 Server software PostgreSQL 9.6 was added, PostgreSQL 9.3-9.5 were updated to latest minor versions MongoDB 3.4 was added MariaDB 10.1 was added NodeJS 7 was added Percona Server 5.5/5.6/5.7 and MariaDB 5.5 were updated to latest minor versions OpenVPN was updated to 2.4.1 ISC Bind was updated to 9.10.4-P8 Squid was updated to 3.5.25 Nginx was updated to 1.12.0 Apache 2.4 was updated to 2.4.25. Apache 2.4 is the default Apache server now. Apache 2.2 will be removed before the next snapshot. ISC ntpd was updated to 4.2.8p10 OpenSSH was updated to 7.4p1 Samba was updated to 4.4.12 Tcpdump was updated to 4.9.0 Snort was updated to 2.9.9.0 Puppet was updated to 3.8.6 A lot of other bug fixes and minor software updates included. *** PKGSRC at The University of Wisconsin–Milwaukee (https://uwm.edu/hpc/software-management/) This piece is from the University of Wisconsin, Milwaukee Why Use Package Managers? Why Pkgsrc? Portability Flexibility Modernity Quality and Security Collaboration Convenience Growth Binary Packages for Research Computing The University of Wisconsin — Milwaukee provides binary pkgsrc packages for selected operating systems as a service to the research computing community. Unlike most package repositories, which have a fixed prefix and frequently upgraded packages, these packages are available for multiple prefixes and remain unchanged for a given prefix. Additional packages may be added and existing packages may be patched to fix bugs or security issues, but the software versions will not be changed. This allows researchers to keep older software in-place indefinitely for long-term studies while deploying newer software in later snapshots. Contributing to Pkgsrc Building Your Own Binary Packages Check out the full article and consider using pkgsrc for your own research purposes. PKGSrc Con is this weekend! (http://www.pkgsrc.org/pkgsrcCon/2017/) *** Measuring the weight of an electron (https://deftly.net/posts/2017-06-01-measuring-the-weight-of-an-electron.html) An interesting story of the struggles of one person, aided only by their pet Canary, porting Electron to OpenBSD. This is a long rant. A rant intended to document lunacy, hopefully aid others in the future and make myself feel better about something I think is crazy. It may seem like I am making an enemy of electron, but keep in mind that isn’t my intention! The enemy here, is complexity! My friend Henry, a canary, is coming along for the ride! Getting the tools At first glance Electron seems like a pretty solid app, it has decent docs, it’s consolidated in a single repository, has a lot of visibility, porting it shouldn’t be a big deal, right? After cloning the repo, trouble starts: Reading through the doc, right off the bat there are a few interesting things: At least 25GB disk space. Huh, OK, some how this ~47M repository is going to blow up to 25G? Continuing along with the build, I know I have two versions of clang installed on OpenBSD, one from ports and one in base. Hopefully I will be able to tell the build to use one of these versions. Next, it’s time to tell the bootstrap that OpenBSD exists as a platform. After that is fixed, the build-script runs. Even though cloning another git repo fails, the build happily continues. Wait. Another repository failed to clone? At least this time the build failed after trying to clone boto.. again. I am guessing it tried twice because something might have changed between now and the last clone? Off in the distance we catch a familiar tune, it almost sounds like Gnarls Barkley’s song Crazy, can’t tell for sure. As it turns out, if you are using git-fsck, you are unable to clone boto and requests. Obviously the proper fix for his is to not care about the validity of the git objects! So we die a little inside and comment out fsckobjects in our ~/.gitconfig. Next up, chromium-58 is downloaded… Out of curiosity we look at vendor/libchromiumcontent/script/update, it seems its purpose is to download / extract chromium clang and node, good thing we already specified --clang_dir or it might try to build clang again! 544 dots and 45 minutes later, we have an error! The chromium-58.0.3029.110.tar.xz file is mysteriously not there anymore.. Interesting. Wut. “Updating Clang…”. Didn’t I explicitly say not to build clang? At this point we have to shift projects, no longer are we working on Electron.. It’s libchromiumcontent that needs our attention. Fixing sub-tools Ahh, our old friends the dots! This is the second time waiting 45+ minutes for a 500+ MB file to download. We are fairly confident it will fail, delete the file out from under itself and hinder the process even further, so we add an explicit exit to the update script. This way we can copy the file somewhere safe! Another 45 minute chrome build and saving the downloaded executable to a save space seems in order. Fixing another 50 occurrences of error conditions let’s the build continue - to another clang build. We remove the call to update_clang, because.. well.. we have two copies of it already and the Electron doc said everything would be fine if we had >= clang 3.4! More re-builds and updates of clang and chromium are being commented out, just to get somewhere close to the actual electron build. Fixing sub-sub-tools Ninja needs to be build and the script for that needs to be told to ignore this “unsupported OS” to continue. No luck. At this point we are faced with a complex web of python scripts that execute gn on GN files to produce ninja files… which then build the various components and somewhere in that cluster, something doesn’t know about OpenBSD… I look at Henry, he is looking a photo of his wife and kids. They are sitting on a telephone wire, the morning sun illuminating their beautiful faces. Henry looks back at me and says “It’s not worth it.” We slam the laptop shut and go outside. Interview - Dan McDonald - allcoms@gmail.com (mailto:allcoms@gmail.com) (danboid) News Roundup g4u 2.6 (ghosting for unix) released 18th birthday (https://mail-index.netbsd.org/netbsd-users/2017/06/08/msg019625.html) Hubert Feyrer writes in his mail to netbsd-users: After a five-year period for beta-testing and updating, I have finally released g4u 2.6. With its origins in 1999, I'd like to say: Happy 18th Birthday, g4u! About g4u: g4u ("ghosting for unix") is a NetBSD-based bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a common setup on a number of PCs using FTP. The floppy/CD offers two functions. The first is to upload the compressed image of a local harddisk to a FTP server, the other is to restore that image via FTP, uncompress it and write it back to disk. Network configuration is fetched via DHCP. As the harddisk is processed as an image, any filesystem and operating system can be deployed using g4u. Easy cloning of local disks as well as partitions is also supported. The past: When I started g4u, I had the task to install a number of lab machines with a dual-boot of Windows NT and NetBSD. The hype was about Microsoft's "Zero Administration Kit" (ZAK) then, but that did barely work for the Windows part - file transfers were slow, depended on the clients' hardware a lot (requiring fiddling with MS DOS network driver disks), and on the ZAK server the files for installing happened do disappear for no good reason every now and then. Not working well, and leaving out NetBSD (and everything else), I created g4u. This gave me the (relative) pain of getting things working once, but with the option to easily add network drivers as they appeared in NetBSD (and oh they did!), plus allowed me to install any operating system. The present: We've used g4u successfully in our labs then, booting from CDROM. I also got many donations from public and private institutions plus companies from many sectors, indicating that g4u does make a difference. In the meantime, the world has changed, and CDROMs aren't used that much any more. Network boot and USB sticks are today's devices of choice, cloning of a full disk without knowing its structure has both advantages but also disadvantages, and g4u's user interface is still command-line based with not much space for automation. For storage, FTP servers are nice and fast, but alternatives like SSH/SFTP, NFS, iSCSI and SMB for remote storage plus local storage (back to fun with filesystems, anyone? avoiding this was why g4u was created in the first place!) should be considered these days. Further aspects include integrity (checksums), confidentiality (encryption). This leaves a number of open points to address either by future releases, or by other products. The future: At this point, my time budget for g4u is very limited. I welcome people to contribute to g4u - g4u is Open Source for a reason. Feel free to get back to me for any changes that you want to contribute! The changes: Major changes in g4u 2.6 include: Make this build with NetBSD-current sources as of 2017-04-17 (shortly before netbsd-8 release branch), binaries were cross-compiled from Mac OS X 10.10 Many new drivers, bugfixes and improvements from NetBSD-current (see beta1 and beta2 announcements) Go back to keeping the disk image inside the kernel as ramdisk, do not load it as separate module. Less error prone, and allows to boot the g4u (NetBSD) kernel from a single file e.g. via PXE (Testing and documentation updates welcome!) Actually DO provide the g4u (NetBSD) kernel with the embedded g4u disk image from now on, as separate file, g4u-kernel.gz In addition to MD5, add SHA512 checksums Congratulation, g4u. Check out the g4u website (http://fehu.org/~feyrer/g4u/) and support the project if you are using it. *** Fixing FreeBSD Networking on Digital Ocean (https://wycd.net/posts/2017-05-19-fixing-freebsd-networking-on-digital-ocean.html) Most cloud/VPS providers use some form of semi-automated address assignment, rather than just regular static address configuration, so that newly created virtual machines can configure themselves. Sometimes, especially during the upgrade process, this can break. This is the story of one such user: I decided it was time to update my FreeBSD Digital Ocean droplet from the end-of-life version 10.1 (shame on me) to the modern version 10.3 (good until April 2018), and maybe even version 11 (good until 2021). There were no sensitive files on the VM, so I had put it off. Additionally, cloud providers tend to have shoddy support for BSDs, so breakages after messing with the kernel or init system are rampant, and I had been skirting that risk. The last straw for me was a broken pkg: /usr/local/lib/libpkg.so.3: Undefined symbol "openat" So the user fires up freebsd-update and upgrades to FreeBSD 10.3 I rebooted, and of course, it happened: no ssh access after 30 seconds, 1 minute, 2 minutes…I logged into my Digital Ocean account and saw green status lights for the instance, but something was definitely wrong. Fortunately, Digital Ocean provides console access (albeit slow, buggy, and crashes my browser every time I run ping). ifconfig revealed that the interfaces vtnet0 (public) and vtnet1 (private) haven’t been configured with IP addresses. Combing through files in /etc/rc.*, I found a file called /etc/rc.digitalocean.d/${DROPLETID}.conf containing static network settings for this droplet (${DROPLETID} was something like 1234567). It seemed that FreeBSD wasn’t picking up the Digital Ocean network settings config file. The quick and dirty way would have been to messily append the contents of this file to /etc/rc.conf, but I wanted a nicer way. Reading the script in /etc/rc.d/digitalocean told me that /etc/rc.digitalocean.d/${DROPLET_ID}.conf was supposed to have a symlink at /etc/rc.digitalocean.d/droplet.conf. It was broken and pointed to /etc/rc.digitalocean.d/.conf, which could happen when the curl command in /etc/rc.d/digitalocean fails Maybe the curl binary was also in need for an upgrade so failed to fetch the droplet ID Using grep to fish for files containing droplet.conf, I discovered that it was hacked into the init system via loadrcconfig() in /etc/rc.subr I would prefer if Digital Ocean had not customized the version of FreeBSD they ship quite so much I could fix that symlink and restart the services: set DROPLET_ID=$(curl -s http://169.254.169.254/metadata/v1/id) ln -s -f /etc/rc.digitalocean.d/${DROPLET_ID}.conf /etc/rc.digitalocean.d/droplet.conf /etc/rc.d/netif restart /etc/rc.d/routing restart Networking was working again, and I could then ssh into my server and run the following to finish the upgrade: freebsd-update install At this point, I decided that I didn’t want to deal with this mess again until at least 2021, so I decided to go for 11.0-RELEASE freebsd-update -r 11.0-RELEASE update freebsd-update install reboot freebsd-update install pkg-static install -f pkg pkg update pkg upgrade uname -a FreeBSD hostname 11.0-RELEASE-p9 FreeBSD 11.0-RELEASE-p9 pkg -v 1.10.1 The problem was solved correctly, and my /etc/rc.conf remains free of generated cruft. The Digital Ocean team can make our lives easier by having their init scripts do more thorough system checking, e.g., catching broken symlinks and bad network addresses. I’m hopeful that collaboration of the FreeBSD team and cloud providers will one day result in automatic fixing of these situations, or at least a correct status indicator. The Digital Ocean team didn’t really know many FreeBSD people when they made the first 10.1 images, they have improved a lot, but they of course could always use more feedback from BSD users ** Stack Clash (https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt) A 12-year-old question: "If the heap grows up, and the stack grows down, what happens when they clash? Is it exploitable? How? In 2005, Gael Delalleau presented "Large memory management vulnerabilities" and the first stack-clash exploit in user-space (against mod_php 4.3.0 on Apache 2.0.53) (http://cansecwest.com/core05/memory_vulns_delalleau.pdf) In 2010, Rafal Wojtczuk published "Exploiting large memory management vulnerabilities in Xorg server running on Linux", the second stack-clash exploit in user-space (CVE-2010-2240) (http://www.invisiblethingslab.com/resources/misc-2010/xorg-large-memory-attacks.pdf) Since 2010, security researchers have exploited several stack-clashes in the kernel-space, In user-space, however, this problem has been greatly underestimated; the only public exploits are Gael Delalleau's and Rafal Wojtczuk's, and they were written before Linux introduced a protection against stack-clashes (a "guard-page" mapped below the stack) (https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2010-2240) In this advisory, we show that stack-clashes are widespread in user-space, and exploitable despite the stack guard-page; we discovered multiple vulnerabilities in guard-page implementations, and devised general methods for: "Clashing" the stack with another memory region: we allocate memory until the stack reaches another memory region, or until another memory region reaches the stack; "Jumping" over the stack guard-page: we move the stack-pointer from the stack and into the other memory region, without accessing the stack guard-page; "Smashing" the stack, or the other memory region: we overwrite the stack with the other memory region, or the other memory region with the stack. So this advisory itself, is not a security vulnerability. It is novel research showing ways to work around the mitigations against generic vulnerability types that are implemented on various operating systems. While this issue with the mitigation feature has been fixed, even without the fix, successful exploitation requires another application with its own vulnerability in order to be exploited. Those vulnerabilities outside of the OS need to be fixed on their own. FreeBSD-Security post (https://lists.freebsd.org/pipermail/freebsd-security/2017-June/009335.html) The issue under discussion is a limitation in a vulnerability mitigation technique. Changes to improve the way FreeBSD manages stack growth, and mitigate the issue demonstrated by Qualys' proof-of-concept code, are in progress by FreeBSD developers knowledgeable in the VM subsystem. FreeBSD address space guards (https://svnweb.freebsd.org/base?view=revision&revision=320317) HardenedBSD Proof of Concept for FreeBSD (https://github.com/lattera/exploits/blob/master/FreeBSD/StackClash/001-stackclash.c) HardenedBSD implementation: https://github.com/HardenedBSD/hardenedBSD/compare/de8124d3bf83d774b66f62d11aee0162d0cd1031...91104ed152d57cde0292b2dc09489fd1f69ea77c & https://github.com/HardenedBSD/hardenedBSD/commit/00ad1fb6b53f63d6e9ba539b8f251b5cf4d40261 Qualys PoC: freebsd_cve-2017-fgpu.c (https://www.qualys.com/2017/06/19/stack-clash/freebsd_cve-2017-fgpu.c) Qualys PoC: freebsd_cve-2017-fgpe.c (https://www.qualys.com/2017/06/19/stack-clash/freebsd_cve-2017-fgpe.c) Qualys PoC: freebsd_cve-2017-1085.c (https://www.qualys.com/2017/06/19/stack-clash/freebsd_cve-2017-1085.c) Qualys PoC: OpenBSD (https://www.qualys.com/2017/06/19/stack-clash/openbsd_at.c) Qualys PoC: NetBSD (https://www.qualys.com/2017/06/19/stack-clash/netbsd_cve-2017-1000375.c) *** Will ZFS and non-ECC RAM kill your data? (http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/) TL;DR: ECC is good, but even without, having ZFS is better than not having ZFS. What’s ECC RAM? Is it a good idea? What’s ZFS? Is it a good idea? Is ZFS and non-ECC worse than not-ZFS and non-ECC? What about the Scrub of Death? The article walks through ZFS folk lore, and talks about what can really go wrong, and what is just the over-active imagination of people on the FreeNAS forums But would using any other filesystem that isn’t ZFS have protected that data? ‘Cause remember, nobody’s arguing that you can lose data to evil RAM – the argument is about whether evil RAM is more dangerous with ZFS than it would be without it. I really, really want to use the Scrub Of Death in a movie or TV show. How can I make it happen? I don’t care about your logic! I wish to appeal to authority! OK. “Authority” in this case doesn’t get much better than Matthew Ahrens, one of the cofounders of ZFS at Sun Microsystems and current ZFS developer at Delphix. In the comments to one of my filesystem articles on Ars Technica, Matthew said “There’s nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem.” Beastie Bits EuroBSDcon 2017 Travel Grant Application Now Open (https://www.freebsdfoundation.org/blog/eurobsdcon-2017-travel-grant-application-now-open/) FreeBSD 11.1-BETA3 is out, please give it a test (https://lists.freebsd.org/pipermail/freebsd-stable/2017-June/087303.html) Allan and Lacey let us know the video to the Postgresql/ZFS talk is online (http://dpaste.com/1FE80FJ) Trapsleds (https://marc.info/?l=openbsd-tech&m=149792179514439&w=2) BSD User group in North Rhine-Westphalia, Germany (https://bsd.nrw/) *** Feedback/Questions Joe - Home Server Suggestions (http://dpaste.com/2Z5BJCR#wrap) Stephen - general BSD (http://dpaste.com/1VRQYAM#wrap) Eduardo - ZFS Encryption (http://dpaste.com/2TWADQ8#wrap) Joseph - BGP Kernel Error (http://dpaste.com/0SC0GAC#wrap) ***
199: Read the source, KARL
FreeBSD 11.1-Beta1 is out, we discuss Kernel address randomized link (KARL), and explore the benefits of daily OpenBSD source code reading This episode was brought to you by Headlines FreeBSD 11.1-Beta1 now available (https://lists.freebsd.org/pipermail/freebsd-stable/2017-June/087242.html) Glen Barber, of the FreeBSD release engineering team has announced that FreeBSD 11.1-Beta1 is now available for the following architectures: 11.1-BETA1 amd64 GENERIC 11.1-BETA1 i386 GENERIC 11.1-BETA1 powerpc GENERIC 11.1-BETA1 powerpc64 GENERIC64 11.1-BETA1 sparc64 GENERIC 11.1-BETA1 armv6 BANANAPI 11.1-BETA1 armv6 BEAGLEBONE 11.1-BETA1 armv6 CUBIEBOARD 11.1-BETA1 armv6 CUBIEBOARD2 11.1-BETA1 armv6 CUBOX-HUMMINGBOARD 11.1-BETA1 armv6 GUMSTIX 11.1-BETA1 armv6 RPI-B 11.1-BETA1 armv6 RPI2 11.1-BETA1 armv6 PANDABOARD 11.1-BETA1 armv6 WANDBOARD 11.1-BETA1 aarch64 GENERIC Note regarding arm/armv6 images: For convenience for those without console access to the system, a freebsd user with a password of freebsd is available by default for ssh(1) access. Additionally, the root user password is set to root. It is strongly recommended to change the password for both users after gaining access to the system. The full schedule (https://www.freebsd.org/releases/11.1R/schedule.html) for 11.1-RELEASE is here, the final release is expected at the end of July It was also announced there will be a 10.4-RELEASE scheduled for October (https://www.freebsd.org/releases/10.4R/schedule.html) *** KARL – kernel address randomized link (https://marc.info/?l=openbsd-tech&m=149732026405941&w=2) Over the last three weeks I've been working on a new randomization feature which will protect the kernel. The situation today is that many people install a kernel binary from OpenBSD, and then run that same kernel binary for 6 months or more. We have substantial randomization for the memory allocations made by the kernel, and for userland also of course. Previously, the kernel assembly language bootstrap/runtime locore.S was compiled and linked with all the other .c files of the kernel in a deterministic fashion. locore.o was always first, then the .c files order specified by our config(8) utility and some helper files. In the new world order, locore is split into two files: One chunk is bootstrap, that is left at the beginning. The assembly language runtime and all other files are linked in random fashion. There are some other pieces to try to improve the randomness of the layout. As a result, every new kernel is unique. The relative offsets between functions and data are unique. It still loads at the same location in KVA. This is not kernel ASLR! ASLR is a concept where the base address of a module is biased to a random location, for position-independent execution. In this case, the module itself is perturbed but it lands at the same location, and does not need to use position-independent execution modes. LLDB: Sanitizing the debugger's runtime (https://blog.netbsd.org/tnf/entry/lldb_sanitizing_the_debugger_s) The good Besides the greater enhancements this month I performed a cleanup in the ATF ptrace(2) tests again. Additionally I have managed to unbreak the LLDB Debug build and to eliminate compiler warnings in the NetBSD Native Process Plugin. It is worth noting that LLVM can run tests on NetBSD again, the patch in gtest/LLVM has been installed by Joerg Sonnenberg and a more generic one has been submitted to the upstream googletest repository. There was also an improvement in ftruncate(2) on the LLVM side (authored by Joerg). Since LLD (the LLVM linker) is advancing rapidly, it improved support for NetBSD and it can link a functional executable on NetBSD. I submitted a patch to stop crashing it on startup anymore. It was nearly used for linking LLDB/NetBSD and it spotted a real linking error... however there are further issues that need to be addressed in the future. Currently LLD is not part of the mainline LLDB tasks - it's part of improving the work environment. This linker should reduce the linking time - compared to GNU linkers - of LLDB by a factor of 3x-10x and save precious developer time. As of now LLDB linking can take minutes on a modern amd64 machine designed for performance. Kernel correctness I have researched (in pkgsrc-wip) initial support for multiple threads in the NetBSD Native Process Plugin. This code revealed - when running the LLDB regression test-suite - new kernel bugs. This unfortunately affects the usability of a debugger in a multithread environment in general and explains why GDB was never doing its job properly in such circumstances. One of the first errors was asserting kernel panic with PT*STEP, when a debuggee has more than a single thread. I have narrowed it down to lock primitives misuse in the doptrace() kernel code. The fix has been committed. The bad Unfortunately this is not the full story and there is further mandatory work. LLDB acceleration The EV_SET() bug broke upstream LLDB over a month ago, and during this period the debugger was significantly accelerated and parallelized. It is difficult to declare it definitely, but it might be the reason why the tracer's runtime broke due to threading desynchronization. LLDB behaves differently when run standalone, under ktruss(1) and under gdb(1) - the shared bug is that it always fails in one way or another, which isn't trivial to debug. The ugly There are also unpleasant issues at the core of the Operating System. Kernel troubles Another bug with single-step functions that affects another aspect of correctness - this time with reliable execution of a program - is that processes die in non-deterministic ways when single-stepped. My current impression is that there is no appropriate translation between process and thread (LWP) states under a debugger. These issues are sibling problems to unreliable PTRESUME and PTSUSPEND. In order to be able to appropriately address this, I have diligently studied this month the Solaris Internals book to get a better image of the design of the NetBSD kernel multiprocessing, which was modeled after this commercial UNIX. Plan for the next milestone The current troubles can be summarized as data races in the kernel and at the same time in LLDB. I have decided to port the LLVM sanitizers, as I require the Thread Sanitizer (tsan). Temporarily I have removed the code for tracing processes with multiple threads to hide the known kernel bugs and focus on the LLDB races. Unfortunately LLDB is not easily bisectable (build time of the LLVM+Clang+LLDB stack, number of revisions), therefore the debugging has to be performed on the most recent code from upstream trunk. d2K17 Hackathon Reports d2k17 Hackathon Report: Ken Westerback on XSNOCCB removal and dhclient link detection (http://undeadly.org/cgi?action=article&sid=20170605225415) d2k17 Hackathon Report: Antoine Jacoutot on rc.d, syspatch, and more (http://undeadly.org/cgi?action=article&sid=20170608074033) d2k17 Hackathon Report: Florian Obser on slaacd(8) (http://undeadly.org/cgi?action=article&sid=20170609013548) d2k17 Hackathon Report: Stefan Sperling on USB audio, WiFi Progress (http://undeadly.org/cgi?action=article&sid=20170602014048) News Roundup Multi-tenant router or firewall with FreeBSD (https://bsdrp.net/documentation/examples/multi-tenant_router_and_firewall) Setting-up a virtual lab Downloading BSD Router Project images Download BSDRP serial image (prevent to have to use an X display) on Sourceforge. Download Lab scripts More information on these BSDRP lab scripts available on How to build a BSDRP router lab (https://bsdrp.net/documentation/examples/how_to_build_a_bsdrp_router_lab). Start the lab with full-meshed 5 routers and one shared LAN, on this example using bhyve lab script on FreeBSD: [root@FreeBSD]~# tools/BSDRP-lab-bhyve.sh -i BSDRP-1.71-full-amd64-serial.img.xz -n 5 -l 1 Configuration Router 4 (R4) hosts the 3 routers/firewalls for each 3 customers. Router 1 (R1) belongs to customer 1, router 2 (R2) to customer 2 and router 3 (R3) to customer 3. Router 5 (R5) simulates a simple Internet host Using pf firewall in place of ipfw pf need a little more configuration because by default /dev/pf is hidden from jail. Then, on the host we need to: In place of loading the ipfw/ipfw-nat modules we need to load the pf module (but still disabling pf on our host for this example) Modify default devd rules for allowing jails to see /dev/pf (if you want to use tcpdump inside your jail, you should use bpf device too) Replacing nojail tag by nojailvnet tag into /etc/rc.d/pf (already done into BSDRP (https://github.com/ocochard/BSDRP/blob/master/BSDRP/patches/freebsd.pf.rc.jail.patch)) Under the hood: jails-on-nanobsd BSDRP's tenant shell script (https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/usr/local/sbin/tenant) creates jail configuration compliant with a host running nanobsd. Then these jails need to be configured for a nanobsd: Being nullfs based for being hosted on a read-only root filesystem Have their /etc and /var into tmpfs disks (then we need to populate these directory before each start) Configuration changes need to be saved with nanobsd configuration tools, like “config save” on BSDRP And on the host: autosave daemon (https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/usr/local/sbin/autosave) need to be enabled: Each time a customer will issue a “config save” inside a jail, his configuration diffs will be save into host's /etc/jails/. And this directory is a RAM disk too, then we need to automatically save hosts configuration on changes. *** OpenBSD Daily Source Reading (https://blog.tintagel.pl/2017/06/09/openbsd-daily.html) Adam Wołk writes: I made a new year's resolution to read at least one C source file from OpenBSD daily. The goal was to both get better at C and to contribute more to the base system and userland development. I have to admit that initially I wasn’t consistent with it at all. In the first quarter of the year I read the code of a few small base utilities and nothing else. Still, every bit counts and it’s never too late to get better. Around the end of May, I really started reading code daily - no days skipped. It usually takes anywhere between ten minutes (for small base utils) and one and a half hour (for targeted reads). I’m pretty happy with the results so far. Exploring the system on a daily basis, looking up things in the code that I don’t understand and digging as deep as possible made me learn a lot more both about C and the system than I initially expected. There’s also one more side effect of reading code daily - diffs. It’s easy to spot inconsistencies, outdated code or an incorrect man page. This results in opportunities for contributing to the project. With time it also becomes less opportunitstic and more goal oriented. You might start with a https://marc.info/?l=openbsd-tech&m=149591302814638&w=2 (drive by diff to kill) optional compilation of an old compatibility option in chown that has been compiled in by default since 1995. Soon the contributions become more targeted, for example using a new API for encrypting passwords in the htpasswd utility after reading the code of the utility and the code for htpasswd handling in httpd. Similarly it can take you from discussing a doas feature idea with a friend to implementing it after reading the code. I was having a lot of fun reading code daily and started to recommend it to people in general discussions. There was one particular twitter thread that ended up starting something new. This is still a new thing and the format is not yet solidified. Generally I make a lot of notes reading code, instead of slapping them inside a local file I drop the notes on the IRC channel as I go. Everyone on the channel is encouraged to do the same or share his notes in any way he/she seems feasable. Check out the logs from the IRC discussions. Start reading code from other BSD projects and see whether you can replicate their results! *** Become FreeBSD User: Find Useful Tools (https://bsdmag.org/become-freebsd-user-find-useful-tools/) BSD Mag has the following article by David Carlier: If you’re usually programming on Linux and you consider a potential switch to FreeBSD, this article will give you an overview of the possibilities. How to Install the Dependencies FreeBSD comes with either applications from binary packages or compiled from sources (ports). They are arranged according to software types (programming languages mainly in lang (or java specifically for Java), libraries in devel, web servers in www …) and the main tool for modern FreeBSD versions is pkg, similar to Debian apt tools suite. Hence, most of the time if you are looking for a specific application/library, simply pkg search without necessarily knowing the fully qualified name of the package. It is somehow sufficient. For example pkg search php7 will display php7 itself and the modules. Furthermore, php70 specific version and so on. Web Development Basically, this is the easiest area to migrate to. Most Web languages do not use specific platform features. Thus, most of the time, your existing projects might just be “drop-in” use cases. If your language of choice is PHP, you are lucky as this scripting language is workable on various operating systems, on most Unixes and Windows. In the case of FreeBSD, you have even many different ports or binary package versions (5.6 to 7.1). In this case, you may need some specific PHP modules enabled, luckily they are available atomically, or if the port is the way you chose, it is via the www/php70-extensions’s one. Of course developing with Apache (both 2.2 and 2.4 series are available, respectively www/apache22 and www/apache24 packages), or even better with Nginx (the last stable or the latest development versions could be used, respectively www/nginx and www/nginx-devel packages) via php-fpm is possible. In terms of databases, we have the regular RDMBS like MySQL and PostgreSQL (client and server are distinct packages … databases/(mysql/portgresql)-client, and databases/(mysql/postgresql)-server). Additionally, a more modern concept of NoSQL with CouchDB, for example (databases/couchdb), MongoDB (databases/mongodb), and Cassandra (databases/cassandra), to name but a few. Low-level Development The BSDs are shipped with C and C++ compilers in the base. In the case of FreeBSD 11.0, it is clang 3.8.0 (in x86 architectures) otherwise, modern versions of gcc exist for developing with C++11. Examples are of course available too (lang/gcc … until gcc 7.0 devel). Numerous libraries for various topics are also present, web services SOAP with gsoap through User Interfaces with GTK (x11-toolkits/gtk), QT4 or QT 5 (devel/qt), malware libraries with Yara (security/yara), etc. Android / Mobile Development To be able to do Android development, to a certain degree, the Linux’s compatibility layer (aka linuxulator) needs to be enabled. Also, x11-toolkits/swt and linux-f10-gtk2 port/package need to be installed (note that libswt-gtk-3550.so and libswt-pi-gtk-3550.so are necessary. The current package is versioned as 3557 and can be solved using symlinks). In the worst case scenario, remember that bhyve (or Virtualbox) is available, and can run any Linux distribution efficiently. Source Control Management FreeBSD comes in base with a version of subversion. As FreeBSD source is in a subversion repository, a prefixed svnlite command prevents conflicts with the package/port. Additionally, Git is present but via the package/port system with various options (with or without a user interface, subversion support). Conclusion FreeBSD has made tremendous improvements over the years to fill the gap created by Linux. FreeBSD still maintains its interesting specificities; hence there will not be too much blockers if your projects are reasonably sized to allow a migration to FreeBSD. Notes from project Aeronix, part 10 (https://martin.kopta.eu/blog/#2017-06-11-16-07-26) Prologue It is almost two years since I finished building Aeronix and it has served me well during that time. Only thing that ever broke was Noctua CPU fan, which I have replaced with the same model. However, for long time, I wanted to run Aeronix on OpenBSD instead of GNU/Linux Debian. Preparation I first experimented with RAID1 OpenBSD setup in VirtualBox, plugging and unplugging drives and learned that OpenBSD RAID1 is really smooth. When I finally got the courage, I copied all the data on two drives outside of Aeronix. One external HDD I regulary use to backup Aeronix and second internal drive in my desktop computer. Copying the data took about two afternoons. Aeronix usually has higher temperatures (somewhere around 55°C or 65°C depending on time of the year), and when stressed, it can go really high (around 75°C). During full speed copy over NFS and to external drive it went as high as 85°C, which made me a bit nervous. After the data were copied, I temporarily un-configured computers on local network to not touch Aeronix, plugged keyboard, display and OpenBSD 6.1 thumb drive. Installing OpenBSD 6.1 on full disk RAID1 was super easy. Configuring NFS Aeronix serves primarily as NAS, which means NFS and SMB. NFS is used by computers in local network with persistent connection (via Ethernet). SMB is used by other devices in local network with volatile connection (via WiFi). When configuring NFS, I expected similar configuration to what I had in Debian, but on OpenBSD, it is very different. However, after reading through exports(5), it was really easy to put it together. Putting the data back Copying from the external drive took few days, since the transfer speed was something around 5MB/s. I didn't really mind. It was sort of a good thing, because Aeronix wasn't overheating that way. I guess I need to figure new backup strategy though. One interesting thing happened with one of my local desktops. It was connecting Aeronix with default NFS mount options (on Archlinux) and had really big troubles with reading anything. Basically it behaved as if the network drive had horrible access times. After changing the default mount options, it started working perfectly. Conclusion Migrating to OpenBSD was way easier than I anticipated. There are various benefits like more security, realiable RAID1 setup (which I know how will work when drive dies), better documentation and much more. However, the true benefit for me is just the fact I like OpenBSD and makes me happy to have one more OpenBSD machine. On to the next two years of service! Beastie Bits Running OpenBSD on Azure (http://undeadly.org/cgi?action=article&sid=20170609121413&mode=expanded&count=0) Mondieu - portable alternative for freebsd-update (https://github.com/skoef/mondieu) Plan9-9k: 64-bit Plan 9 (https://bitbucket.org/forsyth/plan9-9k) Installing OpenBSD 6.1 on your laptop is really hard (not) (http://sohcahtoa.org.uk/openbsd.html) UbuntuBSD is dead (http://www.ubuntubsd.org/) OPNsense 17.1.8 released (https://opnsense.org/opnsense-17-1-8-released/) *** Feedback/Questions Patrick - Operating System Textbooks (http://dpaste.com/2DKXA0T#wrap) Brian - snapshot retention (http://dpaste.com/3CJGW22#wrap) Randy - FreeNAS to FreeBSD (http://dpaste.com/2X3X6NR#wrap) Florian - Bootloader Resolution (http://dpaste.com/1AE2SPS#wrap) ***
198: BSDNorth or You can’t handle the libtruth
This episode gives you the full dose of BSDCan 2017 recap as well as a blog post on conference speaking advice. Headlines Pre-conference activities: Goat BoF, FreeBSD Foundation Board Meeting, and FreeBSD Journal Editorial Board Meeting The FreeBSD Foundation has a new President as Justin Gibbs is busy this year with building a house, so George Neville-Neil took up the task to serve as President, with Justin Gibbs as Secretary. Take a look at the updated Board of Directors (https://www.freebsdfoundation.org/about/board-of-directors/). We also have a new staff member (https://www.freebsdfoundation.org/about/staff/): Scott Lamons joined the Foundation team as senior program manager. Scott’s work for the Foundation will focus on managing and evangelizing programs for advanced technologies in FreeBSD including preparing project plans, coordinating resources, and facilitating interactions between commercial vendors, the Foundation, and the FreeBSD community. The Foundation also planned various future activities, visits of upcoming conferences, and finding new ways to support and engage the community. The Foundation now has interns in the form of co-op students from the University of Waterloo, Canada. This is described further in the May 2017 Development Projects Update (https://www.freebsdfoundation.org/blog/may-2017-development-projects-update/). Both students (Siva and Charlie) were also the conference, helping out at the Foundation table, demonstrating the tinderbox dashboard. Follow the detailed instructions (https://www.freebsdfoundation.org/news-and-events/blog/blog-post/building-a-physical-freebsd-build-status-dashboard/) to build one of your own. The Foundation put out a call for Project Proposal Solicitation for 2017 (https://www.freebsdfoundation.org/blog/freebsd-foundation-2017-project-proposal-solicitation/). If you think you have a good proposal for work relating to any of the major subsystems or infrastructure for FreeBSD, we’d be happy to review it. Don’t miss the deadlines for travel grants to some of the upcoming conferences. You can find the necessary forms and deadlines at the Travel Grant page (https://www.freebsdfoundation.org/what-we-do/travel-grants/travel-grants/) on the Foundation website. Pictures from the Goat BoF can be found on Keltia.net (https://assets.keltia.net/photos/BSDCan-2017/Royal%20Oak/index.html) Overlapping with the GoatBoF, members of the FreeBSD Journal editorial board met in a conference room in the Novotel to plan the upcoming issues. Topics were found, authors identified, and new content was discussed to appeal to even more readers. Check out the FreeBSD Journal website (https://www.freebsdfoundation.org/journal/) and subscribe if you like to support the Foundation in that way. FreeBSD Devsummit Day 1 & 2 (https://wiki.freebsd.org/DevSummit/201706) The first day of the Devsummit began with introductory slides by Gordon Tetlow, who organized the devsummit very well. Benno Rice of the FreeBSD core team presented the work done on the new Code of Conduct, which will become effective soon. A round of Q&A followed, with positive feedback from the other devsummit attendees supporting the new CoC. After that, Allan Jude joined to talk about the new FreeBSD Community Proposal (FCP) (https://github.com/freebsd/fcp) process. Modelled after IETF RFCs, Joyent RFDs, and Python PEP, it is a new way for the project to reach consensus on the design or implementation of new features or processes. The FCP repo contains FCP#0 that describes the process, and a template for writing a proposal. Then, the entire core team (except John Baldwin, who could not make it this year) and core secretary held a core Q&A session, Answering questions, gathering feedback and suggestions. After the coffee break, we had a presentation about Intel’s QAT integration in FreeBSD. When the lunch was over, people spread out into working groups about BearSSL, Transport (TCP/IP), and OpenZFS. OpenZFS working group (https://pbs.twimg.com/media/DBu_IMsWAAId2sN.jpg:large): Matt Ahrens lead the group, and spent most of the first session providing a status update about what features have been recently committed, are out for review, on the horizon, or in the design phase. Existing Features Compressed ARC Compressed Send/Recv Recently Upstreamed A recent commit improved RAID-Z write speeds by declaring writes to padding blocks to be optional, and to always write them if they can be aggregated with the next write. Mostly impacts large record sizes. ABD (ARC buffer scatter/gather) Upstreaming In Progress Native Encryption Channel Programs Device Removal (Mirrors and Stripes) Redacted Send/recv Native TRIM Support (FreeBSD has its own, but this is better and applies to all ZFS implementations) Faster (mostly sequential) scrub/resilver DRAID (A great deal of time was spent explaining how this works, with diagrams on the chalk board) vdev metadata classes (store metadata on SSDs with data is on HDDs, or similar setups. Could also be modified to do dedup to SSD) Multi-mount protection (“safe import”, for dual-headed storage shelves) zpool checkpoint (rollback an entire pool, including zfs rename and zfs destroy) Further Out Import improvements Import with missing top-level vdevs (some blocks unreadable, but might let you get some data) Improved allocator performance -- vdev spacemap log ZIL performance Persistent L2ARC ZSTD Compression Day 2 Day two started with the Have/Want/Need session for FreeBSD 12.0. A number of features that various people have or are in the process of building, were discussed with an eye towards upstreaming them. Features we want to have in time for 12.0 (early 2019) were also discussed. After the break was the Vendor summit, which continued the discussion of how FreeBSD and its vendors can work together to make a better operating system, and better products based on it After lunch, the group broke up into various working groups: Testing/CI, Containers, Hardening UFS, and GELI Improvements Allan lead the GELI Improvements session. The main thrust of the discussions was fixing an outstanding bug in GELI when using both key slots with passphrases. To solve this, and make GELI more extensible, the metadata format will be extended to allow it to store more than 512 bytes of data (currently 511 bytes are used). The new format will allow arbitrarily large metadata, defined at creation time by selecting the number of user key slots desired. The new extended metadata format will contain mostly the same fields, except the userkey will no longer be a byte array of IV-key, Data-key, HMAC, but a struct that will contain all data about that key This new format will store the number of pkcs5v2 iterations per key, instead of only having a single location to store this number for all keys (the source of the original bug) A new set of flags per key, to control some aspects of the key (does it require a keyfile, etc), as well as possibly the role of the key. An auxdata field related to the flags, this would allow a specific key with a specific flag set, to boot a different partition, rather than decrypt the main partition. A URI to external key material is also stored per key, allowing GELI to uniquely identify the correct data to load to be able to use a specific decryption key And the three original parts of the key are stored in separate fields now. The HMAC also has a type field, allowing for a different HMAC algorithm to be used in the future. The main metadata is also extended to include a field to store the number of user keys, and to provide an overall HMAC of the metadata, so that it can be verified using the master key (provide any of the user keys) Other topics discussed: Ken Merry presented sedutil, a tool for managing Self Encrypting Drives, as may be required by certain governments and other specific use cases. Creating a deniable version of GELI, where the metadata is also encrypted The work to implemented GELI in the UEFI loader was discussed, and a number of developers volunteered to review and test the code Following the end of the Dev Summit, the “Newcomers orientation and mentorship” session was run by Michael W. Lucas, which attempts to pair up first time attendees with oldtimers, to make sure they always know a few people they can ask if they have questions, or if they need help getting introduced to the right people. News Roundup Conference Day 1 (http://www.bsdcan.org/2017/schedule/day_2017-06-09.en.html) The conference opened with some short remarks from Dan Langille, and then the opening keynote by Dr Michael Geist, a law professor at the University of Ottawa where he holds the Canada Research Chair in Internet and E-commerce Law. The keynote focused on what some of the currently issues are, and how the technical community needs to get involved at all levels. In Canada especially, contacting your representatives is quite effective, and when it does not happen, they only hear the other side of the story, and often end up spouting talking points from lobbyists as if they were facts. The question period for the keynote ran well overtime because of the number of good questions the discussion raised, including how do we fight back against large telcos with teams of lawyers and piles of money. Then the four tracks of talks started up for the day The day wrapped up with the Work In Progress (WIP) session. Allan Jude presented work on ZSTD compression in ZFS Drew Gallatin presented about work at Netflix on larger mbufs, to avoid the need for chaining and to allow more data to be pushed at once. Results in an 8% CPU time reduction when pushing 90 gbps of TLS encrypted traffic Dan Langille presented about letsencrypt (the acme.sh tool specifically), and bacula Samy Al Bahra presented about Concurrency Kit *** Conference Day 2 (http://www.bsdcan.org/2017/schedule/day_2017-06-10.en.html) Because Dan is a merciful soul, BSDCan starts an hour later on the second day Another great round of talks and BoF sessions over lunch The hallway track was great as always, and I spent most of the afternoon just talking with people Then the final set of talks started, and I was torn between all four of them Then there was the auction, and the closing party *** BSDCan 2017 Auction Swag (https://blather.michaelwlucas.com/archives/2962) Groff Fundraiser Pins: During the conference, You could get a unique Groff pin, by donating more than the last person to either the FreeBSD or OpenBSD foundation Michael W. Lucas and his wife Liz donated some interesting home made and local items to the infamous Charity Auction I donated the last remaining copy of the “Canadian Edition” of “FreeBSD Mastery: Advanced ZedFS”, and a Pentium G4400 (Skylake) CPU (Supports ECC or non-ECC) Peter Hessler donated his pen (Have you read “Git Commit Murder” yet?) Theo De Raadt donated his autographed conference badge David Maxwell donated a large print of the group photo from last years FreeBSD Developers Summit, which was purchased by Allan There was also a FreeBSD Dev Summit T-Shirt (with the Slogan: What is Core doing about it?) autographed by all of the attending members of core, with a forged jhb@ signature. Lastly, someone wrote “I <3 FreeBSD” on a left over conference t-shirt with magic marker, and the bidding began to make OpenBSD developer Henning Brauer wear it to the closing party. The top bid was $150 by Kristof Provost, the FreeBSD pf maintainer. *** Henning Brauer loves FreeBSD (https://twitter.com/henningBrauer) In addition to the $150 donation that resulted in Henning wearing the I love FreeBSD t-shirt, he also took selfies with people in exchange for an additional donation of $10. A total of over $500 was raised. Michael W. Lucas (https://twitter.com/mwlauthor/status/874656462433386497) Michael Dexter (https://twitter.com/michaeldexter/status/874344686885904384) FreeBSD Foundation Interns + Ed Maste + Eric Joyner (https://twitter.com/yzgyyang/status/873714734343880705) Pierre Ponchery (https://twitter.com/khorben/status/873673295903825925) Nick Danger (https://twitter.com/niqdanger/status/873697176513380353) Michael Shirk (https://twitter.com/shirkdog/status/873687910175866881) Calvin Hendryx-Parker (https://twitter.com/calvinhp/status/873686591692255233) Reyk Floeter (https://twitter.com/reykfloeter/status/873673717884346368) Rod Grimes (https://twitter.com/akpoff/status/873673432751370240) Jim Thompson (https://twitter.com/gonzopancho/status/873700951651233792) Sean Chittenden and Sam Gwydir, Henning wearing Theo de Raadt’s badge (https://twitter.com/SeanChittenden/status/873750297113501697) David Duncan (https://twitter.com/davdunc/status/873807305162334208) *** libtrue (https://github.com/libtrue/libtrue) At the hacker lounge, a joke email was sent to the FreeBSD developers list, making it look like a change to true(1)’s manpage had been committed to “document that true(1) supports libxo” While the change was not actually made, as you might expect this started a discussion about if this was really necessary. This spawned a new github repo While this all started as a joke, it then became an example of how rapid collaboration can happen, and an example of implementing a number of modern technologies in FreeBSD, including libxo (json output), and capsicum (security sandboxing) The project has an large number of open issues and enhancement suggestions, and a number of active pull requests including: Add Vagrantfile and Ansible playbooks for VM DTrace Support (Add the trueprov provider to allow tracing of true. A Code of Conduct libtrue.xyz website as a git submodule a false binary Python and Go bindings *** On Conference Speaking (https://hynek.me/articles/speaking/) Phase 1: Idea Until now I’ve never had to sit down and ponder what I could speak about. Over the year, I run into at least one topic I know something about that appears to be interesting to the wider public. I’m positive that’s true for almost anyone if they keep their minds open and keep looking for it. Phase 2: Call for Proposals In the end I have to come up with a good pitch that speaks to as many people as possible and with a speculative outline. Since there are always many more submissions than talk slots, this is the first critical point. There are many reasons why a proposal can be refused, so put effort into not giving the program committee any additional, that are entirely avoidable. Phase 3: Waiting for the CFP Result ...do passive research: if I see something relevant, I add it to my mind map. At this point my mind map looks atypical though: it has a lot of unordered root nodes. I just throw in everything that looks interesting and add some of my own thoughts to it. In the case of the reliability topic, I spend a lot of time to stay on top it anyway so a lot of material emerged. Phase 4: The Road to an Outline Once the talk is accepted, research intensifies. Books and articles I’ve written down for further research are read and topics extracted. In 2017, this started on January 23. But before I start writing, I get the mind map into a shape that makes sense and that will support me. This is the second critical point: I have to come up with a compelling story using the material I’ve collected. Just enumerating facts and wisdom a good talk doesn’t make. It has to have a good flow that makes sense and that keeps people engaged. Phase 5: Slides I have a very strong opinion on slides: use few, big words. Don’t make people read your slides unless it’s code samples. Otherwise they’ll be distracted from what you say. You’ll see me rarely use fonts smaller than 100pt (code ~40–60pt) which is readable from everywhere and forces me to be as brief as possible. Phase 6: Polishing I firmly believe that this phase makes or breaks a talk. Only by practicing again and again you’ll notice rough spots, weak transitions, and redundancies. Each iteration makes the talk a bit better both regarding the slides and my ability to present it. Each iteration adds impressions that my subconscious mind chews on and makes things fall in place and give me inspiration in unlikely moments. Phase 7: Sneak Preview In the past years I was blessed with the opportunity to test my talks in front of a smaller audiences. Interestingly, I’ve come to take smaller events more seriously than the big ones. If a small conference pays for my travels and gives me a prominent slot, I have both more responsibility and attention than if I paid my way to an event where I’m one of many speakers. Phase 8: Refinement and More Polishing The first session is always just going through all slides, reacquainting myself with my deck. Then it always takes quite a bit of willpower until I do a full practice run again: the first time always pretty brutal because I tend to forget pretty fast. On the other hand, it rather quickly comes back too. Which makes it even harder to motivate myself to start. Ideally I’d have access to a video recording from phase 7 to have a closer look at what work could be improved. But since it’s usually smaller conferences, I don’t. Phase 9: Travel Whenever I travel to conferences I bring everything I need for my talk and then some. To make sure I don’t forget anything essential, I have a packing list for my business trips (and vacations too – the differences are so minimal that I use a unified list). My epic packing list for business trips. I print it out the day before departure and cross stuff off as I pack. I highly recommend to anyone to emulate this since packing is a lot less stressful if you just follow a checklist. Phase 10: Showtime! So this is it. The moment everything else led to. People who suffer from fear of public speaking think this is the worst part. But if you scroll back you’ll realize: this is the payoff! This is what you worked toward. This is the fun, easy part. Once you stand in front of the audience, the work is done and you get to enjoy the ride. Beastie Bits Kris and Ken Moore - TrueOS Q&A (https://www.trueos.org/blog/discourse-trueos-qa-61617/) New home for the repository conversions (NetBSD Git mirror is now on GitHub.com/NetBSD) (https://mail-index.netbsd.org/tech-repository/2017/06/10/msg000637.html) Tab completion in OpenBSD's ksh (https://deftly.net/posts/2017-05-01-openbsd-ksh-tab-complete.html) pkgsrcCon July 1&2 (http://pkgsrc.org/pkgsrcCon/2017/) OpenBSD 6.1 syspatch installed SP kernel on MP system (https://www.mail-archive.com/misc@openbsd.org/msg153421.html) KNOXBug meeting this Friday (http://knoxbug.org/content/join-us-freebsd-day) *** Feedback/Questions Rob - FreeNAS Corral (http://dpaste.com/2XEE9JA#wrap) Brad - ZFS snapshot strategy (http://dpaste.com/27GSJK0#wrap) Phil - ZFS Send via Snail Mail (http://dpaste.com/3D02RYZ#wrap) Phillip - Network Limits for Public NTP Server (http://dpaste.com/0ZSMVWH#wrap) Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv)
197: Relaying the good news
We’re at BSDCan, but we have an interview with Michael W. Lucas which you don’t want to miss. This episode was brought to you by Headlines We are off to BSDCan but we have an interview and news roundup for you. Interview - Michael W. Lucas - mwlucas@michaelwlucas.com (mailto:mwlucas@michaelwlucas.com) / @mwlauthor (https://twitter.com/mwlauthor) Books, conferences & how these two combine *** News Roundup In The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary MXes In Play (http://bsdly.blogspot.no/2012/05/in-name-of-sane-email-setting-up-spamd.html) “The Grumpy BSD Guy”, Peter Hansteen is at it again, they have produced an updated version of a full recipe for OpenBSD’s spamd for your primary AND secondary mail servers Recipes in our field are all too often offered with little or no commentary to help the user understand the underlying principles of how a specific configuration works. To counter the trend and offer some free advice on a common configuration, here is my recipe for a sane mail setup. Mailing lists can be fun. Most of the time the discussions on lists like openbsd-misc are useful, entertaining or both. But when your battle with spam fighting technology ends up blocking your source of information and entertainment (like in the case of the recent thread titled "spamd greylisting: false positives" - starting with this message), frustration levels can run high, and in the process it emerged that some readers out there place way too much trust in a certain site offering barely commented recipes (named after a rare chemical compound Cl-Hg-Hg-Cl). 4 easy steps: Make sure your MXes (both primary and secondary) are able to receive mail for your domains Set set up content filtering for all MXes, since some spambots actually speak SMTP Set up spamd in front of all MXes Set up synchronization between your spamds These are the basic steps. If you want to go even further, you can supplement your greylisting and publicly available blacklists with your own greytrapping, but greytrapping is by no means required. Once you have made sure that your mail exchangers will accept mail for your domains (checking that secondaries do receive and spool mail when you stop the SMTP service on the primary), it's time to start setting up the content filtering. The post provides links if you need help getting the basic mail server functionality going At this point you will more likely than not discover that any differences in filtering setups between the hosts that accept and deliver mail will let spam through via the weakest link. Tune accordingly, or at least until you are satisfied that you have a fairly functional configuration. As you will have read by now in the various sources I cited earlier, you need to set up rules to redirect traffic to your spamd as appropriate. Now let's take a peek at what I have running at my primary site's gateway. The articles provides a few different sets of rules The setup includes running all outgoing mail through spamd to auto-populate the whitelists, allowing replies to your emails to get through without greylisting At this point, you have seen how to set up two spamds, each running in front of a mail exchanger. You can choose to run with the default spamd.conf, or you can edit in your own customizations. There is also a link to Peter’s spamd.conf if you want to use “what works for me” The fourth and final required step for a spamd setup with backup mail exchangers it to set up synchronization between the spamds. The synchronization keeps your greylists in sync and transfers information on any greytrapped entries to the partner spamds. As the spamd man page explains, the synchronization options -y and -Y are command line options to spamd. The articles steps through the process of configuring spamd to listen for synchronization, and to send synchronization messages to its peer With these settings in place, you have more or less completed step four of our recipe. The article also shows you how to configure spamd to log to a separate log file, to make the messages easier to find and consolidate between your mail servers After noting the system load on your content filtering machines, restart your spamds. Then watch the system load values on the content filterers and take a note of them from time to time, say every 30 minutes or so Step 4) is the last required step for building a multi-MX configuration. You may want to just leave the system running for a while and watch any messages that turn up in the spamd logs or the mail exchanger's logs The final embellishment is to set up local greytrapping. The principle is simple: If you have one or more addresses in your domain that you know will never be valid, you add them to your list of trapping addresses any host that tries to deliver mail to noreply@mydomain.nx will be added to the local blacklist spamd-greytrap to be stuttered at for as long as it takes. Greytrapping can be fun, you can search for posts here tagged with the obvious keywords. To get you started, I offer up my published list of trap addresses, built mainly from logs of unsuccessful delivery attempts here, at The BSDly.net traplist page, while the raw list of trap email addresses is available here. If you want to use that list in a similar manner for your site, please do, only remember to replace the domain names with one or more that you will be receiving mail for. Let us know how this affects your inbox *** Beastie Bits Status of FreeBSD’s capsicum on Linux (http://www.capsicum-linux.org/) How to build a gateway, from 1979 (http://www.networksorcery.com/enp/ien/ien109.txt) Linux escapee Hamza Sheikh on “Why FreeBSD?” (https://bsdmag.org/why_freebsd/) UNIX is still as relevant as ever (https://blog.opengroup.org/2012/05/17/unix-is-still-as-relevant-as-ever/) Upcoming Summer 2017 FreeBSD Foundation Events (https://www.freebsdfoundation.org/blog/upcoming-summer-2017-freebsd-foundation-events/) ***
196: PostgreZFS
This week on BSD Now, we review the EuroBSDcon schedule, we explore the mysteries of Docker on OpenBSD, and show you how to run PostgreSQL on ZFS. This episode was brought to you by Headlines EuroBSDcon 2017 - Talks & Schedule published (https://2017.eurobsdcon.org/2017/05/26/talks-schedule-published/) The EuroBSDcon website was updated with the tutorial and talk schedule for the upcoming September conference in Paris, France. Tutorials on the 1st day: Kirk McKusick - An Introduction to the FreeBSD Open-Source Operating System, George Neville-Neil - DTrace for Developers, Taylor R Campbell - How to untangle your threads from a giant lock in a multiprocessor system Tutorials on the 2nd day: Kirk continues his Introduction lecture, Michael Lucas - Core concepts of ZFS (half day), Benedict Reuschling - Managing BSD systems with Ansible (half day), Peter Hessler - BGP for developers and sysadmins Talks include 3 keynotes (2 on the first day, beginning and end), another one at the end of the second day by Brendan Gregg Good mixture of talks of the various BSD projects Also, a good amount of new names and faces Check out the full talk schedule (https://2017.eurobsdcon.org/talks-schedule/). Registration is not open yet, but will be soon. *** OpenBSD on the Xiaomi Mi Air 12.5" (https://jcs.org/2017/05/22/xiaomiair) The Xiaomi Mi Air 12.5" (https://xiaomi-mi.com/notebooks/xiaomi-mi-notebook-air-125-silver/) is a basic fanless 12.5" Ultrabook with good build quality and decent hardware specs, especially for the money: while it can usually be had for about $600, I got mine for $489 shipped to the US during a sale about a month ago. Xiaomi offers this laptop in silver and gold. They also make a 13" version but it comes with an NVidia graphics chip. Since these laptops are only sold in China, they come with a Chinese language version of Windows 10 and only one or two distributors that carry them ship to the US. Unfortunately that also means they come with practically no warranty or support. Hardware > The Mi Air 12.5" has a fanless, 6th generation (Skylake) Intel Core m3 processor, 4Gb of soldered-on RAM, and a 128Gb SATA SSD (more on that later). It has a small footprint of 11.5" wide, 8" deep, and 0.5" thick, and weighs 2.3 pounds. > A single USB-C port on the right-hand side is used to charge the laptop and provide USB connectivity. A USB-C ethernet adapter I tried worked fine in OpenBSD. Whether intentional or not, a particular design touch I appreciated was that the USB-C port is placed directly to the right of the power button on the keyboard, so you don't have to look or feel around for the port when plugging in the power cable. > A single USB 3 type-A port is also available on the right side next to the USB-C port. A full-size HDMI port and a headphone jack are on the left-hand side. It has a soldered-on Intel 8260 wireless adapter and Bluetooth. The webcam in the screen bezel attaches internally over USB. > The chassis is all aluminum and has sufficient rigidity in the keyboard area. The 12.5" 1920x1080 glossy IPS screen has a fairly small bezel and while its hinge is properly weighted to allow opening the lid with one hand (if you care about that kind of thing), the screen does have a bit of top-end wobble when open, especially when typing on another laptop on the same desk. > The keyboard has a roomy layout and a nice clicky tactile with good travel. It is backlit, but with only one backlight level. When enabled via Fn+F10 (which is handled by the EC, so no OpenBSD support required), it will automatically shut off after not typing for a short while, automatically turning back once a key is pressed. Upgrades > An interesting feature of the Mi Air is that it comes with a 128Gb SATA SSD but also includes an open PCI-e slot ready to accept an NVMe SSD. > I upgraded mine with a Samsung PM961 256Gb NVMe SSD (left), and while it is possible to run with both drives in at the same time, I removed the Samsung CM871a 128Gb SATA (right) drive to save power. > The bottom case can be removed by removing the seven visible screws, in addition to the one under the foot in the middle back of the case, which just pries off. A spudger tool is needed to release all of the plastic attachment clips along the entire edge of the bottom cover. > Unfortunately this upgrade proved to be quite time consuming due to the combination of the limited UEFI firmware on the Mi Air and a bug in OpenBSD. A Detour into UEFI Firmware Variables > Unlike a traditional BIOS where one can boot into a menu and configure the boot order as well as enabling and disabling options such as "USB Hard Drive", the InsydeH2O UEFI firmware on the Xiaomi Air only provides the ability to adjust the boot order of existing devices. Any change or addition of boot devices must be done from the operating system, which is not possible under OpenBSD. > I booted to a USB key with OpenBSD on it and manually partitioned the new NVME SSD, then rsynced all of the data over from the old drive, but the laptop would not boot to the new NVME drive, instead showing an error message that there was no bootable OS. > Eventually I figured out that the GPT table that OpenBSD created on the NVMe disk was wrong due to a [one-off bug in the nvme driver](https://github.com/openbsd/src/commit/dc8298f669ea2d7e18c8a8efea509eed200cb989) which was causing the GPT table to be one sector too large, causing the backup GPT table to be written in the wrong location (and other utilities under Linux to write it over the OpenBSD area). I'm guessing the UEFI firmware would fail to read the bad GPT table on the disk that the boot variable pointed to, then declare that disk as missing, and then remove any variables that pointed to that disk. OpenBSD Support > The Mi Air's soldered-on Intel 8260 wireless adapter is supported by OpenBSD's iwm driver, including 802.11n support. The Intel sound chip is recognized by the azalia driver. > The Synaptics touchpad is connected via I2C, but is not yet supported. I am actively hacking on my dwiic driver to make this work and the touchpad will hopefully operate as a Windows Precision Touchpad via imt so I don't have to write an entirely new Synaptics driver. > Unfortunately since OpenBSD's inteldrm support that is ported from Linux is lagging quite a bit behind, there is no kernel support for Skylake and Kaby Lake video chips. Xorg works at 1920x1080 through efifb so the machine is at least usable, but X is not very fast and there is a noticeable delay when doing certain redrawing operations in xterm. Screen backlight can be adjusted through my OpenBSD port of intel_backlight. Since there is no hardware graphics support, this also means that suspend and resume do not work because nothing is available to re-POST the video after resume. Having to use efifb also makes it impossible to adjust the screen gamma, so for me, I can't use redshift for comfortable night-time hacking. Flaws > Especially taking into account the cheap price of the laptop, it's hard to find faults with the design. One minor gripe is that the edges of the case along the bottom are quite sharp, so when carrying the closed laptop, it can feel uncomfortable in one's hands. > While all of those things could be overlooked, unfortunately there is also a critical flaw in the rollover support in the keyboard/EC on the laptop. When typing certain combinations of keys quickly, such as holding Shift and typing "NULL", one's fingers may actually hold down the Shift, N, and U keys at the same time for a very brief moment before releasing N. Normally the keyboard/EC would recognize U being pressed after N is already down and send an interrupt for the U key. Unfortunately on this laptop, particular combinations of three keys do not interrupt for the third key at all until the second key is lifted, usually causing the third key not to register at all if typed quickly. I've been able to reproduce this problem in OpenBSD, Linux, and Windows, with the combinations of at least Shift+N+U and Shift+D+F. Holding Shift and typing the two characters in sequence quickly enough will usually fail to register the final character. Trying the combinations without Shift, using Control or Alt instead of Shift, or other character pairs does not trigger the problem. This might be a problem in the firmware on the Embedded Controller, or a defect in the keyboard circuitry itself. As I mentioned at the beginning, getting technical support for this machine is difficult because it's only sold in China. Docker on OpenBSD 6.1-current (https://medium.com/@dave_voutila/docker-on-openbsd-6-1-current-c620513b8110) Dave Voutila writes: So here’s the thing. I’m normally a macOS user…all my hardware was designed in Cupertino, built in China. But I’m restless and have been toying with trying to switch my daily machine over to a non-macOS system sort of just for fun. I find Linux messy, FreeBSD not as Apple-laptop-friendly as it should be, and Windows a non-starter. Luckily, I found a friend in Puffy. Switching some of my Apple machines over to dual-boot OpenBSD left a gaping hole in my workflow. Luckily, all the hard work the OpenBSD team has done over the last year seems to have plugged it nicely! OpenBSD’s hypervisor support officially made it into the 6.1 release, but after some experimentation it was rather time consuming and too fragile to get a Linux guest up and running (i.e. basically the per-requisite for Docker). Others had reported some success starting with QEMU and doing lots of tinkering, but after a wasted evening I figured I’d grab the latest OpenBSD snapshot and try what the openbsd-misc list suggested was improved Linux support in active development. 10 (11) Steps to docker are provided Step 0 — Install the latest OpenBSD 6.1 snapshot (-current) Step 1 — Configure VMM/VMD Step 2 — Grab an Alpine Linux ISO Step 3 — Make a new virtual disk image Step 4 — Boot Alpine’s ISO Step 5 — Inhale that fresh Alpine air Step 6 — Boot Alpine for Reals Step 7 — Install Docker Step 8 — Make a User Step 9 — Ditch the Serial Console Step 10 — Test out your Docker instance I haven’t done it yet, but I plan on installing docker-compose via Python’s pip package manager. I prefer defining containers in the compose files. PostgreSQL + ZFS Best Practices and Standard Procedures (https://people.freebsd.org/~seanc/postgresql/scale15x-2017-postgresql_zfs_best_practices.pdf) Slides from Sean Chittenden’s talk about PostgreSQL and ZFS at Scale 15x this spring Slides start with a good overview of Postgres and ZFS, and how to use them together To start, it walks through the basics of how PostgreSQL interacts with the filesystem (any filesystem) Then it shows the steps to take a good backup of PostgreSQL, then how to do it even better with ZFS Then an intro to ZFS, and how Copy-on-Write changes host PostgreSQL interacts with the filesystem Overview of how ZFS works ZFS Tuning tips: Compression, Recordsize, atime, when to use mostly ARC vs sharedbuffer, plus pgrepack Followed by a discussion of the reliability of SSDs, and their Bit Error Rate (BER) A good SSD has a 4%/year chance of returning the wrong data. A cheap SSD 34% If you put 20 SSDs in a database server, that means 58% (Good SSDs) to 99.975% (Lowest quality commercially viable SSD) chance of an error per year Luckily, ZFS can detect and correct these errors This applies to all storage, not just SSDs, every device fails More Advice: Use quotas and reservations to avoid running out of space Schedule Periodic Scrubs One dataset per database Backups: Live demo of rm -rf’ing the database and getting it back Using clones to test upgrades on real data Naming Conventions: Use a short prefix not on the root filesystem (e.g. /db) Encode the PostgreSQL major version into the dataset name Give each PostgreSQL cluster its own dataset (e.g. pgdb01) Optional but recommended: one database per cluster Optional but recommended: one app per database Optional but recommended: encode environment into DB name Optional but recommended: encode environment into DB username using ZFS Replication Check out the full detailed PDF and implement a similar setup for your database needs *** News Roundup TrueOS Evolving Its "Stable" Release Cycle (https://www.trueos.org/blog/housekeeping-update-infrastructure-trueos-changes/) TrueOS is reformulating its Stable branch based on feedback from users. The goal is to have a “release” of the stable branch every 6 months, for those who do not want to live on the edge with the rapid updates of the full rolling release Most of the TrueOS developers work for iX Systems in their Tennessee office. Last month, the Tennessee office was moved to a different location across town. As part of the move, we need to move all our servers. We’re still getting some of the infrastructure sorted before moving the servers, so please bear with us as we continue this process. As we’ve continued working on TrueOS, we’ve heard a significant portion of the community asking for a more stable “STABLE” release of TrueOS, maybe something akin to an old PC-BSD version release. In order to meet that need, we’re redefining the TrueOS STABLE branch a bit. STABLE releases are now expected to follow a six month schedule, with more testing and lots of polish between releases. This gives users the option to step back a little from the “cutting edge” of development, but still enjoy many of the benefits of the “rolling release” style and the useful elements of FreeBSD Current. Critical updates like emergency patches and utility bug fixes are still expected to be pushed to STABLE on a case-by-case basis, but again with more testing and polish. This also applies to version updates of the Lumina and SysAdm projects. New, released work from those projects will be tested and added to STABLE outside the 6 month window as well. The UNSTABLE branch continues to be our experimental “cutting edge” track, and users who want to follow along with our development and help us or FreeBSD test new features are still encouraged to follow the UNSTABLE track by checking that setting in their TrueOS Update Manager. With boot environments, it will be easy to switch back and forth, so you can have the best of both worlds. Use the latest bleeding edge features, but knowing you can fall back to the stable branch with just a reboot As TrueOS evolves, it is becoming clearer that one role of the system is to function as a “test platform” for FreeBSD. In order to better serve this role, TrueOS will support both OpenRC and the FreeBSD RC init systems, giving users the choice to use either system. While the full functionality isn’t quite ready for the next STABLE update, it is planned for addition after the last bit of work and testing is complete. Stay tuned for an upcoming blog post with all the details of this change, along with instructions how to switch between RC and OpenRC. This is the most important change for me. I used TrueOS as an easy way to run the latest version of -CURRENT on my laptop, to use it as a user, but also to do development. When TrueOS deviates from FreeBSD too much, it lessens the power of my expertise, and complicates development and debugging. Being able to switch back to RC, even if it takes another minute to boot, will bring TrueOS back to being FreeBSD + GUI and more by default, instead of a science project. We need both of those things, so having the option, while more work for the TrueOS team, I think will be better for the entire community *** Logical Domains on SunFire T2000 with OpenBSD/sparc64 (http://www.h-i-r.net/2017/05/logical-domains-on-sunfire-t2000-with.html) A couple of years ago, I picked up a Sun Fire T2000. This is a 2U rack mount server. Mine came with four 146GB SAS drives, a 32-core UltraSPARC T1 CPU and 32GB of RAM. Sun Microsystems incorporated Logical Domains (LDOMs) on this class of hardware. You don't often need 32 threads and 32GB of RAM in a single server. LDOMs are a kind of virtualization technology that's a bit closer to bare metal than vmm, Hyper-V, VirtualBox or even Xen. It works a bit like Xen, though. You can allocate processor, memory, storage and other resources to virtual servers on-board, with a blend of firmware that supports the hardware allocation, and some software in userland (on the so-called primary or control domain, similar to Xen DomU) to control it. LDOMs are similar to what IBM calls Logical Partitions (LPARs) on its Mainframe and POWER series computers. My day job from 2006-2010 involved working with both of these virtualization technologies, and I've kind of missed it. While upgrading OpenBSD to 6.1 on my T2000, I decided to delve into LDOM support under OpenBSD. This was pretty easy to do, but let's walk through it Resources: The ldomctl(8) man page (http://man.openbsd.org/OpenBSD-current/man8/sparc64/ldomctl.8) tedu@'s write-up on Flak (for a different class of server) (http://www.tedunangst.com/flak/post/OpenBSD-on-a-Sun-T5120) A Google+ post by bmercer@ (https://plus.google.com/101694200911870273983/posts/jWh4rMKVq97) Once you get comfortable with the fact that there's a little-tiny computer (the ALOM) powered by VXWorks inside that's acting as the management system and console (there's no screen or keyboard/mouse input), Installing OpenBSD on the base server is pretty straightforward. The serial console is an RJ-45 jack, and, yes, the ubiquitous blue-colored serial console cables you find for certain kinds of popular routers will work fine. OpenBSD installs quite easily, with the same installer you find on amd64 and i386. I chose to install to /dev/sd0, the first SAS drive only, leaving the others unused. It's possible to set them up in a hardware RAID configuration using tools available only under Solaris, or use softraid(4) on OpenBSD, but I didn't do this. I set up the primary LDOM to use the first ethernet port, em0. I decided I wanted to bridge the logical domains to the second ethernet port. You could also use a bridge and vether interface, with pf and dhcpd to create a NAT environment, similar to how I networked the vmm(4) systems. Create an LDOM configuration file. You can put this anywhere that's convenient. All of this stuff was in a "vm" subdirectory of my home. I called it ldom.conf: domain primary { vcpu 8 memory 8G } domain puffy { vcpu 8 memory 4G vdisk "/home/axon/vm/ldom1" vnet } Make as many disk images as you want, and make as many additional domain clauses as you wish. Be mindful of system resources. I couldn't actually allocate a full 32GB of RAM across all the LDOMs I eventually provisioned seven LDOMs (in addition to the primary) on the T2000, each with 3GB of RAM and 4 vcpu cores. If you get creative with use of network interfaces, virtual ethernet, bridges and pf rules, you can run a pretty complex environment on a single chassis, with services that are only exposed to other VMs, a DMZ segment, and the internal LAN. A nice tutorial, and an interesting look at an alternative platform that was ahead of its time *** documentation is thoroughly hard (http://www.tedunangst.com/flak/post/documentation-is-thoroughly-hard) Ted Unangst has a new post this week about documentation: Documentation is good, so therefore more documentation must be better, right? A few examples where things may have gotten out of control A fine example is the old OpenBSD install instructions. Once you’ve installed OpenBSD once or twice, the process is quite simple, but you’d never know this based on reading the instructions. Compare the files for 4.8 INSTALL and 5.8 INSTALL. Both begin with a brief intro to the project. Then 4.8 has an enormous list of mirrors, which seems fairly redundant if you’ve already found the install file. Followed by an enormous list of every supported variant of every supported device. Including a table of IO port configurations for ISA devices. Finally, after 1600 lines of introduction we get to the actual installation instructions. (Compared to line 231 for 5.8.) This includes a full page of text about how to install from tape, which nobody ever does. It took some time to recognize that all this documentation was actually an impediment to new users. Attempting to answer every possible question floods the reader with information for questions they were never planning to ask. Part of the problem is how the information is organized. Theoretically it makes sense to list supported hardware before instructions. After all, you can’t install anything if it’s not supported, right? I’m sure that was considered when the device list was originally inserted above the install instructions. But as a practical matter, consulting a device list is neither the easiest nor fastest way to determine what actually works. In the FreeBSD docs tree, we have been doing a facelift project, trying to add ‘quick start’ sections to each chapter to let you get to the more important information first. It is also helpful to move data in the forms of lists and tables to appendices or similar, where they can easily be references, but are not blocking your way to the information you are actually hunting for An example of nerdview signage (http://languagelog.ldc.upenn.edu/nll/?p=29866). “They have in effect provided a sign that will tell you exactly what the question is provided you can already supply the answer.” That is, the logical minds of technical people often decide to order information in an order that makes sense to them, rather than in the order that will be most useful to the reader In the end, I think “copy diskimage to USB and follow prompts” is all the instructions one should need, but it’s hard to overcome the unease of actually making the jump. What if somebody is confused or uncertain? Why is this paragraph more redundant than that paragraph? (And if we delete both, are we cutting too much?) Sometimes we don’t need to delete the information. Just hide it. The instructions to upgrade to 4.8 and upgrade to 5.8 are very similar, with a few differences because every release is a little bit different. The pages look very different, however, because the not at all recommended kernel free procedure, which takes up half the page, has been hidden from view behind some javascript and only expanded on demand. A casual browser will find the page and figure the upgrade process will be easy, as opposed to some long ordeal. This is important as well, it was my original motivation for working on the FreeBSD Handbook’s ZFS chapter. The very first section of the chapter was the custom kernel configuration required to run ZFS on i386. That scared many users away. I moved that to the very end, and started with why you might want to use ZFS. Much more approachable. Sometimes it’s just a tiny detail that’s overspecified. The apmd manual used to explain exactly which CPU idle time thresholds were used to adjust frequency. Those parameters, and the algorithm itself, were adjusted occasionally in response to user feedback, but sometimes the man page lagged behind. The numbers are of no use to a user. They’re not adjustable without recompiling. Knowing that the frequency would be reduced at 85% idle vs 90% idle doesn’t really offer much guidance as to whether to enable auto scaling or not. Deleting this detail ensured the man page was always correct and spares the user the cognitive load of trying to solve an unnecessary math problem. For fun: For another humorous example, it was recently observed that the deja-dup package provides man page translations for Australia, Canada, and Great Britain. I checked, the pages are in fact not quite identical. Some contain typo fixes that didn’t propagate to other translations. Project idea: attempt to identify which country has the most users, or most fastidious users, by bug fixes to localized man pages. lldb on BeagleBone Black (https://lists.freebsd.org/pipermail/freebsd-arm/2017-May/016260.html) I reliably managed to build (lldb + clang/lld) from the svn trunk of LLVM 5.0.0 on my Beaglebone Black running the latest snapshot (May 20th) of FreeBSD 12.0-CURRENT, and the lldb is working very well, and this includes single stepping and ncurses-GUI mode, while single stepping with the latest lldb 4.0.1 from the ports does not work. In order to reliably build LLVM 5.0.0 (svn), I set up a 1 GB swap partition for the BBB on a NFSv4 share on a FreeBSD fileserver in my network - I put a howto of the procedure on my BLog: https://obsigna.net/?p=659 The prerequesites on the Beaglebone are: ``` pkg install tmux pkg install cmake pkg install python pkg install libxml2 pkg install swig30 pkg install ninja pkg install subversion ``` On the FreeBSD fileserver: ``` /pathtothe/bbb_share svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm cd llvm/tools svn co http://llvm.org/svn/llvm-project/cfe/trunk clang svn co http://llvm.org/svn/llvm-project/lld/trunk lld svn co http://llvm.org/svn/llvm-project/lldb/trunk lldb ``` + On the Beaglebone Black: # mount_nfs -o noatime,readahead=4,intr,soft,nfsv4 server:/path_to_the/bbb_share /mnt # cd /mnt # mkdir build # cmake -DLLVM_TARGETS_TO_BUILD="ARM" -DCMAKE_BUILD_TYPE="MinSizeRel" \ -DLLVM_PARALLEL_COMPILE_JOBS="1" -DLLVM_PARALLEL_LINK_JOBS="1" -G Ninja .. I execute the actual build command from within a tmux session, so I may disconnect during the quite long (40 h) build: ``` tmux new "ninja lldb install" ``` When debugging in GUI mode using the newly build lldb 5.0.0-svn, I see only a minor issue, namely UTF8 strings are not displayed correctly. This happens in the ncurses-GUI only, and this is an ARM issue, since it does not occur on x86 machines. Perhaps this might be related to the signed/unsigned char mismatch between ARM and x86. Beastie Bits Triangle BSD Meetup on June 27th (https://www.meetup.com/Triangle-BSD-Users-Group/events/240247251/) Support for Controller Area Networks (CAN) in NetBSD (http://www.feyrer.de/NetBSD/bx/blosxom.cgi/nb_20170521_0113.html) Notes from Monday's meeting (http://mailman.uk.freebsd.org/pipermail/ukfreebsd/2017-May/014104.html) RunBSD - A site about the BSD family of operating systems (http://runbsd.info/) BSDCam(bridge) 2017 Travel Grant Application Now Open (https://www.freebsdfoundation.org/blog/bsdcam-2017-travel-grant-application-now-open/) New BSDMag has been released (https://bsdmag.org/download/nearly-online-zpool-switching-two-freebsd-machines/) *** Feedback/Questions Philipp - A show about byhve (http://dpaste.com/390F9JN#wrap) Jake - byhve Support on AMD (http://dpaste.com/0DYG5BD#wrap) CY - Pledge and Capsicum (http://dpaste.com/1YVBT12#wrap) CY - OpenSSL relicense Issue (http://dpaste.com/3RSYV23#wrap) Andy - Laptops (http://dpaste.com/0MM09EX#wrap) ***
195: I don’t WannaCry
A pledge of love to OpenBSD, combating ransomware like WannaCry with OpenZFS, and using PFsense to maximize your non-gigabit Internet connection This episode was brought to you by Headlines ino64 project committed to FreeBSD 12-CURRENT (https://svnweb.freebsd.org/base?view=revision&revision=318736) The ino64 project has been completed and merged into FreeBSD 12-CURRENT Extend the inot, devt, nlinkt types to 64-bit ints. Modify struct dirent layout to add doff, increase the size of dfileno to 64-bits, increase the size of dnamlen to 16-bits, and change the required alignment. Increase struct statfs fmntfromname[] and fmntonname[] array length MNAMELEN to 1024 This means the length of a mount point (MNAMELEN) has been increased from 88 byte to 1024 bytes. This allows longer ZFS dataset names and more nesting, and generally improves the usefulness of nested jails It also allow more than 4 billion files to be stored in a single file system (both UFS and ZFS). It also deals with a number of NFS problems, such as Amazon’s EFS (cloud NFS), which uses 64 bit IDs even with small numbers of files. ABI breakage is mitigated by providing compatibility using versioned symbols, ingenious use of the existing padding in structures, and by employing other tricks. Unfortunately, not everything can be fixed, especially outside the base system. For instance, third-party APIs which pass struct stat around are broken in backward and forward incompatible ways. A bug in poudriere that may cause some packages to not rebuild is being fixed. Many packages like perl will need to be rebuilt after this change Update note: strictly follow the instructions in UPDATING. Build and install the new kernel with COMPAT_FREEBSD11 option enabled, then reboot, and only then install new world. So you need the new GENERIC kernel with the COMPAT_FREEBSD11 option, so that your old userland will work with the new kernel, and you need to build, install, and reboot onto the new kernel before attempting to install world. The usual process of installing both and then rebooting will NOT WORK Credits: The 64-bit inode project, also known as ino64, started life many years ago as a project by Gleb Kurtsou (gleb). Kirk McKusick (mckusick) then picked up and updated the patch, and acted as a flag-waver. Feedback, suggestions, and discussions were carried by Ed Maste (emaste), John Baldwin (jhb), Jilles Tjoelker (jilles), and Rick Macklem (rmacklem). Kris Moore (kmoore) performed an initial ports investigation followed by an exp-run by Antoine Brodin (antoine). Essential and all-embracing testing was done by Peter Holm (pho). The heavy lifting of coordinating all these efforts and bringing the project to completion were done by Konstantin Belousov (kib). Sponsored by: The FreeBSD Foundation (emaste, kib) Why I love OpenBSD (https://medium.com/@h3artbl33d/why-i-love-openbsd-ca760cf53941) Jeroen Janssen writes: I do love open source software. Oh boy, I really do love open source software. It’s extendable, auditable, and customizable. What’s not to love? I’m astonished by the idea that tens, hundreds, and sometimes even thousands of enthusiastic, passionate developers collaborate on an idea. Together, they make the world a better place, bit by bit. And this leads me to one of my favorite open source projects: the 22-year-old OpenBSD operating system. The origins of my love affair with OpenBSD From Linux to *BSD The advantages of OpenBSD It’s extremely secure It’s well documented It’s open source > It’s neat and clean My take on OpenBSD ** DO ** Combating WannaCry and Other Ransomware with OpenZFS Snapshots (https://www.ixsystems.com/blog/combating-ransomware/) Ransomware attacks that hold your data hostage using unauthorized data encryption are spreading rapidly and are particularly nefarious because they do not require any special access privileges to your data. A ransomware attack may be launched via a sophisticated software exploit as was the case with the recent “WannaCry” ransomware, but there is nothing stopping you from downloading and executing a malicious program that encrypts every file you have access to. If you fail to pay the ransom, the result will be indistinguishable from your simply deleting every file on your system. To make matters worse, ransomware authors are expanding their attacks to include just about any storage you have access to. The list is long, but includes network shares, Cloud services like DropBox, and even “shadow copies” of data that allow you to open previous versions of files. To make matters even worse, there is little that your operating system can do to prevent you or a program you run from encrypting files with ransomware just as it can’t prevent you from deleting the files you own. Frequent backups are touted as one of the few effective strategies for recovering from ransomware attacks but it is critical that any backup be isolated from the attack to be immune from the same attack. Simply copying your files to a mounted disk on your computer or in the Cloud makes the backup vulnerable to infection by virtue of the fact that you are backing up using your regular permissions. If you can write to it, the ransomware can encrypt it. Like medical workers wearing hazmat suits for isolation when combating an epidemic, you need to isolate your backups from ransomware. OpenZFS snapshots to the rescue OpenZFS is the powerful file system at the heart of every storage system that iXsystems sells and of its many features, snapshots can provide fast and effective recovery from ransomware attacks at both the individual user and enterprise level as I talked about in 2015. As a copy-on-write file system, OpenZFS provides efficient and consistent snapshots of your data at any given point in time. Each snapshot only includes the precise delta of changes between any two points in time and can be cloned to provide writable copies of any previous state without losing the original copy. Snapshots also provide the basis of OpenZFS replication or backing up of your data to local and remote systems. Because an OpenZFS snapshot takes place at the block level of the file system, it is immune to any file-level encryption by ransomware that occurs over it. A carefully-planned snapshot, replication, retention, and restoration strategy can provide the low-level isolation you need to enable your storage infrastructure to quickly recover from ransomware attacks. OpenZFS snapshots in practice While OpenZFS is available on a number of desktop operating systems such as TrueOS and macOS, the most effective way to bring the benefits of OpenZFS snapshots to the largest number of users is with a network of iXsystems TrueNAS, FreeNAS Certified and FreeNAS Mini unified NAS and SAN storage systems. All of these can provide OpenZFS-backed SMB, NFS, AFP, and iSCSI file and block storage to the smallest workgroups up through the largest enterprises and TrueNAS offers available Fibre Channel for enterprise deployments. By sharing your data to your users using these file and block protocols, you can provide them with a storage infrastructure that can quickly recover from any ransomware attack thrown at it. To mitigate ransomware attacks against individual workstations, TrueNAS and FreeNAS can provide snapshotted storage to your VDI or virtualization solution of choice. Best of all, every iXsystems TrueNAS, FreeNAS Certified, and FreeNAS Mini system includes a consistent user interface and the ability to replicate between one another. This means that any topology of individual offices and campuses can exchange backup data to quickly mitigate ransomware attacks on your organization at all levels. Join us for a free webinar (http://www.onlinemeetingnow.com/register/?id=uegudsbc75) with iXsystems Co-Founder Matt Olander and learn more about why businesses everywhere are replacing their proprietary storage platforms with TrueNAS then email us at info@ixsystems.com or call 1-855-GREP-4-IX (1-855-473-7449), or 1-408-493-4100 (outside the US) to discuss your storage needs with one of our solutions architects. Interview - Michael W. Lucas - mwlucas@michaelwlucas.com (mailto:mwlucas@michaelwlucas.com) / @twitter (https://twitter.com/mwlauthor) Books, conferences, and how these two combine + BR: Welcome back. Tell us what you’ve been up to since the last time we interviewed you regarding books and such. + AJ: Tell us a little bit about relayd and what it can do. + BR: What other books do you have in the pipeline? + AJ: What are your criteria that qualifies a topic for a mastery book? + BR: Can you tell us a little bit about these writing workshops that you attend and what happens there? + AJ: Without spoiling too much: How did you come up with the idea for git commit murder? + BR: Speaking of BSDCan, can you tell the first timers about what to expect in the http://www.bsdcan.org/2017/schedule/events/890.en.html (Newcomers orientation and mentorship) session on Thursday? + AJ: Tell us about the new WIP session at BSDCan. Who had the idea and how much input did you get thus far? + BR: Have you ever thought about branching off into a new genre like children’s books or medieval fantasy novels? + AJ: Is there anything else before we let you go? News Roundup Using LLDP on FreeBSD (https://tetragir.com/freebsd/networking/using-lldp-on-freebsd.html) LLDP, or Link Layer Discovery Protocol allows system administrators to easily map the network, eliminating the need to physically run the cables in a rack. LLDP is a protocol used to send and receive information about a neighboring device connected directly to a networking interface. It is similar to Cisco’s CDP, Foundry’s FDP, Nortel’s SONMP, etc. It is a stateless protocol, meaning that an LLDP-enabled device sends advertisements even if the other side cannot do anything with it. In this guide the installation and configuration of the LLDP daemon on FreeBSD as well as on a Cisco switch will be introduced. If you are already familiar with Cisco’s CDP, LLDP won’t surprise you. It is built for the same purpose: to exchange device information between peers on a network. While CDP is a proprietary solution and can be used only on Cisco devices, LLDP is a standard: IEEE 802.3AB. Therefore it is implemented on many types of devices, such as switches, routers, various desktop operating systems, etc. LLDP helps a great deal in mapping the network topology, without spending hours in cabling cabinets to figure out which device is connected with which switchport. If LLDP is running on both the networking device and the server, it can show which port is connected where. Besides physical interfaces, LLDP can be used to exchange a lot more information, such as IP Address, hostname, etc. In order to use LLDP on FreeBSD, net-mgmt/lldpd has to be installed. It can be installed from ports using portmaster: #portmaster net-mgmt/lldpd Or from packages: #pkg install net-mgmt/lldpd By default lldpd sends and receives all the information it can gather , so it is advisable to limit what we will communicate with the neighboring device. The configuration file for lldpd is basically a list of commands as it is passed to lldpcli. Create a file named lldpd.conf under /usr/local/etc/ The following configuration gives an example of how lldpd can be configured. For a full list of options, see %man lldpcli To check what is configured locally, run #lldpcli show chassis detail To see the neighbors run #lldpcli show neighbors details Check out the rest of the article about enabling LLDP on a Cisco switch experiments with prepledge (http://www.tedunangst.com/flak/post/experiments-with-prepledge) Ted Unangst takes a crack at a system similar to the one being designed for Capsicum, Oblivious Sandboxing (See the presentation at BSDCan), where the application doesn’t even know it is in the sandbox MP3 is officially dead, so I figure I should listen to my collection one last time before it vanishes entirely. The provenance of some of these files is a little suspect however, and since I know one shouldn’t open files from strangers, I’d like to take some precautions against malicious malarkey. This would be a good use for pledge, perhaps, if we can get it working. At the same time, an occasional feature request for pledge is the ability to specify restrictions before running a program. Given some untrusted program, wrap its execution in a pledge like environment. There are other system call sandbox mechanisms that can do this (systrace was one), but pledge is quite deliberately designed not to support this. But maybe we can bend it to our will. Our pledge wrapper can’t be an external program. This leaves us with the option of injecting the wrapper into the target program via LD_PRELOAD. Before main even runs, we’ll initialize what needs initializing, then lock things down with a tight pledge set. Our eventual target will be ffplay, but hopefully the design will permit some flexibility and reuse. So the new code is injected to override the open syscall, and reads a list of files from an environment variable. Those files are opened and the path and file descriptor are put into a linked list, and then pledge is used to restrict further access to the file system. The replacement open call now searches just that linked list, returning the already opened file descriptors. So as long as your application only tries to open files that you have preopened, it can function without modification within the sandbox. Or at least that is the goal... ffplay tries to dlopen() some things, and because of the way dlopen() works, it doesn’t go via the libc open() wrapper, so it doesn’t get overridden ffplay also tries to call a few ioctl’s, not allowed After stubbing both of those out, it still doesn’t work and it is just getting worse Ted switches to a new strategy, using ffmpeg to convert the .mp3 to a .wav file and then just cat it to /dev/audio A few more stubs for ffmpeg, including access(), and adding tty access to the list of pledges, and it finally works This point has been made from the early days, but I think this exercise reinforces it, that pledge works best with programs where you understand what the program is doing. A generic pledge wrapper isn’t of much use because the program is going to do something unexpected and you’re going to have a hard time wrangling it into submission. Software is too complex. What in the world is ffplay doing? Even if I were working with the source, how long would it take to rearrange the program into something that could be pledged? One can try using another program, but I would wager that as far as multiformat media players go, ffplay is actually on the lower end of the complexity spectrum. Most of the trouble comes from using SDL as an abstraction layer, which performs a bunch of console operations. On the flip side, all of this early init code is probably the right design. Once SDL finally gets its screen handle setup, we could apply pledge and sandbox the actual media decoder. That would be the right way to things. Is pledge too limiting? Perhaps, but that’s what I want. I could have just kept adding permissions until ffplay had full access to my X socket, but what kind of sandbox is that? I don’t want naughty MP3s scraping my screen and spying on my keystrokes. The sandbox I created had all the capabilities one needs to convert an MP3 to audible sound, but the tool I wanted to use wasn’t designed to work in that environment. And in its defense, these were new post hoc requirements. Other programs, even sed, suffer from less than ideal pledge sets as well. The best summary might be to say that pledge is designed for tomorrow’s programs, not yesterday’s (and vice versa). There were a few things I could have done better. In particular, I gave up getting audio to work, even though there’s a nice description of how to work with pledge in the sio_open manual. Alas, even going back and with a bit more effort I still haven’t succeeded. The requirements to use libsndio are more permissive than I might prefer. How I Maximized the Speed of My Non-Gigabit Internet Connection (https://medium.com/speedtest-by-ookla/engineer-maximizes-internet-speed-story-c3ec0e86f37a) We have a new post from Brennen Smith, who is the Lead Systems Engineer at Ookla, the company that runs Speedtest.net, explaining how he used pfSense to maximize his internet connection I spend my time wrangling servers and internet infrastructure. My daily goals range from designing high performance applications supporting millions of users and testing the fastest internet connections in the world, to squeezing microseconds from our stack —so at home, I strive to make sure that my personal internet performance is running as fast as possible. I live in an area with a DOCSIS ISP that does not provide symmetrical gigabit internet — my download and upload speeds are not equal. Instead, I have an asymmetrical plan with 200 Mbps download and 10 Mbps upload — this nuance considerably impacted my network design because asymmetrical service can more easily lead to bufferbloat. We will cover bufferbloat in a later article, but in a nutshell, it’s an issue that arises when an upstream network device’s buffers are saturated during an upload. This causes immense network congestion, latency to rise above 2,000 ms., and overall poor quality of internet. The solution is to shape the outbound traffic to a speed just under the sending maximum of the upstream device, so that its buffers don’t fill up. My ISP is notorious for having bufferbloat issues due to the low upload performance, and it’s an issue prevalent even on their provided routers. They walk through a list of router devices you might consider, and what speeds they are capable of handling, but ultimately ended up using a generic low power x86 machine running pfSense 2.3 In my research and testing, I also evaluated IPCop, VyOS, OPNSense, Sophos UTM, RouterOS, OpenWRT x86, and Alpine Linux to serve as the base operating system, but none were as well supported and full featured as PFSense. The main setting to look at is the traffic shaping of uploads, to keep the pipe from getting saturated and having a large buffer build up in the modem and further upstream. This build up is what increases the latency of the connection As with any experiment, any conclusions need to be backed with data. To validate the network was performing smoothly under heavy load, I performed the following experiment: + Ran a ping6 against speedtest.net to measure latency. + Turned off QoS to simulate a “normal router”. + Started multiple simultaneous outbound TCP and UDP streams to saturate my outbound link. + Turned on QoS to the above settings and repeated steps 2 and 3. As you can see from the plot below, without QoS, my connection latency increased by ~1,235%. However with QoS enabled, the connection stayed stable during the upload and I wasn’t able to determine a statistically significant delta. That’s how I maximized the speed on my non-gigabit internet connection. What have you done with your network? FreeBSD on 11″ MacBook Air (https://www.geeklan.co.uk/?p=2214) Sevan Janiyan writes in his tech blog about his experiences running FreeBSD on an 11’’ MacBook Air This tiny machine has been with me for a few years now, It has mostly run OS X though I have tried OpenBSD on it (https://www.geeklan.co.uk/?p=1283). Besides the screen resolution I’m still really happy with it, hardware wise. Software wise, not so much. I use an external disk containing a zpool with my data on it. Among this data are several source trees. CVS on a ZFS filesystem on OS X is painfully slow. I dislike that builds running inside Terminal.app are slow at the expense of a responsive UI. The system seems fragile, at the slightest push the machine will either hang or become unresponsive. Buggy serial drivers which do not implement the break signal and cause instability are frustrating. Last week whilst working on Rump kernel (http://rumpkernel.org/) builds I introduced some new build issues in the process of fixing others, I needed to pick up new changes from CVS by updating my copy of the source tree and run builds to test if issues were still present. I was let down on both counts, it took ages to update source and in the process of cross compiling a NetBSD/evbmips64-el release, the system locked hard. That was it, time to look what was possible elsewhere. While I have been using OS X for many years, I’m not tied to anything exclusive on it, maybe tweetbot, perhaps, but that’s it. On the BSDnow podcast they’ve been covering changes coming in to TrueOS (formerly PC-BSD – a desktop focused distro based on FreeBSD), their experiments seemed interesting, the project now tracks FreeBSD-CURRENT, they’ve replaced rcng with OpenRC as the init system and it comes with a pre-configured desktop environment, using their own window manager (Lumina). Booting the USB flash image it made it to X11 without any issue. The dock has a widget which states the detected features, no wifi (Broadcom), sound card detected and screen resolution set to 1366×768. I planned to give it a try on the weekend. Friday, I made backups and wiped the system. TrueOS installed without issue, after a short while I had a working desktop, resuming from sleep worked out of the box. I didn’t spend long testing TrueOS, switching out NetBSD-HEAD only to realise that I really need ZFS so while I was testing things out, might as well give stock FreeBSD 11-STABLE a try (TrueOS was based on -CURRENT). Turns out sleep doesn’t work yet but sound does work out of the box and with a few invocations of pkg(8) I had xorg, dwm, firefox, CVS and virtuabox-ose installed from binary packages. VirtualBox seems to cause the system to panic (bug 219276) but I should be able to survive without my virtual machines over the next few days as I settle in. I’m considering ditching VirtualBox and converting the vdi files to raw images so that they can be written to a new zvol for use with bhyve. As my default keyboard layout is Dvorak, OS X set the EFI settings to this layout. The first time I installed FreeBSD 11-STABLE, I opted for full disk encryption but ran into this odd issue where on boot the keyboard layout was Dvorak and password was accepted, the system would boot and as it went to mount the various filesystems it would switch back to QWERTY. I tried entering my password with both layout but wasn’t able to progress any further, no bug report yet as I haven’t ruled myself out as the problem. Thunderbolt gigabit adapter –bge(4) (https://www.freebsd.org/cgi/man.cgi?query=bge) and DVI adapter both worked on FreeBSD though the gigabit adapter needs to be plugged in at boot to be detected. The trackpad bind to wsp(4) (https://www.freebsd.org/cgi/man.cgi?query=wsp), left, right and middle clicks are available through single, double and tripple finger tap. Sound card binds to snd_hda(4) (https://www.freebsd.org/cgi/man.cgi?query=snd_hda) and works out of the box. For wifi I’m using a urtw(4) (https://www.freebsd.org/cgi/man.cgi?query=urtw) Alfa adapter which is a bit on the large side but works very reliably. A copy of the dmesg (https://www.geeklan.co.uk/files/macbookair/freebsd-dmesg.txt) is here. Beastie Bits OPNsense - call-for-testing for SafeStack (https://forum.opnsense.org/index.php?topic=5200.0) BSD 4.4: cat (https://www.rewritinghistorycasts.com/screencasts/bsd-4.4:-cat) Continuous Unix commit history from 1970 until today (https://github.com/dspinellis/unix-history-repo) Update on Unix Architecture Evolution Diagrams (https://www.spinellis.gr/blog/20170510/) “Relayd and Httpd Mastery” is out! (https://blather.michaelwlucas.com/archives/2951) Triangle BSD User Group Meeting -- libxo (https://www.meetup.com/Triangle-BSD-Users-Group/events/240247251/) *** Feedback/Questions Carlos - ASUS Tinkerboard (http://dpaste.com/1GJHPNY#wrap) James - Firewall question (http://dpaste.com/0QCW933#wrap) Adam - ZFS books (http://dpaste.com/0GMG5M2#wrap) David - Managing zvols (http://dpaste.com/2GP8H1E#wrap) ***