Posts Tagged ‘Linux’

Linux boot security – a discussion

Saturday, May 7th, 2022

Hi. As some know now, I have an article about encrypting and protecting Linux – while using TPM to prevent the need to enter the encryption passphrase every time the system boots. This is one of the most popular article in this blog, and for a very good reason – people want their computers (especially portable devices) to be secure and well protected against data theft (let’s assume that by trivial means. I tend to believe that if the NSA or any country-class intelligence service is targeting you, your data will be compromised, so don’t get targeted by countries please), but they want to be the least-bothered with entering and re-entering security credentials along the way. Eating the cake while leaving it whole, if you would like.

Using TPM (which is welded into the motherboard of the computer) to keep the encryption passphrase is a great solution – you get to have an encrypted disk, but don’t have to reuse the passphrase on every reboot. Also – you can rest assured that when/if your hard drive is pulled out of the computer, there is nothing to do with it, because the key is kept on the motherboard.

However – this is not enough, and my article does not cover the scenario where a malicious user pulls out our hard drive and puts a different hard drive instead of it. The current process is of keeping the passphrase directly in the TPM2 chip, not as an SSL challenge/response mechanism, but just as a storage for the key. As such – any user with physical access to the machine from within the OS level can obtain it – even by replacing the disk and pulling the key out, or even worse – by patching the boot sequence (which partially resided on /boot, which is not encrypted) and by replacing initramfs/initrd or by modifying GRUB boot flow, and just asking the TPM nicely for the key. So – our system is not that secure. At least not as much as we aim at. I have mentioned that in the disk encryption article, but yet to have provided a solution.

The reason I have not provided a solution has to do with how Canonical (Ubuntu) take a look at the chain of trust, and the different way I look at it, and to bring this issue up front, I am writing this article, which is not really a technical, but more of a discussion of options. If some of my readers respond and enlighten me – I would be grateful for that, as I am truly looking for a feasible solution to my Ubuntu – this or the next one.

A chain of trust is a workflow where every step is trusted by the previous step, on one side, and can trust the next step. My desirable chain of trust in this case would look like this:

  • The BIOS is locked and cannot be modified without credentials
  • The BIOS allows booting only from a single disk device, and only a specific boot file
  • Using SecureBoot, only the bootable binary (GRUB2 in our case) – a signed and un-tempered file, can be called
  • GRUB2 EFI executable has its configuration embedded, so that you cannot modify it, add additional parameters, or stop its boot process mid-way.
  • GRUB2 is password protected, and allows only a single entry (default) without additional parameters, without authentication.
  • GRUB2 loader trusts the kernel it boots to be one pre-signed by some trusted authority.
  • GRUB2 loader trusts initramfs/initrd as it is signed in a manner GRUB2 understands and respects, aka – it is unmodified
  • Initramfs/initrd is calling the TPM2 to obtain the key, and by doing so, does two things: Opens the disk encryption by using the key, and trusting the BIOS to be un-tempered with, because resetting the BIOS password would reset the TPM2 contents as well (this is good for us).
  • The system continues boot correctly.

You can notice that every step of the boot process has the current step trusted by the previous one, and has a trust mechanism towards the next step. This can work by using self-signed certificate to sign GRUB2 EFI executable, and saving this certificate (and its verification!) in the SecureBoot section of the BIOS, overriding any ‘common’ certificates the BIOS knows of. Then – using grub-standalone instead of the regular GRUB2, which means that GRUB configuration and modules are embedded into a single, signed, executable – we prevent injection of GRUB modules (drivers) or change of parameters. Also – GRUB is password protected against non-default boot. Following that, GRUB trusts the kernel, which is signed, and initramfs/initrd, which is also signed (GRUB used to trust GPG signed files, however, and this is the source of this discussion, Canonical changed some things there, and in my opinion – not for the better). And from here and onwards – the boot process works fine.

This chain of trust seems reasonable. It will prevent boot sequence intervention, and will make sure that the unencrypted part of the partition gets files replaced. You cannot load any different GRUB (even if replaced) due to BIOS trust issues, and you cannot change the configuration, kernel of initramfs/initrd, and therefore – you are bounded to this predefined flow.

This does not follow the ‘trusted computing’ flow. Trusted computing is a mechanism that is supposed to prevent the kernel (and the hardware assisting the process) loading unsigned kernel drivers, and it should be enforced by the TPM/SecureBoot as well. This is nice and pretty, but the vulnerable areas in the trusted computing areas are with the loaders. You might try to inject kernel drivers or malicious code when running under privileged context – when GRUB gets called in.

To mitigate this, Ubuntu enforced (starting with grub version 2.04-1ubuntu26.2) using SHIM, which is a pre-loader mechanism. The chain of trust works slightly differently here:

  • BIOS trusts SHIM, using Microsoft certificate
  • SHIM is signed using Microsoft certificate trusts a different set of certificates – self-signed or Canonical certificates. SHIM is pre-GRUB loader.
  • SHIM trusts GRUB EFI binary, which is signed by Canonical
  • GRUB trusts the kernel using Canonical certificate (all stock kernels are provided pre-signed by Canonical), or custom kernel (and modules) by self-signed certificate, but it needs to be used for every kernel used, and every module used by this kernel.
  • GRUB trusts initramfs/initrd by whatever means (no trust is checked by default there)

The problem with this flow is that it has external sources in the chain of trust, where different components might get replaced or patched. The problems, as I see them (and I would like to get a different opinion about it, so please feel free to share) are:

  • SHIM can be replaced with another EFI binary trusted by Microsoft (like MS Windows loader, or some other tool used by whatever 3rd party I do not know). This will result in skipping the entire trust chain, into a different flow, of which we have no control.
  • SHIM might be able to get compiled with a self-signed certificate (which then be entered into the BIOS SecureBoot as a trusted certificate), however – this process will prevent easy/automated deployment of SHIM, or will require re-compiling it on every update.
  • Using grub-standalone (reviewed above) will require signing it with Canonical certificate (we cannot do this), or with our self-signed. If we use self-signed, we will remove Canonical certificate from SHIM, but then we will have to use our certificate to re-sign the kernel, or we add our self-signed certificate to SHIM, but then, our locked-down GRUB can be replaced with a general GRUB2, signed by Canonical (the default) and worked around the lockdown easily.
  • Now GRUB2 (also grub-standalone, as this is part of the change introduced in version 2.04-1ubuntu26.2) will not trust GPG signature for the kernel (allowing us to avoid re-signing the kernel using SSL, but adding a GPG signature, thus – easier automation of processes and trust chains), but only the SSL, which has to be known to SHIM.
  • By default – initramfs signature is not checked. At this stage with all the chain breaks along the way – it doesn’t matter much now. An attacker can just hook-in anywhere before, and pull our keys out already, using off-the-shelf security loaders of GRUB EFI binaries.

I was able to recomplie the latest GRUB packages for Ubuntu omitting these three patches, without code modifications, to regain the original security flow I presented above (pending tests. Not done yet):

  • 0096-linuxefi-fail-kernel-validation-without-shim-protoco.patch
  • 0099-chainloader-Avoid-a-double-free-when-validation-fail.patch
  • ubuntu-linuxefi-arm64.patch

Is it to be trusted? I do not know. Is it safe “enough”? Maybe. Would I have to continue maintaining a non-official GRUB package? Maybe. How do I maintain it for Ubuntu 22.04? It seems Canonical has their mind set on their flow, which allows for trusted computing – I cannot ignore that – but fails, to my understanding, to provide “good enough” physical access protection when some of the disk is not encrypted (to be exact: /boot and /boot/efi).

You can take a look at the discussion in this Ubuntu bug thread. You can also get the workflow of using SecureBoot and SHIM in Ubuntu in their official wiki here. I did not discuss MokManager, as a tool used to add certificates to SecureBoot or SHIM since I need to dig deeper into this, but to my belief – it only eases management for some of the tasks which might be set easier from within a working OS.

I would love to hear where my analysing of Canonical workflow is wrong, and where their offered security works better than my (previous) workflow – in regards to physical access to data protection.

I will also link this article to the encrypted disk with TPM2 article, in hope it will draw more interaction with this post.

Thanks for reading this far.

Enable blk_mq on Redhat (OEL/Centos) 7 and 8

Saturday, December 18th, 2021

The IO scheduler blk_mq was designed to increase disk performance – better IOps and lower latency – on flash disk systems, with emphasis on NVME devices. A nice by-product, is that it increases performance on the “older” SCSI/SAS/SATA layer as well, when the disks in the back are SSD, and even when they are rotational disks, in some cases.

When attempting to increase system disk performance (where and when this is the bottleneck, such as in the case of databases), this is a valid and relevant configuration option. If you have a baseline performance metrics to compare to – great ; But even if you do not have other, previous performance details – in most cases – for a modern system – your performance will improve after this change.

NVME devices get this scheduler enabled by default, so no worry there. However, your system might not be aware of the backend storage type of devices, nor of the backend capabilities when dealing with SCSI subsystem, so – by default – these settings are not enabled.

To enable these settings on RHEL/OEL/Centos version 7 and 8, you should do the following:

Edit /etc/default/grub and append to your GRUB_CMDLINE_LINUX the following string:

scsi_mod.use_blk_mq=1 dm_mod.use_blk_mq=y

This will enable blk_mq both on the SCSI subsystem, and the device mapper (DM), which includes software RAID, LVM and device-mapper-multipath (aka – multipath).

Following this, run:

grub2-mkconfig -o /boot/grub2/grub.cfg  # for legacy BIOS

or

grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg  # for EFI

Of course – a reboot is required for these changes to take effect. Following the reboot, you will be able to see that the contents of /sys/block/sda/queue/scheduler (for /dev/sda, in my example. Check your relevant disks, of course) shows “[mq-deadline]”

If you want to read more about blk_mq – you can find it in great detail in this blog post.

Selective dnsmasq logging (split dnsmasq logging)

Sunday, October 31st, 2021

My system provides DNS services, using dnsmasq, to several different subnets. I wish to log specific queries to different files – as I want to identify, and maybe even respond to certain DNS queries of the IoT network.

The (excellent) utility dnsmasq is unable to split the logging into multiple log files, or filter logging by expressions, so we need to combine the power of dnsmasq’s logging with rsyslogd’s expression matching.

Let’s assume I have two networks. One is 192.168.1.x – the home LAN, and the other is 172.16.1.x – the IoT network.

I have added to my /etc/dnsmasq.conf file the following lines:

log-facility=DAEMON
log-async
log-queries=extra

I have created a file called /etc/rsyslog.d/dnsmasq.conf with the following contents:

if $programname == 'dnsmasq' and $msg contains ' 192.168.1.' then /var/log/dnsmasq/dnsmasq-lan.log
if $programname == 'dnsmasq' and $msg contains ' 172.16.1.' then /var/log/dnsmasq/dnsmasq-iot.log
if $programname == 'dnsmasq-dhcp' then /var/log/dnsmasq/dnsmasq-dhcp.log
if $programname == 'dnsmasq' then stop
if $programname == 'dnsmasq-dhcp' then stop

Of course – I need to create the directory /var/log/dnsmasq, and create a logrotate entry /etc/logrotate.d/dnsmasq as follows:

/var/log/dnsmasq/dnsmasq-iot.log /var/log/dnsmasq/dnsmasq-lan.log /var/log/dnsmasq/dnsmasq-dhcp.log {
  monthly
  missingok
  notifempty
  maxsize 5M
  rotate 14
  delaycompress
  # create 0640 dnsmasq root
  sharedscripts
  postrotate
    /usr/bin/systemctl kill -s HUP rsyslog.service >/dev/null 2>&1 || true
  endscript
}

Note that the DNS queries of the networks are kept in a dedicated per-network file (dnsmasq-lan.log and dnsmasq-iot.log) and all general (non IP specific) messages are kept in dnsmasq-dhcp.log file. Logrotate makes sure I do not overfill my directory, and I can later on identify which IoT (or home, for that matter) DNS query is sent and by whom.

Quick items about repackaging Linux ISO

Thursday, October 28th, 2021

There are two topics I would like to describe here, for later reference (by myself, of course. This blog is my extended memory). The first is about how to create a bootable ISO out of RHEL extracted ISO, and the other is about how to download only specific update, or make your own RHEL updates on-prem mirror.

Bootable ISO

From within the (modified?) extracted ISO of RHEL7.x (in this example. Match settings to your needs), in order to be able to boot the ISO both in legacy and uEFI BIOS – you can run this command:

genisoimage -J -T -o ../RHEL-7.9_`date +%F_%H-%M-%S`.iso -b isolinux/isolinux.bin -J -R -l -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e images/efiboot.img -no-emul-boot -graft-points -V "RHEL-7.9 Server.x86_64" .

Create a local mirror of RHEL packages

This is a long one, so I will leave only a link to RedHat’s article about it. I hope you have access (you should, if you want to mirror their repository). If you don’t – it’s easy and free to open an account (even without subscribed systems), so you’ll have access to their articles. The article can be found here

RaspberryPi Zero loses connectivity

Friday, July 9th, 2021

I have had a problem with RPI Zero. The system was working fine, and then it did not. I am using Raspbery Linux (Debian-based) with kernel 5.10.17+. Once a while (usually with network load) the system loses connectivity. Everything seems to be fine, if you have a serial/USB console there, but the wireless network fails. This problem was also mentioned here.

My workaround was to create a script with a cron scheduling. I have identified that the fault lies with the wlan driver, and it needs to get reloaded. So cron calls this script every minute, like this:

*/1 * * * * /usr/local/sbin/check_connection.sh

And the script (/usr/local/sbin/check_connection.sh) has this in it:

#!/bin/bash
# DST is the network gateway
DST=192.168.230.1
if ! ping -c 5 -t 5 $DST > /dev/null
then
  #/usr/sbin/reboot
  /usr/bin/logger "Restarting wlan0 network driver"
  /usr/sbin/rmmod brcmfmac && /usr/sbin/modprobe brcmfmac roamoff=1
fi

Set this script to be executable, and your RPI Zero should work just fine. This is not a solution, but a workaround, of course, but it works well.