Archive for the ‘General Hardware’ Category

Linux boot security – a discussion

Saturday, May 7th, 2022

Hi. As some know now, I have an article about encrypting and protecting Linux – while using TPM to prevent the need to enter the encryption passphrase every time the system boots. This is one of the most popular article in this blog, and for a very good reason – people want their computers (especially portable devices) to be secure and well protected against data theft (let’s assume that by trivial means. I tend to believe that if the NSA or any country-class intelligence service is targeting you, your data will be compromised, so don’t get targeted by countries please), but they want to be the least-bothered with entering and re-entering security credentials along the way. Eating the cake while leaving it whole, if you would like.

Using TPM (which is welded into the motherboard of the computer) to keep the encryption passphrase is a great solution – you get to have an encrypted disk, but don’t have to reuse the passphrase on every reboot. Also – you can rest assured that when/if your hard drive is pulled out of the computer, there is nothing to do with it, because the key is kept on the motherboard.

However – this is not enough, and my article does not cover the scenario where a malicious user pulls out our hard drive and puts a different hard drive instead of it. The current process is of keeping the passphrase directly in the TPM2 chip, not as an SSL challenge/response mechanism, but just as a storage for the key. As such – any user with physical access to the machine from within the OS level can obtain it – even by replacing the disk and pulling the key out, or even worse – by patching the boot sequence (which partially resided on /boot, which is not encrypted) and by replacing initramfs/initrd or by modifying GRUB boot flow, and just asking the TPM nicely for the key. So – our system is not that secure. At least not as much as we aim at. I have mentioned that in the disk encryption article, but yet to have provided a solution.

The reason I have not provided a solution has to do with how Canonical (Ubuntu) take a look at the chain of trust, and the different way I look at it, and to bring this issue up front, I am writing this article, which is not really a technical, but more of a discussion of options. If some of my readers respond and enlighten me – I would be grateful for that, as I am truly looking for a feasible solution to my Ubuntu – this or the next one.

A chain of trust is a workflow where every step is trusted by the previous step, on one side, and can trust the next step. My desirable chain of trust in this case would look like this:

  • The BIOS is locked and cannot be modified without credentials
  • The BIOS allows booting only from a single disk device, and only a specific boot file
  • Using SecureBoot, only the bootable binary (GRUB2 in our case) – a signed and un-tempered file, can be called
  • GRUB2 EFI executable has its configuration embedded, so that you cannot modify it, add additional parameters, or stop its boot process mid-way.
  • GRUB2 is password protected, and allows only a single entry (default) without additional parameters, without authentication.
  • GRUB2 loader trusts the kernel it boots to be one pre-signed by some trusted authority.
  • GRUB2 loader trusts initramfs/initrd as it is signed in a manner GRUB2 understands and respects, aka – it is unmodified
  • Initramfs/initrd is calling the TPM2 to obtain the key, and by doing so, does two things: Opens the disk encryption by using the key, and trusting the BIOS to be un-tempered with, because resetting the BIOS password would reset the TPM2 contents as well (this is good for us).
  • The system continues boot correctly.

You can notice that every step of the boot process has the current step trusted by the previous one, and has a trust mechanism towards the next step. This can work by using self-signed certificate to sign GRUB2 EFI executable, and saving this certificate (and its verification!) in the SecureBoot section of the BIOS, overriding any ‘common’ certificates the BIOS knows of. Then – using grub-standalone instead of the regular GRUB2, which means that GRUB configuration and modules are embedded into a single, signed, executable – we prevent injection of GRUB modules (drivers) or change of parameters. Also – GRUB is password protected against non-default boot. Following that, GRUB trusts the kernel, which is signed, and initramfs/initrd, which is also signed (GRUB used to trust GPG signed files, however, and this is the source of this discussion, Canonical changed some things there, and in my opinion – not for the better). And from here and onwards – the boot process works fine.

This chain of trust seems reasonable. It will prevent boot sequence intervention, and will make sure that the unencrypted part of the partition gets files replaced. You cannot load any different GRUB (even if replaced) due to BIOS trust issues, and you cannot change the configuration, kernel of initramfs/initrd, and therefore – you are bounded to this predefined flow.

This does not follow the ‘trusted computing’ flow. Trusted computing is a mechanism that is supposed to prevent the kernel (and the hardware assisting the process) loading unsigned kernel drivers, and it should be enforced by the TPM/SecureBoot as well. This is nice and pretty, but the vulnerable areas in the trusted computing areas are with the loaders. You might try to inject kernel drivers or malicious code when running under privileged context – when GRUB gets called in.

To mitigate this, Ubuntu enforced (starting with grub version 2.04-1ubuntu26.2) using SHIM, which is a pre-loader mechanism. The chain of trust works slightly differently here:

  • BIOS trusts SHIM, using Microsoft certificate
  • SHIM is signed using Microsoft certificate trusts a different set of certificates – self-signed or Canonical certificates. SHIM is pre-GRUB loader.
  • SHIM trusts GRUB EFI binary, which is signed by Canonical
  • GRUB trusts the kernel using Canonical certificate (all stock kernels are provided pre-signed by Canonical), or custom kernel (and modules) by self-signed certificate, but it needs to be used for every kernel used, and every module used by this kernel.
  • GRUB trusts initramfs/initrd by whatever means (no trust is checked by default there)

The problem with this flow is that it has external sources in the chain of trust, where different components might get replaced or patched. The problems, as I see them (and I would like to get a different opinion about it, so please feel free to share) are:

  • SHIM can be replaced with another EFI binary trusted by Microsoft (like MS Windows loader, or some other tool used by whatever 3rd party I do not know). This will result in skipping the entire trust chain, into a different flow, of which we have no control.
  • SHIM might be able to get compiled with a self-signed certificate (which then be entered into the BIOS SecureBoot as a trusted certificate), however – this process will prevent easy/automated deployment of SHIM, or will require re-compiling it on every update.
  • Using grub-standalone (reviewed above) will require signing it with Canonical certificate (we cannot do this), or with our self-signed. If we use self-signed, we will remove Canonical certificate from SHIM, but then we will have to use our certificate to re-sign the kernel, or we add our self-signed certificate to SHIM, but then, our locked-down GRUB can be replaced with a general GRUB2, signed by Canonical (the default) and worked around the lockdown easily.
  • Now GRUB2 (also grub-standalone, as this is part of the change introduced in version 2.04-1ubuntu26.2) will not trust GPG signature for the kernel (allowing us to avoid re-signing the kernel using SSL, but adding a GPG signature, thus – easier automation of processes and trust chains), but only the SSL, which has to be known to SHIM.
  • By default – initramfs signature is not checked. At this stage with all the chain breaks along the way – it doesn’t matter much now. An attacker can just hook-in anywhere before, and pull our keys out already, using off-the-shelf security loaders of GRUB EFI binaries.

I was able to recomplie the latest GRUB packages for Ubuntu omitting these three patches, without code modifications, to regain the original security flow I presented above (pending tests. Not done yet):

  • 0096-linuxefi-fail-kernel-validation-without-shim-protoco.patch
  • 0099-chainloader-Avoid-a-double-free-when-validation-fail.patch
  • ubuntu-linuxefi-arm64.patch

Is it to be trusted? I do not know. Is it safe “enough”? Maybe. Would I have to continue maintaining a non-official GRUB package? Maybe. How do I maintain it for Ubuntu 22.04? It seems Canonical has their mind set on their flow, which allows for trusted computing – I cannot ignore that – but fails, to my understanding, to provide “good enough” physical access protection when some of the disk is not encrypted (to be exact: /boot and /boot/efi).

You can take a look at the discussion in this Ubuntu bug thread. You can also get the workflow of using SecureBoot and SHIM in Ubuntu in their official wiki here. I did not discuss MokManager, as a tool used to add certificates to SecureBoot or SHIM since I need to dig deeper into this, but to my belief – it only eases management for some of the tasks which might be set easier from within a working OS.

I would love to hear where my analysing of Canonical workflow is wrong, and where their offered security works better than my (previous) workflow – in regards to physical access to data protection.

I will also link this article to the encrypted disk with TPM2 article, in hope it will draw more interaction with this post.

Thanks for reading this far.

RaspberryPi Zero loses connectivity

Friday, July 9th, 2021

I have had a problem with RPI Zero. The system was working fine, and then it did not. I am using Raspbery Linux (Debian-based) with kernel 5.10.17+. Once a while (usually with network load) the system loses connectivity. Everything seems to be fine, if you have a serial/USB console there, but the wireless network fails. This problem was also mentioned here.

My workaround was to create a script with a cron scheduling. I have identified that the fault lies with the wlan driver, and it needs to get reloaded. So cron calls this script every minute, like this:

*/1 * * * * /usr/local/sbin/check_connection.sh

And the script (/usr/local/sbin/check_connection.sh) has this in it:

#!/bin/bash
# DST is the network gateway
DST=192.168.230.1
if ! ping -c 5 -t 5 $DST > /dev/null
then
  #/usr/sbin/reboot
  /usr/bin/logger "Restarting wlan0 network driver"
  /usr/sbin/rmmod brcmfmac && /usr/sbin/modprobe brcmfmac roamoff=1
fi

Set this script to be executable, and your RPI Zero should work just fine. This is not a solution, but a workaround, of course, but it works well.

Max your CPU performance on Linux servers

Friday, July 2nd, 2021

I have had an interesting experience with HPE servers, where the BIOS was defined to allow max performance (in contrast to ‘balanced’ mode), but still – the CPU was not at max all the time.

While we generally, strive to a greener computing, when having low-latency workload, we expect a deterministic performance. We want our database to provide results in a timely, and more important – a consistent way. We want to expect a certain query/job to take the same time, whenever we run it. CPU throttling and frequency manipulation, which are part of CPU power-management are not our friends. A certain query can take longer, if the CPU is currently throttled, and can switch to another, slower, core mid-task. This can result in a very complex performance tuning and server behaviour troubleshooting.

The goal of setting the BIOS to ‘max-performance’ is obvious, then, however – it does not have the desired effect. The CPU keeps on throttling, and, while some power-reducing features are disabled, not all. This is not our goal.

Adding to the boot options of the server (via the GRUB/GRUB2/GRUB-EFI/Whatever) the following parameters, should disable Intel CPU throttling, and all (i)relevant C_states. I have not tested, nor investigated AMD behaviour:

nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce

Of course – a reboot is required for these settings to take effect.

If you want to control power management afterwards, you can manually disable a certain core/CPU by running a command such as this:

echo 0 > /sys/devices/system/cpu/cpu<number>/online

It will disable the CPU (and place it in deep power save) until woken by another CPU, by echoing ‘1’ to that same ‘file’.

OnePlus 7 Pro black screen

Saturday, October 19th, 2019

I have OnePlus 7 Pro. Following the recent update to Android 9, (Just now updated to Android 10, so I don’t know if this problem still relevant) – Once every 2-3 weeks or so, the phone would not wake up from sleep, and remain having a black screen. Long-pressing on the power button has no effect, and the phone remained “dead” as far as I could see. The only indication it was not “entirely dead” was that it generated a very low level of heat.

After leaving it for two days just laying around, I attempted to start it and got a screen message saying the phone needs to be charged enough to start.

It appears, based on this thread, that the phone might have gotten into deep-sleep, from which it could not wake up. A quick workaround was to long-press on Volume Up + Power for about 10-15 seconds. It vibrates (which is the best response ever at that stage) and then you can start the phone normally and it works correctly.

I hope that the Android 10 update solved this issue.

Old Dell iDrac – work around Java failures

Wednesday, June 5th, 2019

I have an old Dell server (R610, if it’s important) and I seem to fail to connect to its iDrac console via Java. No other options exist, and the browser calling Java flow fails somehow.

I have found an explanation here, and I will copy it for eternity 🙂

First – Download the latest JRE version 1.7 from https::/java.com

Then, extract it to a directory of your choice. We’ll call this directory $RUN_ROOT

Download the viewer.jnlp file to this directory $RUN_ROOT, and open it with a text editor. You will see an XML block pointing at a JAR file called avctKVM.jar. Download it manually using ‘wget’ or ‘curl’ from the URL provided in the viewer.jnlp XML file.

Extract the avctKVM.jar file using ‘unzip’. You will get two libraries – avctKVMIO(.so or .dll for Windows) and avmWinLib(.so or .dll for Windows). Move these two files into a new directory under $RUN_ROOT/lib

Download/copy-paste the below .bat or .sh script files (.bat file for Windows, .sh file for Linux).

start-virtual-console.bat
@echo off

set /P drachost="Host: "
set /p dracuser="Username: "
set "psCommand=powershell -Command "$pword = read-host 'Enter Password' -AsSecureString ; ^
    $BSTR=[System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($pword); ^
        [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR)""
for /f "usebackq delims=" %%p in (`%psCommand%`) do set dracpwd=%%p
start-virtual-console.sh
#!/bin/bash

echo -n 'Host: '
read drachost

echo -n 'Username: '
read dracuser

echo -n 'Password: '
read -s dracpwd
echo

./jre/bin/java -cp avctKVM.jar -Djava.library.path=./lib com.avocent.idrac.kvm.Main ip=$drachost kmport=5900 vport=5900 user=$dracuser passwd=$dracpwd apcp=1 version=2 vmprivilege=true "helpurl=https://$drachost:443/help/contents.html"

Run the downloaded script file (with Linux – you might want to give it execution permissions first), and you will be asked for your credentials.

Thanks Nicola for this brilliant solution!