Archive for the ‘Scripting/Programming’ Category

Extracting multi-layered initramfs

Thursday, December 5th, 2019

Modern Kernel specification (can be seen here) defined the initial ramdisk (initrd or initramfs, depends on who you ask) to allow stacking of compressed or uncompressed CPIO archives. It means, in fact, that you can extend your current initramfs by appending a cpio.gz (or cpio) file at the end, containing the additions or changes to the filesystem (be it directories, files, links and anything else you can think about).

An example of this action:

1
2
3
4
5
mkdir /tmp/test
cd /tmp/test
tar -C /home/ezaton/test123 -cf - . | tar xf - # Clones the contests of /home/ezaton/test123 to this location
find ./ | cpio -o -H newc > ../test.cpio.gz # Creates a compressed CPIO file
cat ../test.cpio.gz >> /boot/initramfs-`uname -r`.img

This should work (I haven’t tried, and if you do it – make sure you have a copy of the original initramfs file!), and the contents of the directory /tmp/test would be reflected in the initramfs.

This method allows us to quickly modify existing ramdisk, replacing files (the stacked cpio files are extracted by order), and practically – doing allot of neat tricks.

The trickier question, however, is how to extract the stacked CPIO files.
If you create a file containing multiple cpio.gz files, appended, and just try to extract them, only the contents of the first CPIO file would be extracted.

The Kernel can do it, and so are we. The basic concept we need to understand is that GZIP compresses a stream. It means that there is no difference between a file structured of stacked CPIO files, and then compressed altogether, or a file constructed by appending cpio.gz files. The result would be similar, and so is the handling of the file. It also means that we do not need to run a loop of zcat/un-cpio and then again zcat/un-cpio on the file chunk by chunk, but when we decompress the file, we decompress it in whole.

Let’s create an example file:

1
2
3
4
5
6
cd /tmp for i in {1..10} ; do
    mkdir test${i}
    touch test${i}/test${i}-file
    find ./test${i} | cpio -o -H newc | gzip > test${i}.cpio.gz
    cat test${i}.cpio.gz >> test-of-all.cpio.gz 
done

This script will create ten directories called test1 to test10, each containing a single file called test<number>-file. Each of them will both be archived into a dedicated cpio.gz file (named the same) and appended to a larger file called test-of-all.cpio.gz

If we run the following script to extract the contents, we will get only the first CPIO contents:

1
2
3
mkdir /tmp/extract
cd /tmp/extract
zcat ../test-of-all.cpio.gz | cpio -id # Format is newc, but it is auto detected

The resulting would be the directory ‘test1’ with a single file in it, but with nothing else. The trick to extract all files would be to run the following command:

1
2
3
4
rm -Rf /tmp/extrac # Cleanup
mkdir /tmp/extract
cd /tmp/extract
zcat ../test-of-all.cpio.gz | while cpio -id ; do : ; done

This will extract all files, until there is no more cpio format remaining. Then the ‘cpio’ command will fail and the loop would end.

Some additional notes:
The ‘:’ is a place holder (does nothing) because ‘while’ loop requires a command. It is a legitimate command in shell.

So – now you can extract even complex CPIO structures, such as can be found in older Foreman “Discovery Image” (very old implementation), Tiny Core Linux (see this forum post, and this wiki note as reference on where this stacking is invoked) and more. This said, for extracting Centos/RHEL7 initramfs, which is structured of uncompressed CPIO appended by a cpio.gz file, a different command is required, and a post about it (works for Ubuntu and RHEL) can be found here.

Using yum with SOCKS proxy

Sunday, June 30th, 2019

SSH is a wonderful tool. One of its best features is the ability to pierce a firewall and let you go through it. If you’re using the dynamic port (-D as argument in command line openSSH), you actually get a SOCKS5 proxy over which you can transport all your desired data.

This allows you the freedom of accessing the Internet from a restricted machine, on the condition it can connect via SSH to another unrestricted machine. So – how does it work?

To simplify things – you will need two sessions on your restricted machine. Use the first to connect via SSH to an unrestricted machine, with the argument, in our example, -D 10000

What it tells the SSH connection is to create a SOCKS5 proxy locally (on the restricted machine) over port 10000, and all the transport sent there – to transfer through the remote (unrestricted) server.

Using the other session, we can implement local variable like this:

export http_proxy=socks5://localhost:10000

export https_proxy=socks5://localhost:10000

It sets a variable which yum (among many other programs) can read and use. Afterwards, using the same session, running ‘yum update‘ or ‘yum install package‘ will result in yum running through the proxy connection. Of course – the SSH session to the unrestricted server must be active at all times, or else yum command will fail.

Old Dell iDrac – work around Java failures

Wednesday, June 5th, 2019

I have an old Dell server (R610, if it’s important) and I seem to fail to connect to its iDrac console via Java. No other options exist, and the browser calling Java flow fails somehow.

I have found an explanation here, and I will copy it for eternity 🙂

First – Download the latest JRE version 1.7 from https::/java.com

Then, extract it to a directory of your choice. We’ll call this directory $RUN_ROOT

Download the viewer.jnlp file to this directory $RUN_ROOT, and open it with a text editor. You will see an XML block pointing at a JAR file called avctKVM.jar. Download it manually using ‘wget’ or ‘curl’ from the URL provided in the viewer.jnlp XML file.

Extract the avctKVM.jar file using ‘unzip’. You will get two libraries – avctKVMIO(.so or .dll for Windows) and avmWinLib(.so or .dll for Windows). Move these two files into a new directory under $RUN_ROOT/lib

Download/copy-paste the below .bat or .sh script files (.bat file for Windows, .sh file for Linux).

start-virtual-console.bat

@echo off

set /P drachost="Host: "
set /p dracuser="Username: "
set "psCommand=powershell -Command "$pword = read-host 'Enter Password' -AsSecureString ; ^
    $BSTR=[System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($pword); ^
        [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR)""
for /f "usebackq delims=" %%p in (`%psCommand%`) do set dracpwd=%%p

start-virtual-console.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
#!/bin/bash
 
echo -n 'Host: '
read drachost
 
echo -n 'Username: '
read dracuser
 
echo -n 'Password: '
read -s dracpwd
echo
 
./jre/bin/java -cp avctKVM.jar -Djava.library.path=./lib com.avocent.idrac.kvm.Main ip=$drachost kmport=5900 vport=5900 user=$dracuser passwd=$dracpwd apcp=1 version=2 vmprivilege=true "helpurl=https://$drachost:443/help/contents.html"

Run the downloaded script file (with Linux – you might want to give it execution permissions first), and you will be asked for your credentials.

Thanks Nicola for this brilliant solution!

How to extract modern Ubuntu initramfs

Thursday, May 30th, 2019

Just to remember, there is an explanation here, from which the following directive can be taken:

(cpio -id; zcat | cpio -id) < /path/to/initrd.img

Oracle ACFS autostart on Oracle RAC stand alone (Oracle Restart)

Thursday, January 3rd, 2019

I would like to start with a declaration – I would prefer not to use ACFS for a stand-alone system. It binds the “normal” order of startup and mounts to the cluster. Not only that – but while until version 12.1, RAC stand alone had a built-in service for ACFS, this is no longer the case for 12.2 and above. This resource/service exists only for a two (and above) node clusters.

If you have upgraded from a 12.1 (or 11.2) stand alone RAC to 12.2 or above, you will no longer be able to automatically mount your ACFS disks. This (and some minor bugs I’ve found with ACFS) is part of the reason I would recommend against using ACFS for stand alone system. HOWEVER – there are cases where you have no choice – either because you are using ACFS replication/snapshots, or because you are using Oracle ASM redundancy model (using either “normal” or “high” redundancy) over a JBoD – which forces you to use ADVM and with it – ACFS is only a small addition.

As I’ve written before – ACFS won’t auto start on 12.2 stand alone GI. A possible solution I thought of (but did not apply, and thus – cannot show it here) is to use a method of creating a 3rd party application service (as described in a document called “TWP-Oracle-Clusterware-3rd-party” to implement a custom service which will actually mount your ACFS for you, when the cluster is ready to do so. I would have done it like that in a recent project, however, a nice person called Pierre has done it for me, slightly differently – he used a systemd services to run custom scripts which attempted to run in loop until the cluster was ready to perform the required actions. I have tested it, and it works well. My only comment about it, which you will be able to see in his blog post, was that if your ORACLE_HOME resides on a dedicated mount point (which is my case, usually), you should force your systemd unit to require this mount as well as its prerequisites. Other than that – his solution worked well, and I thank him for his time and efforts. Kudos Pierre!