Running Systems

Kickstart is a great method of hands-free installation of RHEL/Centos (and other derived systems). Its power is in its easy interface and rather powerful %post scripting directives. Its weakness is in its lack of flexibility where it comes to package selection and various custom actions.

On some cases, companies use web interface (usually home-made) which builds kickstart config files on-demand. On some cases, the administrator is required to build several kickstart config files for pre-anticipated setups.

I was looking for something which will give me the power to maintain a fixed configuration on one hand, and will allow me some tweaks and variants, when I want them. I could have used the %post scripting sections, but this gets quite complicated, especially when you want to add only one package (but with its dependencies), or you want to force full update of the system before it goes online, or even select its hostname, assuming it is not yet defined in the DNS.

I base my system on a simple DHCP/BootP + tftp server which answers to all bootp requests and offers a simple menu (just type a number and press on Enter). The original schema was quite simple: type 4 for Centos4.3, and then add -min if you wanted it to use a kickstart file with a minimum configuration. Then I wanted to add the option to update the system in an early stage, so I have added -update, which would have looked in the menu like “4-min-update” option. Quite readable, however, it generated lots of work when maintaining the pxelinux.cfg/default file and the ks themselves. Too many variations tend to require lots of care.

Adding parameters to the boot menu is possible, and would result in them existing in /proc/cmdline for later parsing.

I have decided to parse a set of predefined parameters supplied during boot time, and to change the kickstart config file according to them. It actually works quite well. This is a less-sophisticated and more of a stand-alone system compared to this system. Also, it doesn’t require me to alter the system’s boot process.

This is my ks.cfg file, which includes the flexibility additions:

# Kickstart file generated by Ez-Aton

install
nfs –server=install-server –dir=/mnt/samba/Centos
lang en_US.UTF-8
langsupport –default=en_US.UTF-8 en_US.UTF-8
keyboard us
skipx
network –device eth0 –bootproto dhcp
rootpw –iscrypted RpUKzjDc9k2gU
firewall –disabled
selinux –disabled
authconfig –enableshadow –enablemd5
timezone Asia/Jerusalem
bootloader –location=mbr

%packages
e2fsprogs
grub
lvm2
kernel
net-snmp
net-snmp-utils
kernel-devel
kernel-smp-devel
gcc

%pre
# By Ez-Aton http://www.tournament.org.il/run
for i in `cat /proc/cmdline`; do
echo $i >> /tmp/vars.tmp
done
grep “=” /tmp/vars.tmp > /tmp/vars
KS=/tmp/ks.cfg
update=””
name=””
pkg=””
. /tmp/vars
if [ ! -z “$update” ]; then
echo “yum update -y” >> $KS
fi
if [ ! -z “$name” ]; then
value=”dhcp –hostname $name”
cat $KS | sed s/dhcp/”$value”/ > $KS.tmp
cat $KS.tmp > $KS
fi
if [ ! -z “$pkg” ]; then
pkg_line=`grep -n ^%packages $KS | cut -f 1 -d :`
max_line=`wc -l $KS | awk ‘{print $1}’`
head -n $pkg_line $KS > $KS.tmp
for i in `echo $pkg | sed s/,/ /g`; do
echo $i >> $KS.tmp
done
let tail_line=$max_line-$pkg_line
tail -n $tail_line $KS >> $KS.tmp
cat $KS.tmp > $KS
fi

%post

So, as you can see, I take the following parameters:

update=yes (it can be update=anything)

name=hostname (in case it cannot be retrieved from the DHCP server)

pkg=pkg1,pkg2,{pkg3,…} (To add specific packages to the installation)

It was tested to work on Centos4.3 system, and will probably work on RHEL and Centos versions 4.x all along. I didn’t test it on RHEL5/Centos5 yet.

If you use the script, please leave my name and blog URL in it. Also, if you modify it for your needs, I would be glad to get back the modifications you have made, to include them.

Enjoy.

I was required to utilize a transparent proxy. The general idea was to follow a diagram as the one here:

The company did not want any information (http, https, ftp, whatever) to pass directly through the firewall from the internal network to the external network. If we can move it all via some sort of proxy, the general idea says, the added security is well worth it.

Getting an initial configuration to work is rather simple. For port 80 (HTTP), all need not do more than install squid with transparent directives included (can be found here, for example, and on dozens of other web sites), and make sure the router redirects all outbound HTTP traffic to the Linux proxy.

It worked like a charm. Few minor tweeks, and caching was done well.

It didn’t work when it came to other protocols. It appreas Squid cannot transparently redirect (I did not expect it to actually cache the information) SSL requests. The whole idea of SSL is to prevent the possibility of “A-Man-in-the-Middle” attack, so Squid cannot be part of the point-to-point communication, unless directed to do so by the browser, with the CONNECT command. This command can be assigned ONLY if the client is aware of the fact that there is a proxy on the way, aka, configured to use it, which is in contrast to the whole idea of Transparent Proxy.

When it failed, I’ve came up with the next idea – let the Linux machine route onwards the forwarded packets, by acting as a self-sustained NAT server. If it can translate all requests as comming from it, I will be able to redirect all traffic through it. It did not work, and working hard into IPTables chains, and adding logging (iptables -t nat -I PREROUTING -j LOG –log-prefix “PRERouting: “) into it, I’ve discovered that although the PREROUTING chain accepted the packets, they never reached the FORWARD or POSTROUTING chains…

The general conclusion was that the packets were destinated to the Linux machine. The Firewall/Router has redirected all packets to the Linux server not by altering the routing table to point at the Linux server as the next hop, but by altering the destination of the packets themselves. It meant that all redirected packets were to go to the Linux machine.

Why did HTTP succeed in passing the transparent proxy? Because HTTP packets contain the target name (web address) in their data, and not only in their headers. This allows for “Name based shared hosting”, and thus the transparent proxy can actually exist.

There is no such luck with other protocols, I’m afraid.

The solution in this case can be achieved via few methods:

1. Use non-transparent proxy. Set the clients to use it via some script, which will enable them to avoid using it when outside the company. Combined with transparent HTTP proxy, it can block unwanted access.

2. Use stateful inspection on any allowed outbound packets, except HTTP, which will be redirected to the proxy server transparently.

3. Set the Linux machine in the direct path outside, as an additional line of defence.

4. If the firewall/Router is capable of it, set a protocol-based routing. If you only route differently packets outbound for some port, you do not rewrite the packet destination.

I tend to chose option 1, as it allows for access to work silently when using HTTP, and prevents unconfigured clients from accessing disallowed ports. Such a set of rules could look something like (the proxy listens on port 80):

1. From *IN* to *LINUX* outbound to port 80, ALLOW

2. From *IN* to *INTERNET* outbound to port 80 REDIRECT to Linux:80

3. From *IN* to *INTERNET* DENY

Clients with defined proxy settings will work just alright. Clients with undefined proxy settings, will not be able to access HTTPS, FTP, etc, but will still be able to browse the regular web.

In all these cases, control over the allowed URLs and destinations is in the hands of the local IT team.

It was long, it was tiresome, and it was nasty. We’ve been to a hosting farm, in one of Israel’s largest ISPs,where their (and our) customer needed to relocate servers, and change his server’s IPs, settings, etc.

I don’t know why, but we’ve tried to come as prepared as possible. One of the things you learn, doing such
projects in an un-controlled environment, far away from your own personal lab, is this – “Trust no one”. Just like X-Files, but for real.

If it’s not obvious, here’s an example – Assuming you get there, and you find out you need some drivers for one of the machines. In a controlled environment, you would get these drivers from the Internet, but in an uncontrolled environment, you must make sure you get them with you before, and make sure the CD, floppy, USB port, or whatever is being used there, is actually functioning, and in good condition. Not only, you must make sure you either get in this place with a whole pack of methods to get the files/info/drivers/data into the machine in question, or a method of transferring between media types, like cd -> Disk on Key, or DoK -> Floppy.

So, trying to be as prepared as possible for the machine (plus extra ~400 domains) transfer and change, we’ve came with the following inventory:

• 1 IBM 1U server, preinstalled with Linux, predefined as DNS server, and web server, saying “The server is under maintenance. It will be solved soon” or something alike.
• 2 Laptops running Linux/Windows, including backup of all configurations of the Virtual servers, and the root servers.
• Cables
  (We’ve discovered only on last minute we don’t get anything out of the hosting farm. We have to bring it all with us. It was night, and we just picked anything we could for it, hoping it would do. It did).
• Tools
• extras
• Exact written procedure of which files to change, where, and into what. New IPs pre-assigned, passwords, etc.

We were only half prepared. Half prepared, because the only thing we didn’t predict as much was the ill tempered and lazy SoB who was our contact in the farm. I have no idea why, and I do not care why, but he has some grudge with our (and his!) customer, and he made everything he could to “not help us”. Meaning he didn’t deliberately hinder us, but he did the least he could to help, up to nothing.

Example? Sure. We needed network link for the new rack, so he said we had one. I’ve asked him to activate it, and soon he claimed he did. Not long after, when reconfigured the router, and moved it into the new location, I needed to connect it to this link. Not working. I started debugging the problem (maybe bad cable, maybe interface in “shut” mode. Maybe we need laplink cable. Don’t know). Soon I had the obvious idea, and asked him if the link was up. He said “No. I was just waiting for you”. I’ve asked him to bring the link up, keeping my temper as down as possible. It took him 15-30 minutes, while we just stood and waited (it was a show stopper. You can’t start moving servers before you know you have where to connect them to, right?). Finally, and after lots of intervention on our side (like testing and seeing the link was still down, changing cables, etc), the link was brought up, and we could
continue.

Things like this piss me off. You expect the man to do any and every thing he can to assist, so all of you can go home already (the job started at midnight), and this lazy SoB was supposed to hand us the cable link, everything predefined per our demands, and wait for us to finish. Not starting to set it up during our work, and
“waiting” for us. We had to wait for him, that’s for sure, but he had no reason to wait for us.

So that’s a hostile, and uncontrolled environment.

Don’t get me wrong. We had tons of laughs, and enjoyed the job (and the A/C), but the lack of cooperation, and the stinking attitude of our contact person was, least to say, a problem. Another example is when asking for coffee (to remind you – midnight, no coffee-shops open for kilometers around us), he showed us into their “kitchen”, and pointed out how much he was nice, because of the special time and all, else we wouldn’t supposed to use this “kitchen”. Man, this is only a cup of coffee, and it’s not yours, nor your mom’s! Stinking attitude.

And we had our share of technical difficulties. The person setting up our client’s servers was, how to say, amature. He predefined the machine’s IP address in around a dozen different locations. Three times in the firewall settings (for each, virtual of otherwise real machine’s IP), twice in each network configuration file (per machine), once for every major service each machine (again, virtual or real) was running, such as sshd Listen address, or FTPD Listen address, httpd Listen address, etc. It was a major hell. Hosted domains zone files were not using CNAME record for a single, one-time-only-defined IP address (which each Vserver had. Only one), but had a full A record for the whole IP address. We had to “sed” them all to the new ones, decrease the TTLs for each domain (again, “sed”, or friend), and so on.

It wasn’t easy, but it went rather well, summing it wall up. Why we did it? For the money, of course. And besides, the hosting farm had better A/C than
I have 🙂

Well, it sums a night without sleep, filled with work, before I’ve started traveling around, doing all kind of chores I could accumulate around this area of Israel. It went quite well, after all, and I managed to keep my eyes open when driving, which was good, generally speaking.

So, here’s me, back home, about to go to sleep, behind me a very, very long day.

*Addition*

I have managed to take pictures at the place. Attached in Thumbnails. Sorry for the choppy quality, as they were taken using a cell phone camera, and not a real camera.

The rack was a bit shorter than we’ve expected, so our power cables are to be pressed in, to allow closing the doors. Tomorrow night, we are to add a router into the system, and change the firewall’s settings,
accordingly. Will be fun. Not.