Finished customer’s project

It was long, it was tiresome, and it was nasty. We’ve been to a hosting farm, in one of Israel’s largest ISPs,where their (and our) customer needed to relocate servers, and change his server’s IPs, settings, etc.

 

I don’t know why, but we’ve tried to come as prepared as possible. One of the things you learn, doing such
projects in an un-controlled environment, far away from your own personal lab, is this – “Trust no one”. Just like X-Files, but for real.

 

If it’s not obvious, here’s an example – Assuming you get there, and you find out you need some drivers for one of the machines. In a controlled environment, you would get these drivers from the Internet, but in an uncontrolled environment, you must make sure you get them with you before, and make sure the CD, floppy, USB port, or whatever is being used there, is actually functioning, and in good condition. Not only, you must make sure you either get in this place with a whole pack of methods to get the files/info/drivers/data into the machine in question, or a method of transferring between media types, like cd -> Disk on Key, or DoK -> Floppy.

So, trying to be as prepared as possible for the machine (plus extra ~400 domains) transfer and change, we’ve came with the following inventory:

• 1 IBM 1U server, preinstalled with Linux, predefined as DNS server, and web server, saying “The server is under maintenance. It will be solved soon” or something alike.
• 2 Laptops running Linux/Windows, including backup of all configurations of the Virtual servers, and the root servers.
• Cables
  (We’ve discovered only on last minute we don’t get anything out of the hosting farm. We have to bring it all with us. It was night, and we just picked anything we could for it, hoping it would do. It did).
• Tools
• extras
• Exact written procedure of which files to change, where, and into what. New IPs pre-assigned, passwords, etc.

 

We were only half prepared. Half prepared, because the only thing we didn’t predict as much was the ill tempered and lazy SoB who was our contact in the farm. I have no idea why, and I do not care why, but he has some grudge with our (and his!) customer, and he made everything he could to “not help us”. Meaning he didn’t deliberately hinder us, but he did the least he could to help, up to nothing.

 

Example? Sure. We needed network link for the new rack, so he said we had one. I’ve asked him to activate it, and soon he claimed he did. Not long after, when reconfigured the router, and moved it into the new location, I needed to connect it to this link. Not working. I started debugging the problem (maybe bad cable, maybe interface in “shut” mode. Maybe we need laplink cable. Don’t know). Soon I had the obvious idea, and asked him if the link was up. He said “No. I was just waiting for you”. I’ve asked him to bring the link up, keeping my temper as down as possible. It took him 15-30 minutes, while we just stood and waited (it was a show stopper. You can’t start moving servers before you know you have where to connect them to, right?). Finally, and after lots of intervention on our side (like testing and seeing the link was still down, changing cables, etc), the link was brought up, and we could
continue.

Things like this piss me off. You expect the man to do any and every thing he can to assist, so all of you can go home already (the job started at midnight), and this lazy SoB was supposed to hand us the cable link, everything predefined per our demands, and wait for us to finish. Not starting to set it up during our work, and
“waiting” for us. We had to wait for him, that’s for sure, but he had no reason to wait for us.

 

So that’s a hostile, and uncontrolled environment.

 

Don’t get me wrong. We had tons of laughs, and enjoyed the job (and the A/C), but the lack of cooperation, and the stinking attitude of our contact person was, least to say, a problem. Another example is when asking for coffee (to remind you – midnight, no coffee-shops open for kilometers around us), he showed us into their “kitchen”, and pointed out how much he was nice, because of the special time and all, else we wouldn’t supposed to use this “kitchen”. Man, this is only a cup of coffee, and it’s not yours, nor your mom’s! Stinking attitude.

And we had our share of technical difficulties. The person setting up our client’s servers was, how to say, amature. He predefined the machine’s IP address in around a dozen different locations. Three times in the firewall settings (for each, virtual of otherwise real machine’s IP), twice in each network configuration file (per machine), once for every major service each machine (again, virtual or real) was running, such as sshd Listen address, or FTPD Listen address, httpd Listen address, etc. It was a major hell. Hosted domains zone files were not using CNAME record for a single, one-time-only-defined IP address (which each Vserver had. Only one), but had a full A record for the whole IP address. We had to “sed” them all to the new ones, decrease the TTLs for each domain (again, “sed”, or friend), and so on.

It wasn’t easy, but it went rather well, summing it wall up. Why we did it? For the money, of course. And besides, the hosting farm had better A/C than
I have 🙂

 

Well, it sums a night without sleep, filled with work, before I’ve started traveling around, doing all kind of chores I could accumulate around this area of Israel. It went quite well, after all, and I managed to keep my eyes open when driving, which was good, generally speaking.

 

So, here’s me, back home, about to go to sleep, behind me a very, very long day.

 

*Addition*

I have managed to take pictures at the place. Attached in Thumbnails. Sorry for the choppy quality, as they were taken using a cell phone camera, and not a real camera.

 

The rack was a bit shorter than we’ve expected, so our power cables are to be pressed in, to allow closing the doors. Tomorrow night, we are to add a router into the system, and change the firewall’s settings,
accordingly. Will be fun. Not.