Archive for July, 2005
It was long, it was tiresome, and it was nasty. We’ve been to a hosting farm, in one of Israel’s largest ISPs,where their (and our) customer needed to relocate servers, and change his server’s IPs, settings, etc.
I don’t know why, but we’ve tried to come as prepared as possible. One of the things you learn, doing such
projects in an un-controlled environment, far away from your own personal lab, is this – "Trust no one". Just like X-Files, but for real.
If it’s not obvious, here’s an example – Assuming you get there, and you find out you need some drivers for one of the machines. In a controlled environment, you would get these drivers from the Internet, but in an uncontrolled environment, you must make sure you get them with you before, and make sure the CD, floppy, USB port, or whatever is being used there, is actually functioning, and in good condition. Not only, you must make sure you either get in this place with a whole pack of methods to get the files/info/drivers/data into the machine in question, or a method of transferring between media types, like cd -> Disk on Key, or DoK -> Floppy.
So, trying to be as prepared as possible for the machine (plus extra ~400 domains) transfer and change, we’ve came with the following inventory:
- 1 IBM 1U server, preinstalled with Linux, predefined as DNS server, and web server, saying "The server is under maintenance. It will be solved soon" or something alike.
- 2 Laptops running Linux/Windows, including backup of all configurations of the Virtual servers, and the root servers.
(We’ve discovered only on last minute we don’t get anything out of the hosting farm. We have to bring it all with us. It was night, and we just picked anything we could for it, hoping it would do. It did).
- Exact written procedure of which files to change, where, and into what. New IPs pre-assigned, passwords, etc.
We were only half prepared. Half prepared, because the only thing we didn’t predict as much was the ill tempered and lazy SoB who was our contact in the farm. I have no idea why, and I do not care why, but he has some grudge with our (and his!) customer, and he made everything he could to "not help us". Meaning he didn’t deliberately hinder us, but he did the least he could to help, up to nothing.
Example? Sure. We needed network link for the new rack, so he said we had one. I’ve asked him to activate it, and soon he claimed he did. Not long after, when reconfigured the router, and moved it into the new location, I needed to connect it to this link. Not working. I started debugging the problem (maybe bad cable, maybe interface in "shut" mode. Maybe we need laplink cable. Don’t know). Soon I had the obvious idea, and asked him if the link was up. He said "No. I was just waiting for you". I’ve asked him to bring the link up, keeping my temper as down as possible. It took him 15-30 minutes, while we just stood and waited (it was a show stopper. You can’t start moving servers before you know you have where to connect them to, right?). Finally, and after lots of intervention on our side (like testing and seeing the link was still down, changing cables, etc), the link was brought up, and we could
Things like this piss me off. You expect the man to do any and every thing he can to assist, so all of you can go home already (the job started at midnight), and this lazy SoB was supposed to hand us the cable link, everything predefined per our demands, and wait for us to finish. Not starting to set it up during our work, and
"waiting" for us. We had to wait for him, that’s for sure, but he had no reason to wait for us.
So that’s a hostile, and uncontrolled environment.
Don’t get me wrong. We had tons of laughs, and enjoyed the job (and the A/C), but the lack of cooperation, and the stinking attitude of our contact person was, least to say, a problem. Another example is when asking for coffee (to remind you – midnight, no coffee-shops open for kilometers around us), he showed us into their "kitchen", and pointed out how much he was nice, because of the special time and all, else we wouldn’t supposed to use this "kitchen". Man, this is only a cup of coffee, and it’s not yours, nor your mom’s! Stinking attitude.
And we had our share of technical difficulties. The person setting up our client’s servers was, how to say, amature. He predefined the machine’s IP address in around a dozen different locations. Three times in the firewall settings (for each, virtual of otherwise real machine’s IP), twice in each network configuration file (per machine), once for every major service each machine (again, virtual or real) was running, such as sshd Listen address, or FTPD Listen address, httpd Listen address, etc. It was a major hell. Hosted domains zone files were not using CNAME record for a single, one-time-only-defined IP address (which each Vserver had. Only one), but had a full A record for the whole IP address. We had to "sed" them all to the new ones, decrease the TTLs for each domain (again, "sed", or friend), and so on.
It wasn’t easy, but it went rather well, summing it wall up. Why we did it? For the money, of course. And besides, the hosting farm had better A/C than
I have 🙂
Well, it sums a night without sleep, filled with work, before I’ve started traveling around, doing all kind of chores I could accumulate around this area of Israel. It went quite well, after all, and I managed to keep my eyes open when driving, which was good, generally speaking.
So, here’s me, back home, about to go to sleep, behind me a very, very long day.
I have managed to take pictures at the place. Attached in Thumbnails. Sorry for the choppy quality, as they were taken using a cell phone camera, and not a real camera.
The rack was a bit shorter than we’ve expected, so our power cables are to be pressed in, to allow closing the doors. Tomorrow night, we are to add a router into the system, and change the firewall’s settings,
accordingly. Will be fun. Not.
I’ve played with MRTG a bit further.
Well, I have a long lasting, and well used MRTG configurations in various locations and on different servers. I’ve decided just few days ago to add monitoring of Apache (httpd) to my MRTG graphs. So I’ve enabled the server-status page, for a limited set of addresses, and added the MRTG stuff. I’ve had to tinker with the perl script a bit, since it returned the same value twice for almost all and every query, which meant lots of lost screen space. I’ve decided to merge some, and with luck, it will probve useful.
I will not publish here the MRTG settings, not the re-edited scripts, not until I test it further, and for the time being. Not at all. If you’ll ask for it, I would be happy to give it away, but not online now, and no screen shots either. I don’t have the patiance to screenshot it, and erase valid information, so I won’t do it now. Sorry.
I’m on the brink of the actual and physical commitment of a project for a customer. moving a banch of Linux servers at their hosting into another physical location, changing the IP address, and making sure everything’s working correctly.
It could have been a pieve of cake, but this machine runs vservers, and it is using some management interface, etc, which demands carefull setup. not only this, but this machine, acting as a hosting server, has DNS A records for every and any virtual host, instead of having CNAME record, which means we’ll have fun.
I’ve just talked to the hosting supplier (the owner of the farm itself), and they are no thrilled. If it were me, the whole transfer setup would have been cut at that point, and I would have moved to another computer farm. It was a lousy service, and it should be paid accordingly.
So it won’t happen this weekend. The farm is not pre-ready, as one person there said (and he is now abroad, so there’s no one to prove I’m right about that). It will have to be next week. Damn. I was hoping to go on a short diving cruse next week.
Well, it’s just me ranting about. At least a friend’s supposed to come over, have an (exellent) Humus with me, and help me plan the transfer. Now I can get some sleep (look at the time!!! So early in the morning!)
As my quest for a fully working laptop is advancing, I have decided to invest my time (or got drawn to it, donno why) in making the modem work. Actually, I’m stuck here. I cannot see the modem’s device in lspci, I have not isapnp devices whatsoever, so I cannot claime it’s an ISA device, and the AMR is supposed to depend on the PCI bus, am I right?
The modem is supposed to be a Lucent AMR modem. According to linmodems, this modem is not supported whatsoever, and never will be. According to some other parts of the internet, this modem might be supported via Smartlink module, which I was able to compile. Well, as we sometimes learn, compiling isn’t everything. The module failed to load because it claimed to have some unresolved symbols. Great. Searching google, I’ve found a web site dealing with activating the modem on an ALSA based systems (sounds just like my case!) so I tried that direction as well. It was well documented withing the driver’s README itself. I’ve discovered one tweak, though. "make install" rebuilds the whole code. It’s like "make all" with the install part afterwards, so if you’re into it, make sure you read the damn README file, and type "make install SUPPORT_ALSA=1" right from the start. No luck, though, as there is no extra ALSA device, not after this procedure, and not after I rebuild the modules, per the instructions in the README file.
Just now I’ve tested an alternate method, as described in this website. It didn’t work either, but was another try, right?
Still, no PCI id for the modem. I might be searching in the wrong direction. Maybe it’s the ISA Bridge, and I’ll have to pinpoint it there. I can’t tell now. I will get back to it sometimes later.
Last night, after I finished writing the previous entry, I was looking for Fujitsu & Linux related sites, for somewhat more information about my modem.
I found out a site, which reminded me of the Fujitsu special keys which I’ve never used nor managed to setup. Well, the version the site pointed at was old, and I could not compile it, however, the newer version (courtasy of google) of fjkeys and apanel allowed me to finally, for the first time, to comile and insert this module. I was so surprised to find out that there was one led of which I never knew.
Nothing is perfect, of course. I have tons of message such as this:
i2c_adapter i2c-0: Error: command never completed
but besides that (which does not damage nor slow down my suspend/resume), all seems to work. Good. This blog existance (no readers so far, and still no search engine index) pushed me towards solving some of my problems, and towards a better laptop-wise life.
Another set of reasons are that it’s damn hot in the house (but in the balcony it’s cooler, and there’s some wind), and that I have a new set of two batteries (Extended primary battery, and the modular bay battery) which allow me up to 8-9 hours of work. Under high loads, this time tends to get shorter, but not less than 5 hours or so. Cool.