I just took the Brainbench Linux Administration (General) and I passed at the Master's Level with a score of 4.69 which puts me on the 1% top of this test takers. I made me feel a little better after screwing up half of the questions of my Google interview for a Linux administrator.

The skills certification I earned today is supposed to be a step towards a bigger certification for Systems Architect Network Support Specialist, Linux System Administrator or Information Security Administrator if I was willing to pay for more tests.

The test was pretty good, and the questions were not that outdated as on other tests. I thought it insisted too much on LILO, when almost all the big Linux distributions use GRUB bootloader. Also, I got no questions on iptables/netfilter. It told me that I have no weak areas, and I know I have. Taking the test I realized that I don't know about pam_auth as much as I want.

This is my scouts badge for mastering the Linux Administration test:

Master in Linux Administration

I got interviewed by a junior HR at Google for a Linux administration position. Now, I know that I know some stuff, but I also know that there's so much to know that I don't think I'm ready for Google yet. But Google didn't knew that, so they wanted to talk to me. I said, what the hell, I might as well talk to them.

On the phone interview, the recruiter asked me first some self assessment questions, in which I could have rated myself from 0 (know nothing) to 9 (wrote the book). I think I competent enough to know that are some really tough Linux gurus out there in the Stratosphere to rate myself below 5 on all questions. So my self-assessed score was between 0 and 5 out of 9.

Then the recruiter gave me a small three questions test. I could have chosen from "Linux Administration", "TCP/IP" and "Computer Science". Since Linux Administration is such a huge field and I forgot almost all Computer Science that I learned in University, I chose TCP/IP networking, a much smaller field.

The first question was: how many bits are in a mac address? I thought, six groups of hex numbers, a hexadecimal letter is 4 bits, so 4×6=24. Since I did that in my mind and my registers are limited, I forgot that there are TWO letters in each group, so it's 4×2x6=48. Whoops, got that wrong! Next one was to describe the first exchanges in a TCP handshake. I mumbled something along the lines: SYN, ACK, then SYN. Forgot about the ACK. The third question was how many usable IP addresses are in /23 network. I knew that if they asked usable it means that they want me to not count the broadcast and the network address... so the answer was 512-2=510. Got that right!

The recruiter congratulated me on the answer I got right and the one I got almost right (the TCP handshake) and told me to send him my resume so people up the line can have a look at it. Yeah, sure I will... k tnx bye, Bill (ironically, his name was Bill).

The question is: would I trade the experience that I'm gaining now working for diverse projects to working for the geek heaven that it's Google (even the recruiter, which was supposedly a business major had a typical geeky nasal voice). It's like going into a monogamous relationship. Would I go for that if I got a really good offer? Or would I still consider my independence more important?

I made a pretty complicated dev box / router for a client, which has three ethernet network cards and a wireless card, some complicated routing and bridging setup, an asterisk sip server, http server, ftp, smtp and pop3. Also, a lot of perl CPAN modules are installed and C and kernel development libraries.

Then I was asked make another one. I'm paid by the hour as a contractor, but even if it brings money into the bank I didn't want to spend another two days going to the process of reinstalling and configuring everything again.

I finished the job in one hour in a half using just netcat and dd (with some help for sfdisk for fixing partitions after that).

Boot with a live CD on the slave box, and open a listening socket:
slave# ip address add 192.168.0.254/24 dev eth0
slave# nc -l -p 8823 | dd of=/dev/sda

Then on the master, go to runlevel 1:
master# init 1
The bring up the interface:
master# ip address add 192.168.0.253/24 dev eth0
master# ip link set eth0 up
master# dd if=/dev/sda | nc 192.168.0.254 8823

In my situation, the slave disk was 20gig and the master was 40gig. Fortunately, I just needed the first 10 gigs. Unfortunately, the partition table was invalid and linux complained every time it booted. Fdisk and Cfdisk refuse to work on an invalid partition, so I had to use sfdisk
slave# sfdisk -d /dev/hda > partitions
This command will write the current tables on the file "partitions". Edit the file and feed it back to sfdisk
slave# sfdisk /dev/hda < partitions

Then, just edit the /etc/udev/rules.d/z25_persistent-net.rules for the new ethernet cards (This is on Debian testing, which uses udev).

That's it. Full cloning, over the network with basic tools. No Norton Ghost or other black box software.

Puppy Linux is a very lightweight Linux distribution very suitable for older laptops. A customer called me and asked about installing Linux on his recently bought off Ebay Toshiba Portege. He also wanted to replace the hard drive with a Compact Flash with an IDE adapter. Another problem was that the laptop was designed to be so lightweight and small that it didn't even had a CDROM drive. In this post I will describe how I installed Puppy Linux over the network, using PXE, LTSP and my trusty dev box which sits under my desk, running Ubuntu Linux.

The laptop cannot boot from a fixed drive, because it does not have a CDROM or a floppy. But... it can boot from the network with PXE! I can even "send" the operating" system over the network and have a fully working machine, even without a hard-disk. In Ubuntu, this is pretty easy, as documented here: https://help.ubuntu.com/community/UbuntuLTSP/LTSPQuickInstall
You basically have to do just:
sudo apt-get install ltsp-server-standalone openssh-server sudo ltsp-build-client

Activate the LTSP shell on the client:
Do this on your "server" (I did this on my Ubuntu workstation):
Chroot to /opt/ltsp/i386. (chroot /opt/ltsp/i386)
Copy /usr/sbin/sshd to /opt/ltsp/i386/usr/sbin.
Copy /etc/init.d/ssh to /opt/ltsp/i386/etc/init.d
Duplicate the symlink structure in /etc/rc*.d to the corresponding directories in the lstp tree so that ssh will start on boot.
Edit passwd and shadow in /opt/ltsp/i386/etc and add the entries for sshd from the files of the same name in /etc/
Create an appropriate set of host keys with ssh-keygen.

Or do it the smart & easy way (which I didn't think of):
apt-get install ssh (while you are chroot-ed).
Got this idea from https://help.ubuntu.com/community/HowToSetupLTSPDevelEnvironment

Run passwd to set a root password in the local /etc/passwd shadow. (passwd root)
OR
Add the user debug. (adduser debug)
Set a password for the debug user (passwd debug)
Add debug to sudoers (use visudo to see de sudo config file), add debug to the group that can sudo.

Now, login to your LTSP client (my Toshiba Portege):
ssh debug@192.168.0.250
(it may take a long time before you get a prompt, it's because you don't have a DNS, just wait for about 30 seconds for the DNS request to timeout)
Then on the box, type the folowing commands:
sudo cfdisk
Here you delete all partitions, and create a 1 gig partition on hda and mark it as bootable and make a extended partion and for your swap partition.
After that, create the filesystems:
sudo mkfs.ext3 /dev/hda1
sudo mkswap /dev/hda5

On the LTSP client the root filesystem is readonly, but you can write on /tmp. Tmp is a ramdrive device.
sudo mkdir /tmp/hda1
sudo mount /dev/hda1 /tmp/hda1
cd /tmp/hda1
sudo tar -xvf /srv/puppy.tar
puppy.tar is an archive of puppy linux.
sudo mv /tmp/hda1/media/hda1/* /tmp/hda1
sudo grub
You are now in the grub shell.
grub- root (hd0)
(choosing the first partition of the harddrive)

grub- setup (hd0)
you are installing grub in your hard drive MBR
you may want to install the boot loader on your bootable partition too (hda1): "setup (hd0,0)" does this.

grub- quit

Reboot your laptop and remove your ethernet cable so you can boot from the harddrive. You are done.

I'm using a Drupal 4.7 on this website and for all my pages it was showing a "page not found" error. The error.log file said
Premature end of script php5.cgi
Switching over the php4 didn't help either, just the error message changed a little:
Premature end of script php.cgi

After disabling some modules in drupal, I found the the
Textile module is the culprit. The Drupal 4.7 version is not maintained anymore and is from nov. 2006. I didn't want to upgrade to Drupal 5.x so I got to find another solution.
A solution is here http://leon.info.tm/en/salvation-textile-drupal-module, but it was to much work for me, the lazy programmer that I am.

So I went to the php.net website and on the PCRE docs I found that starting with PHP 5.2 two new configurations variables appeared: pcre.backtrack_limit and pcre.recursion_limit, both with default "100000".

The solution seemed simple. Open for editing sites/default/settings.php and add these lines:
// fix for textilePHP pcre bugs ini_set('pcre.backtrack_limit', '1000'); ini_set('pcre.recursion_limit', '1000');

The default for this variables are 100,000, which due to bugs in Textile or PCRE consumes all the available process stack and eventually crashes PHP. I guess you were wondering what those "core" files were doing in your site. Those are
"core" dumps of the memory when PHP crashes. You can safely delete them.

Remote Linux help and configuration is available from A+ Computer Services, a Chicago based IT support company.

Just use the email address contact@apluscompuservices.com to set up a instant messenger (AOL, Yahoo, Jabber) session and let me help you with your Linux server. Supported systems are CentOS, Redhat and RHEL, Debian and Ubuntu.

Prices are very reasonable and the Linux support is top-notch. As an example, I'm attaching a e-check from paypal from a customer in Indianapolis for whom I solved an "exim not sending mail from php application problem."

If you may have a large LAN, maintaining the DNS and DHCP server on Linux can be time consuming if you edit the zones and the dhcpd.conf file by hand. As a consequence, DDNS updates for ISC dhcpd server and BIND9 have been available for a few years.

But SELinux and CentOS/RHEL the procedure explained in the hundreds of tutorials on the net about is not sufficient. Let's start with a good and short tutorial for Fedora, a very close cousin of CentOS/RHEL:

A very important step is to set the permissions, because you probably made the zone files from the root console:
chown named:named /var/named/var/named/intranet.zone
chown named:named /var/named/var/named/intranet.reverse.zone
chmod 770 /var/named/chroot/var/named

After that, set the security context for SELinux:
chcon system_u:object_r:named_zone_t intranet.reverse.zone
chcon system_u:object_r:named_zone_t intranet.zone

Now, everything should work, right? Well, you will get this error:
Mar 20 18:53:31 localhost kernel: audit(1174434811.548:82): avc: denied { write } for pid=6969 comm="named" name="named" dev=md3 ino=957224 context=root:system_r:named_t tcontext=system_u:object_r:named_zone_t tclass=dir

It seems the named process needs to write to the zone file and SELinux it does not let him. Use the swiss army knife of SELinux: tail /var/log/messages | audit2allow. You will find out that you need to allow named_t named_zone_t:dir write;.

To edit SELinux policies on CentOS you need do yum install selinux-policy-targeted-sources. This creates a the src directory in /etc/selinux/targeted.

Take the line that audit2allow spit out to you and add it to the file /etc/selinux/targeted/src/policy/domains/program/named.te. Run make load in the src/policy folder. Sweet! Now it should will work! Well... guess again, because
only the error message has changed, subtly:
Mar 20 19:19:40 localhost kernel: audit(1174436380.095:87): avc: denied { add_name } for pid=6970 comm="named" name="intranet.zone.jnl" scontext=root:system_r:named_t tcontext=system_u:object_r:named_zone_t tclass=dir

You will need some this line: allow named_t named_zone_t:dir add_name;

I'll save you the pain and give you all the lines that you need to add to named.te:

allow named_t named_zone_t:dir write;
allow named_t named_zone_t:dir add_name;
allow named_t named_zone_t:file create;
allow named_t named_zone_t:dir remove_name;
allow named_t named_zone_t:file write;

And, success!!

Mar 20 19:41:41 localhost named[7669]: client 127.0.0.1#33008: updating zone 'intranet/IN': adding an RR
Mar 20 19:41:41 localhost named[7669]: client 127.0.0.1#33008: updating zone 'intranet/IN': adding an RR
Mar 20 19:41:41 localhost dhcpd: Added new forward map from HP29674251632.intranet. to 192.168.1.252
Mar 20 19:41:41 localhost named[7575]: journal file intranet.zone.jnl does not exist, creating it
Mar 20 19:41:41 localhost named[7669]: client 127.0.0.1#33008: updating zone '1.168.192.in-addr.arpa/IN': deleting an rrset
Mar 20 19:41:41 localhost named[7669]: client 127.0.0.1#33008: updating zone '1.168.192.in-addr.arpa/IN': adding an RR
Mar 20 19:41:41 localhost named[7669]: journal file intranet.reverse.zone.jnl does not exist, creating it
Mar 20 19:41:41 localhost dhcpd: added reverse map from 252.1.168.192.in-addr.arpa. to HP29674251632.intranet.

Some Linux admins turn off SELinux just because of this extra configuration steps when you do something unusual, like Windows Domain authentication with kerberos/winbindd or , like in this case, Dynamic DNS updates. Remember that chcon, audit2allow and the targeted-sources package are your allies.

I had a computer that was very slow for it's processor and memory: a Celeron 2.4Ghz with 512Mb RAM. Extracting the files from a 4MB zip file took 4 minutes and it took the processor to 100%.

Opening task manager (with alt+ctrl+del), and then going to the tab processes and sorting by CPU usage was telling me that "explorer" was taking all this processor power! When you see that, usually you need to think if

# Is the process really working at user level? or...
# Are there some system calls that are going wrong?

To analyze that, you have "Performance Monitor": click start -> run, and type "perfmon", OK. What you want to watch for is:

* leave on the cpu usage counter on
* add privilege time counter (very important)
* add system calls counter
* feel free to add some more that are helping you, like IRQ/s (which may detect a hardware problem), disk queue, bytes read/s, bytes written/s

Adjust the scale on each counter that is too low or high flat on your graph. Maybe you will end up with a graph configuration like mine:

What did perfmon told me? I found out that the privileged time counter was following closely the cpu usage counter and the system calls counter literally exploded upwards when I tried to to a simple unzipping.

Something was interfering with normal file open/read/write operations. It could have been only two:

# A virus
# An antivirus

The computer had the silly Yahoo SBC Online Protection installed, with a rebranded CA antivirus that "never catches anything". I uninstalled that and the computer was snappy as it was brand new.

Always look out for misbehaving antiviruses and real time protection programs when you're troubleshooting poor performance on your own computer.

I had to repair a Sony Vaio notebook displayed symptoms of overheating because of dust: very noisy fans and random shutdowns.

The Sony support page for PCGFRV31 does not provide a service manual. So I had to go back to the trusty old google. The two most helpful pages were:
A guy taking apart his Vaio, a different model, but helpful nevertheless
A company that's charging $150 to repair your Vaio laptop.

There were a couple of dust bunnies inside of the heatsink and I almost hit a wall when I couldn't (at first) remove the CPU from the heatsink because it was glued to it. With the use of a hammer and a razor blade - that's everyone in the business uses, no kidding - I managed to get it off from the heatsink.

When it booted up I was the happiest man on the planet. I always have a feeling of joy when I fix a computer problem, but with that laptop it was really a fight - man vs machine to fix it.

Where this computer repair and data recovery service call was located: near Ashland & Bryn Mawr, Chicago).

After some previous appointed work I was back at he office at about 1:00 PM. Checked my email, checked website statistics... and then nothing to do.

So I guessed... right after I thought "This is going to be a slow day", I got a phone call from a feminine voice (for unknown reasons, most of my clients are male):

"Are you doing data recovery services?", asked my new customer, Sarah. "I can't get into Windows anymore".

"Do you see the windows logo screen?", I asked, knowing that we don't do much data recovery if the hard drive is a "dead lemon".