A C Zoom

This was supposed to be easy - extending logical volumes. But if you install a virtual machine, then it all becomes a mess. Search the web for how to extend a partition "nested" in an LV, and there are only questions and no answers.

KVM Disk Management Issues shows an alternative to using the standard install - "just put a filesystem on it and you are done" which basically means that manual partitioning should be chosen during a Ubuntu install. Resize KVM Image shows another alternative which basically involves deleting the swap partition inside the KVM which allows the root partition to be enlarged. That would be necessary when there is no nested LVM, when the partitions were created in the hosts' logical volume. And for a general introduction to LVM see Logical Volume Management (IBM).

virt-manager makes it easy to install the virtual machine using an .iso image of the OS to install. It is not easy to resize storage on a KVM virtual machine, if installed following the standard instrutions - make a logical volume on the host machine, let the virtual machine installer use it like a raw disk to create its partitions, and install the OS.

Well, the trouble is now that the standard LVM resize procedures are not helpful. This is what the picture looks like, where VG is the host machine volume group:

Host Machine:

   vg:VG
    -- lv:kvm1
    -- lv:unused (lot of unused space on the volume group VG)

lv:kvm1 is the logical volume used to install the virtual machine. Let us say it is a Debian-based system, like Ubuntu. Using the default "Guided Partitioning", it will install a logical volume on the disk.

    -- lv:kvm1
       -- vg:VIRT
          -- lv:vroot     --- this is the partition we want to extend
          -- lv:vswap

This site uses Drupal. Drupal has turned into a nightmare. It was fine when there was a single 4.x version out there, but soon after 4.x, there was 5.x. Then 6.x. Upgrading from a older version is near impossible.

There always was the assumption that some amount of coding would be required by anyone running a Drupal site. But be prepared - you will be hacking modules left-and-right to get any thing running. At this time, one has to question whether the amount of hacking required to get things to run are worth it. Maybe all CMSes have this problem, but certainly Drupal is really a poster-child for impossible-to-ever-upgrade software.

The problem occurs because Drupal changes the API every release, adds new incompatible features, and modules and themes become unusable. And since modules and themes are merely someone's weekend project, it can be months or years before a module becomes compatible with the newer Drupal version.
Core drupal does not have image handling capabilities or spam fighting capability so even a basic site will need to use external modules. Add things like forums, automatic aliases, FAQs, it becomes a large collection of non-core modules.

The advantage of Drupal is that it is extensively customizable, and has a wide range of modules. This is exactly the same thing that makes a Drupal site near-impossible to upgrade. Once a site is up and starts to depend on a bunch of modules, rest assured that when a new Drupal version comes out quite a few required modules will not make it to that new version!

Drupal core does get upgraded without problems. But Drupal itself has become super-bloated. Web hosts that worked fine with Drupal 4.7 will not support Drupal 6.x because of heavily increased memory and CPU requirements.

dd-wrt is third-party firmware that can be loaded on many routers and it makes available many additional features such as advanced routing as well as a keep alive mechanism. It is maintained by BrainSlayer.

In the few days of using it, some advantages of dd-wrt are evident. It has been far easier to configure on my network of Linux and Windows computers, which use both static and DHCP IP addressing. The bundled Linksys software on the new WRT54G V8 device had long DNS lookup times on the Linux computers (probably needed to use the remote DNS resolvers instead of pointing to the Linksys box), for all lookups, at all times. But instead of re-configuring the Linux boxes, in the same amount of time, it was quite easy to install dd-wrt micro-edition using these instructions: How to Flash Linksys wrt54g

Notes on things to look out for working with Fedora Core installs.

perl CGI scripts hanging

Fedora Core 7 current perl package is perl-5.8.8-27 [Jan 2008]. This perl package contains version 3.15 of CGI.pm, which has a problem handling POST_MAX. Any HTTP post of size greater than POST_MAX value will cause the CGI script to peg the CPU at the $q = new CGI statement and not terminate for a very long time. After the CGI script is automatically terminated, a HTTP 500 Internal Sever Error status code returned to the requester.

This problem exists in all distributions using Perl 5.8.8, and is not limited to Fedora Core 7.

This issue is very hard to track down. Most of the time, the CGI scripts will work fine. When they do fail, there will invariably be nothing more than a single line in the Web server log stating that the job took around 300 seconds and a HTTP 500 Internal Server Error message was sent back. The times at which the failures occur will be all over the clock, there will be no pattern to the time when the failure occurs. If you do manage to see this rare failure happening, the CPU will show that all free CPU time is being used by the CGI script - but the script has no logging or database activity thus indicating that it locked up very early in the script. That will eventually lead to looking at the $q = new CGI statement, and to the fix.

The fix is simple - upgrade to CGI.pm version 3.21 or later. But to make this stick will require some additional configuration because while CGI.pm can be updated on its own, it is also bundled with perl. So to avoid a future yum update restoring CGI.pm back to version 3.15 when perl 5.8.8 is updated, requires preventing perl from being updated, or keeping two copies of CGI.pm around and changing scripts to load the right one.

A very small site started running into performance problems - some pages taking too long to load, and certain MySQL queries taking over 5000 to 6000 milliseconds, and being killed because of resource limits set on the hosting computer. The pages affected were the watchdog log display pages - one of which is the Menu -> administer page when logged in as the adminstrator, and it displays data from the watchdog table.

This seemed odd - for a site with less than 200 nodes, and very low traffic, there should be no performance issue, and no single database query should be taking as long as 6000 milliseconds.

So the options were to increase the time limit for queries, or to spend the time debugging the problem.

Drupal is very feature rich, and this may have negative impact on performance, but in this case, it turned out to be a database issue. The Drupal site has many performance related tips, including a subsection on Tuning MySQL for Drupal.

After looking around in the database for the site, it was discovered that the overhead for the watchdog table was over 40 times its actual size! So, the size was 176MiB, and the overhead was 172MiB. Running optimize on this table got the size down to under 4MiB, overhead to 0, and got the queries to be much faster - way below the 6000 millisecond time limit, and the administer and log display pages now rendered much faster, way below the old times.

One question remains - why did removing overhead fix the query times?

Time has never been easy to work with, and while it is best to use UTC to store time, it is necessary to use local time zones when displaying time to the user.

With regards to displaying local timezones using strftime and the format specifications %Z for timezone name and %z for the RFC-2822 conformant [+-]hhmm displays, Perl and PHP work fine, at least on Fedora FC5. Python 2.4 does not yet have support for %z, and the best support in Python for timezone is using the basic time module; the enhanced datetime module has no built-in support for time zones.

That takes care of strftime, what about strptime? Unfortunately, that is a topic that is even more convoluted, so try all variations out before you use it on any system.

Here's a strftime support summary for Python date/time users:

• Use the datetime module if you don't need any timezone handling. The standard library comes with no support for any timezones, not even local timezones, so datetime module is useless for those who need to use time zone information. This can be done of course, but requires coding up your own timezone routines.
• Use the time module if you can live with just %Z and don't need %z. Python 2.4 time module always prints wrong value +0000 for %z, even while it gets %Z correct.
• Need %z, the RFC-2822 conformant time display? This requires writing your own code, Python does not support this. Using the basic time module, here's example code on how to get these values:
  

  lt = localtime(t)
  if lt.tm_isdst > 0 and time.daylight:
      tz = time.tzname[1]
      utc_offset_minutes = - int(time.altzone/60)
  else:
      tz = time.tzname[0]
      utc_offset_minutes = - int(time.timezone/60)
  utc_offset_str = "%+03d%02d" % (utc_offset_minutes/60.0, utc_offset_minutes % 60)

  Note: in the utc_offset_str computation, the use of 60.0 float in the / operation is necessary to get a value rounded to 0 instead of negative infinity, for example, -90 minutes offset should be -0130 and not -0230.

Here's the sample code and the output from Perl, Php, and Python, for two example strftime calls:

Updating drupal using the standard instructions is very time consuming - have to turn off modules/themes, update settings, reinstall modules/themes.

But many users find that for many updates to the same major release, for example, 4.7.x series, simpler upgrades can work - official Drupal install instructions do not allow this, nor is the structure of Drupal folder structure a help, since user installed modules/themes are in the same folder as the drupal files (would be good to have these separate), and no easy way exists to try out a new release before switching to it on a running site.

Given all that, here's how an update can be done fast - very important to read the UPGRADE.txt Drupal document first, and do all backups, and be ready to restore quickly if things don't work.
No guarantees on this method, but it has been known to work.

Assume that old drupal install is in current/ and new one is new/

1. extract the new drupal release, into the new/ directory.
2. copy over all the new files and directories from your current install, to this new/ directory. See script "update.sh" below which does this.
3. update the config files - sites/default
4. login to current drupal as admin
5. backup: rename current/ to current.backup/
6. rename new/ dir to current/
7. run current/update.php from drupal and on success, log off the admin user. and test it out

And here's the update.sh script - edit $OLD variable, run this from inside the new/ directory, and redirect output to update.run, take a look at the update.run commands, and then run the commands in update.run:

#!/bin/sh
OLD="current/"

echo "# Current Directory: " `pwd`

for i in `cd $OLD; find . -print`
do
    if [ ! -e "$i" ]
    then
        if [ -d "$OLD/$i" ]
        then
            echo mkdir -p -v "$i"
        fi
        if [ -f "$OLD/$i" ]
        then
            echo cp -p -v -i --reply=no "$OLD/$i"  `dirname "$i"`
        fi
    fi
done

Fun with perl vs php regular expression handling.

PHP has a regular expression search and replace function, preg_replace. This is supposed to be the standard greedy algorithm for patterns, for example, A* means zero or more A characters, and it matches the longest string of A's at that point.

So, if you want to match zero or more slash (/) characters at end of a string the pattern to use is: /*$

And, if we want to replace zero or more slash characters at end of a string with a single slash character, the perl code is:   s!/*$!/!

Full example in perl:

foreach ('path/to', 'path/to//', 'path/to///', 'path/to////') {
    my $s = $_;
    $s =~ s!/*$!/!;
    print "string: $_ changed to $s\n";
}

Output is: 
string: path/to changed to path/to/
string: path/to// changed to path/to/
string: path/to/// changed to path/to/
string: path/to//// changed to path/to/

And the above output is all correct.

Equivalent PHP code does not work, using PHP version 5.1.6.

Following PHP code:  $t = preg_replace('!/*$!', '/', $s); fails, it will end up with one / if the input has 0 or 1 / characters, but it will end up with two / characters if the input has 2 or more / characters, instead of a single / character. So, the match was not greedy, but for reason, was split into two matches, and then each matched group was replaced with a single / character.
The workaround for this is to specify a limit count of 1 to preg_replace, which makes the code work fine. preg_match seems to work fine, only preg_replace has this problem.

Here's php code, and its output, showing the failure:

<?php
foreach (array('path/to', 'path/to//', 'path/to///', 'path/to////') as $s) {
     # preg_match('!/*$!', $s, $matches); // works fine, is greedy - $matches[0] is zero or all slashes
     # $t = preg_replace('!/*$!', '/', $s, 1); // works, limit of 1 helps

On a Fedora FC5 system, it is very easy to update the software using yum update.

This also updates the kernel, which is good, but given the number of "non-standard" items installed on my system (VMWare Server, NVidia Display, PVR-150 TV Tuner/MPEG Encoder), it requires a lot of fixing up after every yum update. [So I have automatic update disabled, do the update only when time permits to do all the fixups.]

After a kernel update:

• Check to see that the matching kmod-nvidia-* module also made it in, otherwise the display won't work well. In some cases, the kmod-nvidia module site is down, so may have to build module by hand.
• yum install ivtv ivtv-kmdl-`uname -r`
  to get the driver for the PVR-150 TV-Tuner/MPEG Encoder card.
  This should also install the firmware, if it does not, visit the ivtvdriver pages to do the firmware install manually.
• If mplayer has problems with audio, check alsa-lib. I have the atrpms site listed for updates, and that is considered risky, packages may not match the other sites. In Sep 2006, started to get this error when using mplayer:
  alsa-lib: pcm_direct.c:1632:(snd_pcm_direct_parse_open_conf) Unknown field ipc_sem
  The fix is to remove alsa-lib:
  yum erase alsa-lib.i386
  (which may uninstall packages like firstboot, and control-center,m which I don't use )
  and then re-install it using this:
  yum --disablerepo=atrpms install alsa-lib
  (and if needed, reinstall firstboot, control-center, etc).
• Rerun vmware-config.pl

Why have a blog in Drupal? After all, the story node is generic enough, and for single-user sites, good enough to use story for all postings.

Still, I added the blog module to my site.

The blog is useful for dated articles - some postings only apply for a limited duration, and may not be valid after a few days or months.
Those kind of articles are better posted in a blog.

So, will be using the "story" node - article is probably a better description - for tools, travel tips, recipes, etc - things that may still be valid for months or even an year or more.
Will use the blog to post items that may be relevant only for a short while.