about:benjie

Image via Wikipedia

It has bothered me a little for a while that the MythWeb (part of the fantastic MythTV package for Linux) aspect ratio is hard-coded to 4:3. Most of the TV that I watch (received over Freeview (DVB-T) in the UK) is in 16:9, so watching it back on 4:3 is a bit of a pain. Thus I was motivated to change the hard coding to 16:9. The process is quite simple:

1. Modify line 102 of mythweb/modules/stream/handler.pl - change “3/4” to “9/16“.
2. Modify lines 35 and 37 of mythweb/modules/mythweb/tmpl/default/set_flvplayer.php - change “3/4” to “9/16” and “4:3” to “16:9” respectively.
3. Modify line 505 of mythweb/modules/tv/tmpl/default/detail.php - change “3/4” to “9/16“.
4. Optional: I also added to the end of line 165 of mythweb/modules/stream/handler.pl (which detailed the ${width}x$height) - adding
   .’ -aspect ‘.shell_escape(”16:9″)
   (make sure you get the fullstop at the beginning!), though I am not sure if this modification is necessary or even beneficial!

There is a minor bug now where the player does not show the control bar at the bottom properly initially, but a click on the preview picture solves this.

I’m currently working on modifications to stream the video in 3gp format to my mobile (a Nokia 6120 Classic), however this seems a lot harder as I have to implement a RTSP server, and have to re-encode all jobs in advance (by using a MythTV User Job) which is not quite what I am after. There is a page about it in the MythTV wiki. I wonder if I can find a cunning way around it…

If this helps you, please let me know in the comments!

Image via Wikipedia

I told you about moving my data over to a RAID1+0 and RAID5 system in this previous post, but, as expected, I never got round to finishing it. Until now, that is…

I went to London on business for a few days, and came back to find MythTV had stopped working. The disk was full, so it had given up, and wouldn’t even let me get in to view the recordings - so I couldn’t even delete some to get it working again! (I wonder why Auto-Expire wasn’t working.) Anyway, this spurred me on to finally finish partitioning/setting up my drives. The process was quite simple, really. The following steps generally assume you are root already (sudo su) because I am a “bad” sysadmin and don’t believe in all this constant sudo malarchy. Following my instructions is, as always, at your own risk. I highly recommend that you read the relevant documentation before proceeding (such as this).

1. Reboot into Ubuntu LiveCD, don’t forget to upgrade the software on the LiveCD to prevent any issues!
2. Disable swapspace (swapoff /dev/sd[abcd]1).
3. Use cfdisk to finish formatting the drives (remembering to change partition types to the hexadecimal “fd” - Linux RAID Autodetect). Be careful NOT to modify ANY of the details of the partitions you are already using for data or you will probably lose data!
4. Reboot back into your real system (not LiveCD - minimizes downtime).
5. Optional: add hot spares to current RAID5 devices (mdadm /dev/md1 -a /dev/sdf5).
6. Create the new RAID6 devices (mdadm -C /dev/md3 -l6 -n6 /dev/sd[abcdef]7).
7. Optional: wait for the devices to finish resyncing (watch cat /proc/mdstat).
8. Turn the new RAID devices into LVM physical volumes (pvcreate /dev/md3).
9. Stop any services that depend on /data (/etc/init.d/mythtv-backend stop; /etc/init.d/mpd stop).
10. Unmount the data drive (umount /data).
11. Add the new physical volumes to the current LVM volume group, “raid5” (vgextend raid5 /dev/md3).
12. Expand the logical volume to the full size of the volume group (use pvdisplay to find out the size [411.05GB], and then run lvextend -L+411.05G /dev/raid5/data).
13. Expand the filesystem [ext3] to the full size of the logical volume (e2fsck -f /dev/raid5/data; resize2fs /dev/raid5/data) - running e2fsck on a 600GB drive does take a while… Took about 30 minutes for me with little else running.
14. Remount the data drive (mount /data).
15. Restart the services you stopped eariler (/etc/init.d/mythtv-backend start; /etc/init.d/mpd start).
16. Remember to update (I prefer to re-create) your initramfs and update grub (update-initramfs -k all -c; update-grub)
17. All done!

I chose RAID6 over 6 disks over RAID5 over 5 disk with one hot spare because it has better redundancy and similar performance. It was added to the kernel at the end of 2003 so I think it should be fairly stable by now.

This process was not too complicated, and all of this can be done with very little downtime (if you are clever/daring, you can even re-partition without rebooting, but that was too risky for me!). You can even do the LVM stuff without taking /data offline! I wouldn’t advise it though.

If this post helps you, please let me know in the comments.

… well, not any more - most of it has melted now - but it was this morning! Snow during April, what is the world coming to? I have a few theories:

1. Global warming is starting to take hold
2. There are actually 53 weeks in a year, not 52, and thus our calendars are wrong and winter comes a week later each year.
3. Government conspiracy.
4. Its always been this way, but we’ve had a few years of warmth that have made us forget the snow in March/April.

Anyway, enjoy the pictures from Gosport!

Image from Wikipedia

Image from Wikipedia

Yesterday I tried to install the latest version of synce in order to get Jem’s Dad’s Windows Mobile 6 phone to share files with Linux (Ubuntu Gutsy Gibbon in this case). After managing to get the software installed, I have been very impressed with it, however actually installing it was a bit of a challenge, though the solution is quite simple and I share it with you now.

1. Uninstall everything synce related before starting.
2. Follow the Synce with Ubuntu instructions.
3. pls should work at this time.
4. Follow the SynceVfs instructions.
   Use ./configure –prefix=/usr
   make; sudo make install
5. Heres the important bit:
   cp /usr/etc/gnome-vfs-2.0/modules/synce-module.conf /etc/gnome-vfs-2.0/modules/
6. killall gnome-vfs-daemon

I think that you can do step 5 alternatively by adding –sysconfdir=/etc to your ./configure command in step 4, however I have not tested this.

Once this is done you should be able to just plug your phone (or other Windows Mobile device) in to the USB, and type synce:/// into Nautilus’ address bar. Simple!

Image from Wikipedia

I moved my blog (in fact my entire website!) over to Wordpress a couple of days ago. The move was not without it’s challenges - for a start I remembered Wordpress likes to have a well defined hostname, and I didn’t want any downtime. To get around this, I placed an entry in my /etc/hosts file for www.benjiegillam.com, pointing to the new domain, this way I could set up the new Wordpress blog privately (no-one else would know where it was) under the correct domain name, whilst still having access to the old blog to copy content over from.

My first issue was how to transfer the posts from the old blog to the new. I acheived this by doing a few minor hacks to serendipity, and using the export function (where you can export all posts as an RSS feed). To do this, I had to disable the “extended body” feature (i.e. make sure it was output as part of the feed), as explained in solution, part 1, here. Make sure your browser is not caching at this stage!

Once I had acquired the RSS file, I then had to convert it into a format that wordpress would understand. I cheated and wrote a very bad PHP file, here:

<?php //Import the feed $rss = file_get_contents('s9y.rss'); //Opening <![CDATA[s $rss = str_replace("<content:encoded>","<content:encoded><![CDATA[",$rss); //Closing ]]>s $rss = str_replace("</content:encoded>","]]></content:encoded>",$rss); //Now replace all newline characters with a " " (this will BREAK any preformatted tags, but will stop wordpress putting <br />s everywhere $rss = str_replace(array("\n","\r")," ",$rss); //Finally remove all the htmlentities from the file and output to STDOUT, which you can then redirect to a file echo html_entity_decode($rss,ENT_COMPAT,'UTF-8'); // I called this as convert_s9y_rss.php > wordpress.rss Download this code: /code/convert_s9y_rss.phps

I then used Wordpress' RSS importer to import the posts (no comments, unfortunately). I then copied all of the uploaded files into the same file structure on the new site. The next thing to do was to go back through and edit all the posts and update their links. Only joking, I really couldn't be bothered to do that! Instead, I made a folder called "serendipity" in the webroot (all of my posts were /serendipity/archives/... previously), and placed in it the following two files:

RewriteEngine On #Direct *EVERYTHING* to the index.php file RewriteRule .* index.php [L] Download this code: /code/s9y_htaccess

<?php //What URI was I accessed as? $uri = $_SERVER['REQUEST_URI']; //Remove everything except the last section $uri = explode("/",$uri); $uri = array_pop($uri); //Convert to lower case (as in Wordpress) $uri = strtolower($uri); //Remove the post id from the beginning of the post $uri = explode("-",$uri); array_shift($uri); $uri = implode("-",$uri); //Remove the extension (.html) $uri = explode(".",$uri); array_pop($uri); $uri = implode(".",$uri); // Now send a 301 Moved Permanently and the new location header("Location: /$uri",TRUE,301); exit(); Download this code: /code/s9y_index.phps

These caused all posts links to be re-written to a guess at the page name, and thankfully Wordpress was clever enough to work out what was meant. I am not sure if it worked for all posts, but it did for all that I tested.

I hope this helps someone, if so leave me a comment (please! I lost all my old comments in the move!).

Image from Wikipedia

On 5th February, only of my hard drives failed, and my computer started to choke. My root partition was on the failed drive, so only programs that were already in RAM could continue to run, though most of my storage was on a RAID-5 across 4 320GB disks, so that was still intact. Fortunately, due to my 6GB of RAM, swap was not in use also, so this did not crash my PC, but it did leave me in quite a bad position, not being able to run many of the built in system tools, in particular the tools from “smartmontools”. I did what I could to copy the important details over to Jem’s PC whilst my system was still running, albeit crippled.

It turned out that, upon rebooting of the system, everything was fine, and the disk worked, but it didn’t half give me the willies! I vowed then to get my root drive onto some form of redundant storage, and to have a hot spare always on hand. To that end I bought an Icy Dock 5-in-3 SATA cage and 2 500GB SATAII drives. Unfortunately, due to the amount of time I was spending working on buzzspotr.com with i-together, I was unable to actually use these immediately. I finally got round to incorporating them into my system last weekend. It was quite a challenge to do, so I thought I would document it for future reference.

The first thing I did was delete as much data as I could. The main things I focussed on were:

• Old MythTV programs that I had recorded and seen, or that I was not going to watch
• My ripped DVD library (I rip my DVDs to make it easy to get them to play from MythTV without having to get out of the sofa! I could always re-rip them later)
• Old duplicate backups (for example, I had backups every 30 minutes for Blog Friends, which summed to almost 50 GB! I removed all of these except for those from 11:30pm each night)
• Duplicate files
• Caches

After removing all this data I reduced the “valuable” data on my computer to somewhere around 650GB.

I decided that the best way to lay out my computer data would be to have the following:

• First 0.5GB of each drive - swap space and /boot partition. I chose to not make the swap redundant as it is rarely used (and I don’t mind if computer crashes if it HAS to!)
• Next 39.5GB of each drive - RAID1+0 for /, totalling 117GB of fast redundant storage (theoretical peak bandwidth: 1800mbps read, 900mbps write). High priority data here - the root, my home directory and desktop, the mysql databases, the webroots of apache, etc etc. Ultimately everything where speed and redundancy are highest priority. This setup allows the loss of any one, and possibly up to 3, drives, and has less probability of a total fail than RAID0+1. I did *NOT* use the kernel raid10 driver.
• Then 80GB stripes over 5 disks of RAID5, with one hot spare (on the 6th disk) up to the capacity of the 320GB drives, which would all be combined through LVM into one huge partition for lower priority data - music/TV/etc. Note that RAID5 write speed is not great.
• The rest of the 500GB drives are currently unpartitioned, but I might use them as overflow for MythTV, or as hot spares for the RAID1+0

This seemed to me to be the best way to lay out my filesystem, but how on earth could I move my current data over to the new system, and be assured that it would still boot?

My previous setup was (4 320GB drives, remember):

• First 0.5GB swap and /boot
• Then 30GB stripes for RAID5, up to the last 20GB. Combined with LVM, two logical volumes.
• Last 20GB was used for /home on one drive, 64-bit / on another, and a 32-bit / on another. The final drive was blank.

My first issue was how to boot into the Ubuntu LiveCD, and still have volume management. I found the best way to do this was the following:

1. Boot the Gutsy Gibbon LiveCD, remembering to set screen resolution and keymap (for some reason, it crashes for me if I don’t…)
2. Open up synaptic
3. Edit the software sources (repositories) - tick all the boxes, and all the updates boxes (gparted is broken on the LiveCD if you try and use it on a completely raw (fresh from manufacturer) drive)
4. Install all the updates (you can leave out obvious things like OpenOffice.org if you want)
5. Install mdadm and lvm2 packages
6. (Optionally?) Run modprobe raid0; modprobe raid1; modprobe raid5
7. Then run mdadm -A -s –no-degraded
8. (Optionally?) Run modprobe dm-mod
9. Run vgchange -ay
10. Now you should have all your RAID and LVM partitions up and running

If you don’t understand the commands, I highly recommend that you read their man pages to ensure that these are the right commands for you. You CAN lose data if you mess this up! I always have to check each command 4 or 5 times before I run it when this much data is involved!

Once I had done that, I had to take the plunge. First I checked each of the NEW UNFORMATTED disks with a long read-write test (actually, I did this before ever rebooting), by running badblocks -s -w /dev/sde. This is a destructive command, please be careful using it! It will erase any data already on the drive.

Once I was convinced that the drives would withstand the 2 days where they would be the critical data point, I partitioned them. Both got the standard 0.5GB and 39.5GB partitions at their fronts, and then sde got 3 80GB partitions, and the rest (220GB) turned into another partition; whilst sdf got the rest (460GB) turned into one large partition. I then copied everything over to these drives (starting with sdf, and then working backwards through the partitions in sde). I then had to take the jump and make my RAID1+0 (which was formed of striping the pairs sda-sdd, sdb-sde, sdc-sdf).

It was at this point I thought I would be clever. If I just deleted the partitions of sda, then the RAID5 would still be holding the data, and I could make my severely cripped RAID1+0 (really only a RAID0 in this idea) by combining sda5, sde5 and sdf5. I could then copy the data over and check if it booted, whilst still having lots of redundancy for my data. Unfortunately, the system would not boot like this (I guessed it was because I had two md0s - one for RAID1+0, and one from the old RAID5, though I was later proven wrong), so I had to give up and take the risk. I deleted the partitions from the other drives, and formed my RAID1+0, and tried to boot into it. It still would not boot. I even chrooted into the new environment and ran update-initramfs and update-grub, but still nothing.

At this point I was a little frustrated, and spent a long time researching. In the end I discovered that the initramfs was not being updated, and it still contained the old /etc/mdadm/mdadm.conf. Upon deleting and regenerating the initramfs, I could boot into the system. I quickly rebooted into the LiveCD and did all the other necessary changes (setting up the RAID5 across the 5 available disks (leaving the last disk with the data on), copying the data from sdf onto the new RAID5, updating the fstab, etc). I then rebooted into the system, and (not surprisingly) had to make some quite considerably changes due to the amount of data I had moved to new, “better”, locations. And finally, just 2 days later(!), I had my ultra fast and acceptably redundant system online.

I’m very glad that I took the time to do this, though I still have not got round to formatting sdf and setting it up as hot spares… it still has most of the old data on as a duplicate copy!