Less than 24 hours after release, Blog Friends has reached 100 users!

Blognation have done a review of Blog Friends here.

If you haven’t already, why not read my previous article: Blog Friends RELEASED!

Brain Bakery Ltd. have been working for the last two weeks on a facebook application called Blog Friends, and finally it has been released! I have been logging the hours I spent on the project and it appears that in the last 14 days I have worked the equivalent of almost 4 standard 40-hour weeks - and that’s just me, the others at Brain Bakery have also worked quite a bit! Thats pretty hardcore, I think, so I will be chilling out a bit now…\ Anyway - if you have a blog, and you would like your friends blog posts filtered for you according to a list of your interests/dislikes, why not give Blog Friends a try? If you have problems with the software, please leave a comment or email me and I will get back to you ASAP. Also, if you have suggestions for version two, feel free to post them here, though Blog Friends has absolutely loads of features in the pipeline…

Hope you enjoy it!

I was just helping my father-in-law-to-be to install MythTV on his old computer to run through his TV. It took quite a while to get the computer (with an AGP MX440) to get running under ubuntu through the TV, but we did it in the end. Read on for details.

Here is the XOrg config file:

Section “Files”\ FontPath “/usr/share/fonts/X11/misc”\ FontPath “/usr/share/fonts/X11/cyrillic”\ FontPath “/usr/share/fonts/X11/100dpi/:unscaled”\ FontPath “/usr/share/fonts/X11/75dpi/:unscaled”\ FontPath “/usr/share/fonts/X11/Type1”\ FontPath “/usr/share/fonts/X11/100dpi”\ FontPath “/usr/share/fonts/X11/75dpi”\ # path to defoma fonts\ FontPath “/var/lib/defoma/x-ttcidfont-conf.d/dirs/TrueType”\ EndSection\ \ Section “Module”\ Load “bitmap”\ Load “ddc”\ Load “dri”\ Load “extmod”\ Load “freetype”\ Load “glx”\ Load “int10”\ Load “vbe”\ EndSection\ \ Section “InputDevice”\ Identifier “Generic Keyboard”\ Driver “kbd”\ Option “CoreKeyboard”\ Option “XkbRules” “xorg”\ Option “XkbModel” “pc105”\ Option “XkbLayout” “uk”\ EndSection\ \ Section “InputDevice”\ Identifier “Configured Mouse”\ Driver “mouse”\ Option “CorePointer”\ Option “Device” “/dev/input/mice”\ Option “Protocol” “ImPS/2”\ Option “ZAxisMapping” “4 5”\ EndSection\ \ Section “InputDevice”\ Driver “wacom”\ Identifier “stylus”\ Option “Device” “/dev/input/wacom”\ Option “Type” “stylus”\ Option “ForceDevice” “ISDV4” # Tablet PC ONLY\ EndSection\ \ Section “InputDevice”\ Driver “wacom”\ Identifier “eraser”\ Option “Device” “/dev/input/wacom”\ Option “Type” “eraser”\ Option “ForceDevice” “ISDV4” # Tablet PC ONLY\ EndSection\ \ Section “InputDevice”\ Driver “wacom”\ Identifier “cursor”\ Option “Device” “/dev/input/wacom”\ Option “Type” “cursor”\ Option “ForceDevice” “ISDV4” # Tablet PC ONLY\ EndSection\ \ Section “Device”\ Identifier “MX440”\ Driver “nvidia”\ BusID “PCI:1:0:0”\ Option “UseFBDev” “true”\ Option “TwinView” “True”\ Option “TwinViewOrientation” “Clone”\ Option “ConnectedMonitor” “CRT,TV”\ Option “SecondMonitorHorizSync” “30-50”\ Option “SecondMonitorVertRefresh” “60”\ #Option “MetaModes” “1280x1024, 640x480; 1024x768, 640x480; 800x600, 640x480; 640x480, 640x480;”\ Option “MetaModes” “800x600,800x600;”\ Option “TVStandard” “PAL-I”\ Option “Xinerama” “0”\ # Option “UseInt10Module” “true”\ \ EndSection\ \ Section “Monitor”\ Identifier “Generic Monitor”\ Option “DPMS”\ HorizSync 28-38\ VertRefresh 43-72\ EndSection\ \ Section “Screen”\ Identifier “Default Screen”\ Device “MX440”\ Monitor “Generic Monitor”\ DefaultDepth 24\ SubSection “Display”\ Depth 1\ Modes “800x600” “640x480”\ EndSubSection\ SubSection “Display”\ Depth 4\ Modes “800x600” “640x480”\ EndSubSection\ SubSection “Display”\ Depth 8\ Modes “800x600” “640x480”\ EndSubSection\ SubSection “Display”\ Depth 15\ Modes “800x600” “640x480”\ EndSubSection\ SubSection “Display”\ Depth 16\ Modes “800x600” “640x480”\ EndSubSection\ SubSection “Display”\ Depth 24\ Modes “800x600” “640x480”\ EndSubSection\ EndSection\ \ Section “ServerLayout”\ Identifier “Default Layout”\ Screen “Default Screen”\ InputDevice “Generic Keyboard”\ InputDevice “Configured Mouse”\ InputDevice “stylus” “SendCoreEvents”\ InputDevice “cursor” “SendCoreEvents”\ InputDevice “eraser” “SendCoreEvents”\ EndSection\ \ Section “DRI”\ Mode 0666\ EndSection

Not quite sure why but I had to disable the int10 thing. Ubuntu seems to have a bug that won’t let it load. Also, I could not get it running on monitor and tv at same time, but it was only the TV we wanted anyway, so shrugs meh.\ \ Currently we are installing MythTV, wish us luck! I’m not even sure if the freeview (DVB-T in the UK) card I gave him works…

If this post helps you (or if it doesn’t…), please leave a message.

I made this command to download a series of websites including all files, and to do so without being stopped by any automated protection methods (e.g. robots.txt, request frequency analysis, …). It served its purpose well. The command is this:

“ wget -o log -erobots=off -r -t 3 -N -w 2 –random-wait –retry-connrefused –protocol-directories –ignore-length –user-agent=“Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.8.1.2) Gecko/20060601 Firefox/2.0.0.2 (Ubuntu-edgy)” -l 100 -E -k -K -p http://web.site.here/

Quite long, don’t ya think? Read on for a description of what it does (that is, if you don’t have the wget manpage memorized…)

Basically it does this:

• -o log –> outputs messages to log file “log” instead of the terminal. Allows for easier debugging (and you can tail -F it in another terminal anyway…)
• -erobots=off –> tells wget to NOT respect the robots.txt file - i.e. abuse webservers. This is naughty, but was necessary for the sites I wanted.
• -r –> recursive
• -t 3 –> retry each URL 3 times
• -N –> turn on timestamping
• -w 2 –> wait 2 seconds between requests
• –random-wait –> instead of waiting 2 seconds, wait a random time, that averages to 2 seconds
• –retry-connrefused –> for sites that go down frequently (e.g. ones on home computers) use this to retry if the connection is refused, rather than just skipping
• –protocol-directories –> makes files from http://web.site.here/dir/ be laid out as ./http/web.site.here/dir/ e.g. puts https files in a different directory to http files
• –ingore-length –> If the webserver gives the wrong Length: header, ignore it.
• –user-agent=“” –> pretend to be Firefox rather than wget - prevents connections being refused by anti-spidering measures that filter just by user agent.
• -l 100 –> go up to 100 levels deep
• -E –> put .html on the end of files that don’t end with HTML but are HTML files. This is useful so that you can download a dynamic site and stick it in a static webserver and it still works
• -k –> convert links - VERY ADVANCED - edits the HTML files and changes the links to be relative and point to the right file names (e.g. adds .html for the last option). NOTE: this only works for HTML files - you will have to modify the CSS or JavaScript yourself.\
• -K –> when doing -k, keep a backup of the original file, without links converted.\
• -p –> download everything needed to properly display the files you download. e.g. all style sheets, JS files, images, …\

So as you can see, a lot of work went into making it (I read the entire wget man page) but I think it was worth it. I now keep it in a file so whenever I need to do the same thing again, I have it. And now I have a blog, I thought “why not share?!”

Enjoy. And please tell me if you find it useful.

On Thursday night, I went to a networking event in London called “Chinwag.” The idea of this was to get to know some of the people “in the business” and make some links that later we could form a kind of symbiotic business relationship. It also gives us a better feel for the scope of technology, and we get to talk about what’s hot and whats not in the tech/design/media world.

It was a good event, with around 2,000 attendees. There was free booze (well, for a long while there was…) and free food. I met many interesting people, though I must admit it was Jof who did most of the forging ahead and meeting new people, I opted to follow a short while later and talk specifics.

As I already said - it was a good event. It would have been brilliant if it wasn’t for the poor organisation of the door. We queued to get in for the best part of a hour! They were checking people off against their registered email addresses, which were printed onto paper (yes you heard me - dead trees!), which they had to flick through to check, at around 50 email addresses per sheet. The sheets were in mostly alphabetical order, but it still took a while to look through - I think part of the reason might have been the time taken to actually understand what the visitor was saying their email address was. One pile of paper was email addresses starting A-M and the other N-Z. For some reason the N-Z queue seemed to move a lot later, I wonder if that says something about English email addresses/first names.

We finally got back to Jof’s house in Kent at around 1am, and I slept in the latest that I had in a good while - 9:30am!

It is hailing. In July. Not 2 weeks after the “longest day.” Global warming… That’s what I blame.\

Cake caches database queries. This is good, generally, it speeds things up a lot. But it is not always good. For example, if I want to query the database once a second to look for updates (e.g. for my chat application) the you don’t want the cache. Here’s how to clear it:

1) Create /app/app_model.php, and add in it the following

1
2
3
4
5
6
7

<?php
class AppModel extends Model{
 function \_clearDBCache() {
  $db =& ConnectionManager::getDataSource($this-\>useDbConfig);
  $db-\>\_queryCache = array();
  }
}

Note: you don’t always need a close “?>”

2) Call $modelNameHere->_clearDBCache() whereever you need the DB cache cleared.

That’s it! Have fun!

Whilst I am on the subject of CakePHP tips, why not look into using expects() to make your CakePHP queries more efficient - only request the data you need. Much better than using unbindModel and bindModel all over the place!

I am currently working on making AJAX chat. This has had many issues associated with it, the major one being bandwidth usage. By default the headers sent/received in a basic AJAX request under Firefox with mootools are are around 1kb! That is huge, and will put massive stretches on bandwidth. For this reason, I tried to minimize the headers as best I could, and got them down to around 150bytes each way, total. Read on for what I did…

I used the following options for the client (mootools, but you should be able to get the gist):

Client headers (JavaScript with Mootools):

new XHR({method:‘get’, headers:{‘Accept’:‘’, ‘Accept-Language’:‘’, ‘Accept-Encoding’:‘’, ‘Accept-Charset’:‘’, ‘User-Agent’:‘’ }});

This saved a lot of the unnecessary details being sent (if you require these headers, why not cache them in your $_SESSION variable in PHP?). The biggest of these headers was actually the User-Agent header. Turning this off gave me issues with sessions under CakePHP though. It turns out that Cake uses the User-Agent string to increase the security of Sessions. Well, for now I class the bandwidth saved more important than the security implications, so I added the following lines to /app/webroot/index.php in cake:

Fix sessions (Apache server with CakePHP):

$_SERVER[‘OLD_HTTP_USER_AGENT’] = $_SERVER[‘HTTP_USER_AGENT’];\ $_SERVER[‘HTTP_USER_AGENT’] =“REMOVED”;

Which made Cake “think” that all web browsers were called “REMOVED.” An alternative method could be to introduce a new session purely for usage through the chat interface, but that seemed like too much hassle to me.

I also removed the following headers from the server side, using PHP code:

Server headers (Apache, PHP and CakePHP):

header(‘X-Powered-By:’);\ header(‘Cache-Control:’);\ header(‘Expires:’);\ header(‘Vary:’);\ header(‘P3P:’);\ header(‘Server:’,true);\ header(‘Pragma:’);\

And edited my Apache config to reduce the size of the ‘Server’ header (as apparently I cannot remove it using PHP):

Reducing Apache signature header:

ServerSignature Off\ ServerTokens Prod

The ServerSignature option is more of a security thing - stopping people from knowing exactly what versions of software my server runs.\

This worked well. I then compressed all my XML outputs as far as I could, using one character tags with one character properties, and reducing empty f tags to <f/> wherever possible. This worked fine. But I realised that we were still sending 0.5kb/second/client. It was at this point that I thought

Why not get the server to wait for new content?

So by using the php usleep() function I had the data fetching function poll the database every $delay milliseconds. If there were changes, it would output the changes and return. If there were no changes after 30 seconds, it would exit anyway with an “empty” message: <f/>. If the client disconnected it would discontinue.

This seemed to be working great, except new posts that I wrote to the chat were not appearing until the aforementioned 30seconds had expired. It took me a while to track down the reason, but it is due to CakePHP and PHP serializing session access so as to prevent race conditions. Well my chat function just cached all the session data (it did not need to edit it) and then ran

Releasing the session (so other connections work):

session_write_close();

in order to release the session, so other connections could continue. Now I am aware of this issue, I think I will be building session_write_close() into many more of my functions!

But now I have a chat solution with variable delays (currently set to 0.5s) that allows you to have AJAX chat without requiring huge amounts of bandwidth. I can even increase the 30 seconds by changing PHP’s max_execution_time.

What if the server suddenly comes under attack?

I didn’t mention: I also (optionally) send out the delay number to the scripts, so if I want to I can change the delay from 0.5s to 5s across all currently connected clients within 30 seconds, without any of them having to reload, or even realise the change! This means that if bandwidth is suddenly a problem, I can ask the web browsers to reduce their pounding of the servers, and they will almost instantly. Pretty neat, eh?

I am beginning to think that AJAX is a viable solution to chat, so long as you think about it enough.

Keywords for Google:

Increasing AJAX efficiency, making efficient AJAX code, make AJAX more efficient, reduce header count, decrease number of headers, remove headers, reduce AJAX bandwidth\

Ideas for the future:

Currently the chat is stored in tables in a MySQL database, but I think we will be using memcached to improve efficiency some time soon. We will also probably remove the overhead of CakePHP as a framework just for the processor intensive chatroom feed script.

I like IBM. I have even visited them a couple of times if you look at my homepage. Well, whilst reading LifeHacker, I saw a link to the following article on the IBM website: Whistle while you work to run commands on your computer. Well, the title sounded like fun so I read on a little. They say it may take around an hour to set up, so perhaps I will do it on a weekend. What made me smile was the following excerpt:

“Note that you will have to be running this program in a relatively noise-free environment, so plug in your headphones and make sure your CD drive is spun down. If your laptop’s battery is on fire, try unplugging the smoke detector before running this program.”

I might try and stick more subtle humour like this into the stuff I make for Brain Bakery.

I wrote two facebook applications yesterday. Yes, you heard me - two, yesterday. Admittedly I did work for 12 hours almost solid yesterday.

One is an application that lists new posts on blog feeds that you are interested in, and the other allows you to SMS your facebook page to tell your mates “I’m down the pub, why don’t you join me?” They are both, obviously, in their infancy, and the SMS one took me another days coding a few weeks ago (by day, I mean 10-12 hours…) to get SMSs sent to my phone to hook up to my computer well… but it works very well now, tested it last night!

So… watch this space. When they get a little more professional I will consider releasing them. Before then, I need a new phone that I can dedicate to the purpose, or a Nokia usb data cable for my 3310.