After a morning of research on the subject, I was pointed at mranney’s essay on the subject by `3rdEden on irc.freenode.net/socket.io. It turns out that the issue is due to V8 (the JavaScript engine used by Node.JS) using the UCS encoding internally rather than the more modern UTF-16. Emoji require 17 bits (their code values are larger than 65,535), which is more than UCS can give (UCS uses exactly 2 bytes (16 bits) to represent every character, so it only supports values between 0 and 65,535; whereas UTF-8 and UTF-16 use a variable number of bytes).

I’ve confirmed with WireShark that Safari is sending valid UTF-8 (11110000 10011111 10011000 10010011 or hex: f0 9f 98 93, which gives the Unicode codepoint U+1F613 😓). So I know it’s Node’s issue receiving and processing it.

This is probably one of the reasons that Google Chrome doesn’t support Emoji - check out this page in Chrome, then view it in Safari to see what you’re missing! (Chrome uses the V8 engine.)

Solution

You could base64-encode your payload, or simply escape()/unescape() it. Or you could trim anything outside of UCS. Or you could do a custom encode/unencode such as this one. I’m not really happy with any of these, so I’m still looking for a solution.

Update

In the end I implemented this (see below) on the client side, and left the data encoded server side. All clients are responsible for encode()ing characters when sending, and decode()ing upon receiving (whether that be via websockets or HTTP). It seems to work quite well and doesn’t massively inflate the content-size for ASCII and possibly more[citation needed], so I’m relatively happy with it. Besides it’s midnight and I need to get some sleep.

1
2

encode = (s) -> unescape encodeURIComponent s
decode = (s) -> decodeURIComponent escape s

1
2
3
4
5
6
7
8
9

var decode, encode;

encode = function(s) {
  return unescape(encodeURIComponent(s));
};

decode = function(s) {
  return decodeURIComponent(escape(s));
};

Commands

Vim commands are formed from a combination of verbs and targets. The targets could be objects (words, sentences, paragraphs, lines, the contents of parentheses) or movements (jump to end of word, jump to end of paragraph, jump forward until the letter ‘e’, etc). Forming objects generally involves the use of a modifier. You can also add a count to perform the action count times.

Verbs

Here’s some example verbs (don’t try and learn them all at once!):

• i - insert - enter insert mode
• a - append - enter insert mode after the carat
• I - enter insert mode at the beginning of the line
• A - append to line (enter insert mode at the end of the line)
• o - open a line after the current one and enter insert mode
• O - open a line before the current one and enter insert mode
• d[target] - delete (remove from the document and put in buffer)
• y[target] - yank (copy)
• p - paste the buffer after the cursor
• P - paste the buffer before the cursor
• c[target] - change (delete and then enter insert mode)
• r[char] - replace the character under the cursor with [char]
• x - delete the character under the cursor
• u - undo the last command
• Ctrl-R - redo the last undo (sidenote: in vim undo/redo forms a tree, changes aren’t lost)
• / - enter regex search mode
• n - find the next instance of the search term
• N - find the previous instance of the search term
• . - repeat last change (extremely powerful)

Nouns/Movements

Nouns or movements are commands for moving within the document or representing an area within a document. Common movements are:

• h, j, k, l - equivalent to the arrow keys left, down, up, right - you should aim to use these rather than the arrow keys, but don’t worry too much yet!
• 0 - move to the very beginning of the current line
• ^ - move to the first non-whitespace character on the line
• $ - move to the end of the line
• w, b - move to the next/previous word
• W, B - as w/b only Words are bigger
• ), ( - move to the next/previous sentence
• }, { - move to the next/previous paragraph
• f/F[char] - find the next/previous instance of [char] in the current line
• t/T[char] - until - find the character before the next/after the previous instance of [char]
• /[regexp] - like t but instead of finding a character it finds a regexp
• % - jump to the matching parenthesis (vim understands nested parenthesis)
• _ - move to the current line (useful for making commands line-based)

Trying it out

Try some of these movements out now - you can start to imagine how much faster document navigation will be once you’ve mastered them.

Once you’ve got a feel for the movements, try combining them with verbs. Don’t forget that if you do something wrong (or don’t understand what happened) you can simply press u to undo, and try it again. For example:

• cw - “change” until the end of the current “word”
• d} - delete until the next paragraph (note that this is line-based delete, so it will delete the whole of the current line too)
• d_ - delete the current line (this can also be entered as dd for brevity)
• y_ - yank the current line (this can also be entered as yy for brevity)
• p/P - paste the previously yanked (or deleted) text. If the yank (or delete) was character based then it will paste after/before the cursor, otherwise it will paste on the next/previous line.
• 27x - delete the next 27 characters

Counts

You can generally precede a verb or a movement with a count, which will perform that action that many times. For example:

• 3w move to the third word from here
• 3j move the cursor three lines down
• 3} move to the 3rd paragraph from here
• 3i will write whatever you insert 3 times. (The duplication will not occur until you exit insert mode, e.g. with <Esc>.)
• c3w will change the next 3 words (i.e. it will delete 3 words and then enter insert mode).
• 3dd will delete the current line and the two below it
• 3p will paste the contents of the buffer three times

Modifiers

Modifiers can be used between a verb and a noun to modify its meaning. For example, let’s look at i[noun] (inner/inside):

• ciw - change the whole word the cursor is on, even if the cursor is in the middle of the word
• ci) - change the contents of the current pair of brackets (understands bracket pairing)
• ci" - change the contents of the current pair of doublequotes (understands quote escaping)

another modifier is a - around - like i but includes the container.

Next time

In my next post, I’ll talk about some of the ways I’ve used these basic commands to speed up my editing.

Installing Vim

I recommend you start with a graphical Vim app so that you can use your mouse to click around. Ultimately you will rarely if ever use your mouse (moving your had that far will seem really inefficient) but to start with it can significantly alleviate the frustration of being slow with the movement commands.

Mac

If you use Homebrew then you can simply issue

1

$ brew install macvim

And voila, you’re done. Otherwise take a look at the MacVim instructions.

Linux

You’re going to want gVim, probably. Install it through your usual package manager.

Windows

You’re on your own.

Themes and Plugins

To get you off to a running start, I highly recommend using the Janus vim distribution for all your plugin needs. You can install more plugins later, but these will cover all the basics you’re likely to need including syntax highlighting for a huge number of languages (even stuff like Markdown files) and a vast array of colour schemes (I like ir_black).

If security is not a concern for you then you can run their installer via the following command:

1

$ curl -Lo- http://bit.ly/janus-bootstrap | bash

Vimtutor

The first thing you should do is to try out vimtutor - just run it from the command line assuming it was installed with Vim. Spend 25 minutes going through everything to get a feel for Vim. Don’t worry if it doesn’t all click into place at first - you can use Vim just like a plain text editor at first, and slowly improve your skill over time. If you’re a programmer and if you haven’t used a modal editor before then I assure you it will be worth the time spent learning it.

Absolute basics

Initially you only need to worry about two of vims modes: normal mode and insert mode.

Normal mode

This is the mode that vim opens in. Here you can type commands. Typing :w<Enter> saves the current document. Typing i puts you into insert mode.

Insert mode

Insert mode is where you enter text. Initially you can just treat it like a plain text editor - write stuff and use the arrow keys to move around. When you want to save, press <Esc> to exit back to normal mode and then type :w<Enter> to save. You can then press i to get back into insert mode.

When you are in insert mode you will see -- INSERT -- displayed at the bottom of the screen. Remember: to exit insert mode just press <Esc>.

Moving on

Take a while to master the above before moving on. It doesn’t matter how long it takes you - the key is to avoid the frustration of having deminished productivity by trying to do too much at once.

My next post will focus on the fundamental commands in Vim, like those you touched upon in vimtutor.

I want to take this opportunity to thank my wife for everything. Her constant loving support helps me in every aspect of my life from fatherhood to business to helping me deal with my health problems. I honestly don’t know what I would do without her.

❤ Thank You, Jem ❤

You’re the most wonderful woman I know - beautiful, funny, clever, loving, honest, understanding and a fantastic wife and mother. Every day I feel grateful to have you in my life and I truly hope I make you feel loved and appreciated. Together we make a brilliant team - the future never scares me, because I know whatever happens you’ll be there, by my side.

To me, you are perfect; I love you now, always and forever.

John Hrvatin from the Internet Explorer team:

The transition to a plug-in free Web is happening today. Any site that uses plug-ins needs to understand what their customers experience when browsing plug-in free. Lots of Web browsing today happens on devices that simply don’t support plug-ins. Even browsers that do support plug-ins offer many ways to run plug-in free.

Metro style IE runs plug-in free to improve battery life as well as security, reliability, and privacy for consumers.

How long until Google joins the party?

Frequently friends and acquaintances will talk to me about iOS/Android (being an iOS developer) and one of the things they’ll very often say is “yeah, but iOS can’t run Flash,” as if that’s a bad thing! Banning Flash from the platform is one of the things that makes me really glad I own iOS devices and that they’ve such a large market share. Many websites that use Flash are being forced to redevelop into open and standards-compliant HTML & JS so that they can capture the mobile audience.

Flash is the most common browser plugin, and I loathe it. I block it on all my computers and only use it where I absolutely have to. If a shopping website forces me to use Flash to shop there then I simply don’t shop there. And with HTML5 Video/Audio and the speed of JavaScript in todays modern browsers (even the latest Internet Explorers!) there’s less and less justification for websites to use Flash (or worse: Silverlight! I’m looking at you, LoveFilm).

I’m not the only person who hates Flash, just look at all the plugins to disable it!

Why Plugins Suck

There’s many reasons, but mostly it comes down to non-compatibility, performance, security and user experience.

Compatibility

For a long time there was no Flash player for Linux. Then there was no Flash player for 64-bit operating systems. There’s no Flash for iOS. My Blu-ray player can’t run Flash. Apparently my Tivo does run Flash, but it’s god-awful.

And because Flash runs in a different rendering pipeline it doesn’t integrate well with web browsers - especially when it comes to transparency. (See “User Experience”)

Performance

A lot of devices are easily capable of rendering HTML/CSS/JS but Flash pushes them too hard and the experience is ugly and slow. And Flash seems to be used for a whole host of things that HTML/CSS/JS have been capable of for years and years across all major browsers - one thing that comes to mind is all the advertising you get on certain websites.

Security

Bugs in plugins expose your browser to a lot of potential security risks - cross site scripting (XSS) attacks, information disclosure, even sharing your webcam feed without your permission (October 2011)! Over the years a lot of these issues have been patched, but it should be the job of the web browser to allow/deny access to various resources, having two interfaces increases the attack surface - it’s risky and confusing.

User Experience

When hover to expand adverts first came out they used to actually block the viewing of the content on some websites in some browsers - so I simply stopped visiting those website. What’s the point if you can’t actually read the content?!

Once you’ve got 10 Flash adverts on a web-page the scroll performance drops significantly, even on a dual core computer. There’s also the increased render time of the webpage as a whole, and the annoying (and unnecessary!) loading indicators.

When you close a browser tab, plugins on that page playing audio/etc often won’t close until a couple of seconds later, which can be very irritating. And don’t get me started on websites that suddenly start talking to you without you asking them to! (Though this is possible with HTML5 also…)

Acceptable Use Cases

There used to be lots of use-cases for which I found the use of Flash acceptable, but many of these are being diminished due to HTML5 adding extra capabilities to the browser:

• 3D browser games (soon to be invalid due to WebGL and similar technologies)
• streaming video/audio (now supported by HTML5 audio and video)
• video conferencing (there’s many workarounds for this - W3 are attempting to create a Media Capture API)
• real-time socket communications (WebSockets allow real time communications by extending HTTP)

Progress

Apple, Microsoft I applaud your efforts to rid the internet of these plugins (despite the existence of QuickTime, Silverlight and the like).

Google, I’m disappointed with you integrating Flash so deeply into Chrome, but Chrome’s an awesome browser for development so I shall use it anyway. Good work on making YouTube work with HTML5 though :)

HTML5 guys, you all rock.

This year I’m making an effort to do the things I’ve always meant to do as a programmer; and playing with lots of new (to me) tech in the process! I intend to cover a few of these topics in the coming weeks/months, but here’s a summary:

Vim

I’ve started using Vim as my main editor (or more specifically MacVim with the Janus vim distribution) and I’m loving it. It’s a relatively steep learning curve, but you don’t have to jump in at the deep end - just start with i to enter insert mode, use it like a normal editor, and then to save press <Esc> followed by :wq<Enter> to save and exit (drop the q if you don’t want to exit). Slowly start learning new things and forcing yourself to start using normal mode (where you type commands like :wq) and as you use more, you’ll want to use more. Once you’ve mastered the whole count-verb-modifier-noun grammar of vim you’ll be using it for everything and wondering how you ever managed without it!

Node.js

Okay, so I’ve been playing with this for a while, but I’m using it for more and more stuff now - pretty much anything that needs scripting I now do in JavaScript, or more specifically…

CoffeeScript

… CoffeeScript, which is bloody brilliant - especially if you have a strong understanding of how the underlying JavaScript functions. This isn’t new either, but I’m now using it for ALL my JavaScript programming

Underscore.js

Underscore.js is a light library of helpful JavaScript functions for working with objects, arrays, collections and other such things in a functional programming way without extending the native objects. jQuery/Mootools use $ for working with DOM nodes, Underscore uses _ for dealing with everything else.

Backbone.js

Backbone.js extends Underscore.js and is a great way of keeping the code of your RIAs organised. It’s very flexible and can fit to many different programming methodologies, but I use it in a M-VC (Model, ViewController) manner not dissimilar to iOS. Event are emitted from models when they’re added, changed or deleted; hooking into these events allows your app to always display consistent information without the need for lots of hard-coded callbacks.

jQuery

It seems everybody uses jQuery, and has done since its inception. I refused to jump on the band-wagon and I’m glad I did - I coded everything in JavaScript from scratch using native methods and help from articles across the web (especially sites such as QuirksMode) and now have a very good understanding of how different parts of a modern web browser piece together and the faster/slower ways to go about doing things.

However, jQuery is a lot faster than it used to be, as are modern web browsers, so my concerns are much less valid. It reduces programming time significantly and adds flexibility so I now use it (or Zepto.js) in a lot of my projects and I really like it - it doesn’t take long to learn and master, and it improves the speed of Backbone development considerably.

Git

Git, as you should know, is a brilliant version control software. I’ve been using SVN for years but have had the itch to switch for ages, so with some new projects that I’ve been working on (using the above technologies) I decided to switch to using git where possible. It’s brilliant - lightening fast and so flexible. The lack of the old /trunk/, /branches/, /tags/ layout makes me happy too - trying to convince employees to use svn switch was very hard, but with git they’ve got no choice but to do it the right way! I’m using gitolite for our internal (private) hosting, and …

Open Sourcing

… I’ve started Open Sourcing some stuff I’m working on to GitHub. First is my ec2-prune-snapshots script for automatically deleting sensibly old EBS snapshots on AWS, but I intend to release much more in the coming year, and possibly even retro-actively releasing some older software, such as MythPyWii which currently sits on Google Code.

What Are You Using?

If you’re using some cool technology for web/mobile application development that you think I might enjoy experimenting with then please let me know in the cooments!

One method of solving this issue is to chain the asynchronous calls, however this means that they’re run one after the other (serially) - and thus it will take longer to complete them. A better way would be to run them in parallel and have something track their completion. This is exactly what my very simple AsyncBatch class does:

1
2
3
4
5
6
7
8
9
10
11

class AsyncBatch extends EventEmitter
  constructor: ->
    @_complete = {}
    @_scheduled = {}

  wrap: (name,cb) ->
    @_scheduled[name] = true
    return =>
      @_complete[name] = cb.apply @, arguments
      if Object.keys(@_complete).length == Object.keys(@_scheduled).length
        @emit 'done', @_complete

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

var AsyncBatch,
  __hasProp = Object.prototype.hasOwnProperty,
  __extends = function(child, parent) { for (var key in parent) { if (__hasProp.call(parent, key)) child[key] = parent[key]; } function ctor() { this.constructor = child; } ctor.prototype = parent.prototype; child.prototype = new ctor; child.__super__ = parent.prototype; return child; };

AsyncBatch = (function(_super) {

  __extends(AsyncBatch, _super);

  function AsyncBatch() {
    this._complete = {};
    this._scheduled = {};
  }

  AsyncBatch.prototype.wrap = function(name, cb) {
    var _this = this;
    this._scheduled[name] = true;
    return function() {
      _this._complete[name] = cb.apply(_this, arguments);
      if (Object.keys(_this._complete).length === Object.keys(_this._scheduled).length) {
        return _this.emit('done', _this._complete);
      }
    };
  };

  return AsyncBatch;

})(EventEmitter);

To use AsyncBatch, just do the following

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

###
Create a new AsyncBatch instance
###
batch = new AsyncBatch

###
Wrap your callbacks with batch.wrap and a name
The name is used to store the result returned by your callback
(Don't forget to return the result you're interested in!)
###
delay 50, batch.wrap 'timer', ->
  return "Timer complete"

###
Add a completion handler that accepts `results` as a parameter.
  `results` is a JS object where the keys are the callback names 
  from above and the values are the return values of the
  callbacks.
###
batch.on 'done', (results) ->
  console.log "Batch complete, timer result: #{results.timer}"

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

/*
Create a new AsyncBatch instance
*/
var batch;

batch = new AsyncBatch;

/*
Wrap your callbacks with batch.wrap and a name
The name is used to store the result returned by your callback
(Don't forget to return the result you're interested in!)
*/

delay(50, batch.wrap('timer', function() {
  return "Timer complete";
}));

/*
Add a completion handler that accepts `results` as a parameter.
  `results` is a JS object where the keys are the callback names 
  from above and the values are the return values of the
  callbacks.
*/

batch.on('done', function(results) {
  return console.log("Batch complete, timer result: " + results.timer);
});

NOTE: If your asynchronous method accepts both a success and failure callback then simply wrap both individually but ensure you use the same name for both.

NOTE: Other than the case in the previous NOTE, all callbacks should have different names.

A full example (including a stub EventEmitter implementation) follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

#EventEmitter = require('events').EventEmitter

if !EventEmitter?
  class EventEmitter
    on: (eventName, cb) ->
      @_eventCallbacks = {} unless @_eventCallbacks
      @_eventCallbacks[eventName] = [] unless @_eventCallbacks[eventName]
      @_eventCallbacks[eventName].push cb
    emit: (eventName, args...) ->
      if @_eventCallbacks && @_eventCallbacks[eventName]
        cb.apply @, args for cb in @_eventCallbacks[eventName]

class AsyncBatch extends EventEmitter
  constructor: ->
    @_complete = {}
    @_scheduled = {}

  wrap: (name,cb) ->
    @_scheduled[name] = true
    return =>
      @_complete[name] = cb.apply @, arguments
      if Object.keys(@_complete).length == Object.keys(@_scheduled).length
        @emit 'done', @_complete


batch = new AsyncBatch

##########################
# Task 1
http = require 'http'

options =
  host: 'www.example.com'
  port: 80
  path: '/'
  method: 'GET'

req = http.request options, batch.wrap 'status', (res) ->
  console.log 'STATUS: ' + res.statusCode
  console.log 'HEADERS: ' + JSON.stringify(res.headers)
  return res.statusCode

req.on 'error', batch.wrap 'status', (e) ->
  console.log 'problem with request: ' + e.message
  return null

req.end()

#########################
# Task 2
fs = require 'fs'

fs.readFile '/etc/hosts', 'utf8', batch.wrap 'hosts', (err, data) ->
  if err
    return null
  console.log data
  tmp = data.split "\n"
  return tmp[Math.floor(Math.random()*tmp.length)]

########################
# Task 3
delay = (ms, cb) -> setTimeout cb, ms

delay Math.random()*100, batch.wrap 'timer', ->
  console.log "Timer done"
  return true

#########################
# Completion:
batch.on 'done', (results) ->
  console.log "\n\n\nThe fetch, read and timer have completed."
  if results.status != null
    console.log "Fetch succeeded, status code: #{results.status}"
  else
    console.error "Fetch failed!"
  if results.hosts != null
    console.log "Random entry from /etc/hosts: "+results.hosts
  else
    console.error "Couldn't read /etc/hosts!"

batch.on 'done', (results) ->
  console.log "\nSecond completion callback, raw results:"
  console.log results

The following JavaScript creates a new closure inside each iteration of a loop to store the value of the loop variable i into the closure’s local variable j so that we don’t just console.log(10) 10 times. Though this example is trivial it represents a common method for solving closure related issues in asynchronous code.

1
2
3
4
5
6
7
8

for (var i = 0; i < 10; i++) {
  (function(){
    var j = i;
    delay(0, function(){
      console.log(j);
    });
  })();
}

The following CoffeeScript does exactly the same thing, you can see the translation line by line from the JavaScript. However once you’ve learnt the syntax you can much more clearly see what is going on with the CoffeeScript than in the verbose and parenthesis-heavy JavaScript above.

1
2
3
4
5

for i in [0..10]
  do ->
    j = i
    delay 0, ->
      console.log j

1
2
3
4
5
6
7
8
9
10
11
12

var i, _fn;

_fn = function() {
  var j;
  j = i;
  return delay(0, function() {
    return console.log(j);
  });
};
for (i = 0; i <= 10; i++) {
  _fn();
}

As a bonus, when CoffeeScript compiles to JavaScript it automatically takes the anonymous function from line 2 and defines it outside the loop so that it need not be interpretted/defined 10 times - it does this optimization for us. (Though I suspect most modern JavaScript engines do this automatically anyway.) To view the compiled JavaScript press the View JavaScript link at the top left of the code snippet.

Note the delay function is just a reordered setTimeout:

1

delay = (ms, cb) -> setTimeout cb, ms

1
2
3
4
5

var delay;

delay = function(ms, cb) {
  return setTimeout(cb, ms);
};

Strict guidelines will also be placed on internet providers which offer so-called unlimited packages, where users can download to their heart’s content. If a user incurs an “additional charge or suspension of service as a consequence of exceeding a usage threshold,” the ISP cannot use the term “unlimited”.

Finally! I hate it when providers are allowed to say ‘Unlimited’ and then place a limit on it (generally via a Fair Usage Policy). What a load of rubbish! I’m a heavy bandwidth user so it makes it very hard for me to find out what the limit is and then choose the relevant provider. Fortunately Virgin’s XXL broadband truly is unlimited - at least for now!

Scope issue

The developer of Octopress has kindly protected our window object from being polluted with various unnecessary variables by wrapping the whole disqus include in an anonymous function.

However, the Disqus script actually injects another <script> element into the <head> (or <body>) of the page. This injected script, like all other scripts on the page, inherits the global (window) scope, and not the scope of this anonymous function. Thus it will not have access to the disqus variables such as disqus_shortname defined within the anonymous function. Without these variables, disqus loads such domains as undefined.disqus.com which is not very helpful!

Solution

The fix is simple - open up source/_includes/disqus.html and move the beginning of the anonymous function ((function () {) so that it does not contain the var disqus_* variable definitions. We also need to insert a couple of semi-colons that have been accidentally omitted. Here’s the resulting file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

{% comment %} Load script if disquss comments are enabled and `page.comments` is either empty (index) or set to true {% endcomment %}
{% if site.disqus_short_name and page.comments != false %}
<script type="text/javascript">
      var disqus_shortname = '{{ site.disqus_short_name }}';
      {% if page.comments == true %}
        {% comment %} `page.comments` can be only be set to true on pages/posts, so we embed the comments here. {% endcomment %}
        // var disqus_developer = 1;
        var disqus_identifier = '{{ site.url }}{{ page.url }}';
        var disqus_url = '{{ site.url }}{{ page.url }}';
        var disqus_script = 'embed.js';
      {% else %}
        {% comment %} As `page.comments` is empty, we must be on the index page. {% endcomment %}
        var disqus_script = 'count.js';
      {% endif %}

    (function () {
      var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
      dsq.src = 'http://' + disqus_shortname + '.disqus.com/' + disqus_script;
      (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
    }());
</script>
{% endif %}

One more thing

If you want your links to be absolute, then you should set root to be your root URL in _config.yml - e.g. root: http://www.benjiegillam.com/ rather than root: /. This may or may not be required for Disqus to function properly.

EDIT: @Octopress has confirmed that root must not contain the scheme and host.

Bonus

Now that I’ve implemented the above, my comments and comment counts successfully display on my benjiegillam.dev site (hosted by Pow)!

Or use this solution with sed:
sed ':a;N;$!ba;s/\n/ /g'
This will read the whole file in a loop, then replaces the newline(s) with a space.

Update: explanation.

1. create a register via :a
2. append the current and next line to the register via N
3. if we are before the last line, branch to the created register $!ba ($! means not to do it on the last line (as there should be one final newline)).
4. finally the substitution replaces every newline with a space on the pattern space (which is the contents of the a register = the whole file.

The solution I found was to split the command up into explicit separate commands:

1

sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/ /g'

$ rake generate
(in /Users/benjiegillam/Documents/Blog/octopress)
## Generating Site with Jekyll
Configuration from /Users/benjiegillam/Documents/Blog/octopress/_config.yml
unchanged sass/screen.scss
Building site: source -> public
/Users/benjiegillam/Documents/Blog/octopress/plugins/raw.rb:11:in `gsub': invalid byte sequence in UTF-8 (ArgumentError)
    from /Users/benjiegillam/Documents/Blog/octopress/plugins/raw.rb:11:in `unwrap'
    [...]

However, converting the file was perfectly valid UTF-8 as confirmed by an iconv -c conversion followed by a diff -u.

After a quick bit of hacking in the octopress/plugins/raw.rb file to spit out the content that was being converted, I found the file at fault. After some iteration I got to the root of the issue - Octopress’ default markdown parser, rdiscount, REALLY doesn’t like UTF-8 characters in URLs. I’ve built a test here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# encoding: utf-8
require 'rdiscount'

print "Test 1: "
markdown = RDiscount.new("http://commons.wikipedia.org/wiki/Image:Dara_Ó_Briain.jpg")
if markdown.to_html == "<p>http://commons.wikipedia.org/wiki/Image:Dara<em>Ó</em>Briain.jpg</p>\n"
  puts "PASS"
else
  print "FAIL: "
  puts markdown.to_html
end
# Correct: <p>http://commons.wikipedia.org/wiki/Image:Dara<em>Ó</em>Briain.jpg</p>

print "Test 2: "
markdown = RDiscount.new("[Wikipedia](http://commons.wikipedia.org/wiki/Image:Dara_Ó_Briain.jpg)")
if markdown.to_html == "<p><a href=\"http://commons.wikipedia.org/wiki/Image:Dara_%C3%93_Briain.jpg\">Wikipedia</a></p>\n"
  puts "PASS"
else
  print "FAIL: "
  puts markdown.to_html
end
# Incorrect: <p><a href="http://commons.wikipedia.org/wiki/Image:Dara_?%93_Briain.jpg">Wikipedia</a></p>
# Note the invalid character (represented by ?)
# Expected: <p><a href="http://commons.wikipedia.org/wiki/Image:Dara_%C3%93_Briain.jpg">Wikipedia</a></p>

It was converting Dara_Ó_Briain.jpg to Dara_?%93_Briain.jpg, where the ? is an invalid UTF-8 character. (Should be Dara_%C3%93_Briain.jpg)

Solution?

Annoyingly pandoc (a tool employed by exitwp) seems to be converting the link from Dara_%C3%93_Briain.jpg to Dara_Ó_Briain.jpg in the markdown file, which is then breaking when it is rdiscounted. As it only affected 2 characters in my entire blog history I’ve not bothered with an automated fix - I just manually re-encoded the characters. I’ve commented on this issue with pandoc so that hopefully they will fix it.

Crossed out code blocks

The markdown produced by pandoc (an exitwp dependency) uses ~~~~ in places to denote a code block. However, rdiscount (Octopress’ markdown parser) recognises this instead as strike-through. A simple bash command fixes this (though it does lose some of the formatting details).

1
2
3
4
5

for I in *; do sed -i '' -e 's/^~~~~ {\(.*\)}$/\
<!-- \1 -->\
```/g' $I; done
for I in *; do sed -i '' -e 's/^~~~~$/\
```/g' $I; done

Line breaks in links

For some reason exitwp added some linebreaks into the markup for the links. And for some other reason, octopress’ style sheet (screen.css) tells <a> tags that their white-space should be pre-wrap, these two issues combine to give you broken links.

This next command should fix these newline issues. (It finds a [ with no matching ] on the same line and then removes the newline at the end of said line, being careful not to have two newlines next to one another.)

1
2

for I in source/*.markdown source/*/*.markdown; do sed -E -i '' -e ':a' -e 'N' -e '$!ba' -e 's/\n?(\[[^]]*)\n([\t ]*)/\
\2\1 /g' "$I"; done

NOTE: You may need to run the above more than once.

Turns out that the above broke some image embeds - no worries - here’s the solution:

1

for I in *; do sed -i '' -e ':a' -e 'N' -e '$!ba' -e 's/\!\n\[/\![/g' $I; done;

Ordered/Unordered Lists

exitwp does not insert newlines before lists, and rdiscount does not always recognise it as a list unless it has a blank newline above it. Simply find the ‘1.’ entry and put a blank newline above it.

Tildes

rdiscount doesn’t do strikethrough with a single tilde, so it does not need escaping. However exitwp escapes them, so text like ~/Documents/ will become \~/Documents/

Random quotes

rdiscount and exitwp seem to disagree on indentation - causing rdiscount to render indentation (sometimes) as a quote rather than continuation of the text.

URLs with brackets in

exitwp or pandoc do not correctly escape (with a backslash) brackets within links, so captions/etc may be broken. Simply add in the backslashes.

/blog/archives/

I was looking for a line in _config.yml to configure the archives. Turns out it’s actually the file source/blog/archives/index.html that needs to move - I moved this to source/archives/index.html

1
2

mv source/blog/archives source/
rmdir source/blog/

And then updated the navigation source/_includes/custom/navigation.html

Hosting on S3

I followed these instructions from Ian Wootten (which reference these instructions by Jerome Bernard) - thanks!

Disclaimer

This code in this post worked for me, but it is quite dumb - you will need to check the changes it makes are good (this is where git diff comes in handy - make sure to commit before running these commands!)

1
2
3
4
5

if session and not session.prototype?.reload
    req =
        sessionStore: sessionStore
        sessionID: data.sessionID
    session = new express.session.Session req, session

1
2
3
4
5
6
7
8
9

var req, session, _ref;

if (session && !((_ref = session.prototype) != null ? _ref.reload : void 0)) {
  req = {
    sessionStore: sessionStore,
    sessionID: data.sessionID
  };
  session = new express.session.Session(req, session);
}

immediately inside the sessionStore.get callback.

1
2
3
4
5
6
7
8
9
10
11
12

# Alias for setTimeout that reorders parameters to look neater in CoffeeScript
delay = (ms, cb) -> setTimeout cb, ms
# Handle SIGUSR2, but only once
process.once 'SIGUSR2', ->
  console.log 'Doing shutdown tasks...'
  delay 5000, ->
    console.log 'All done, exiting'
    # Kill ourself with the SIGUSR2 signal again
    process.kill process.pid, 'SIGUSR2'
# This next line just to prevent the app exiting in case you want to use this as a demo
delay 99999999, ->
  console.log 'App exiting naturally'

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

var delay;

delay = function(ms, cb) {
  return setTimeout(cb, ms);
};

process.once('SIGUSR2', function() {
  console.log('Doing shutdown tasks...');
  return delay(5000, function() {
    console.log('All done, exiting');
    return process.kill(process.pid, 'SIGUSR2');
  });
});

delay(99999999, function() {
  return console.log('App exiting naturally');
});

By the way, to install nodemon globally: sudo npm install -g nodemon

Original post follows:

(Mac OS X only, works on Lion, Xcode 4.1, Skype 5.3)

Here are the steps to starting work on interfacing a Cocoa app with Skype:

1. Don’t try and get the framework from the documentation (what the hell?! Invalid link? PowerPC issues? Come on Skype!)

2. Copy Skype.framework into your app’s folder from within Skype (find the Skype app, control-click and Show Package Contents)

3. Drag your copy of Skype.framework into the Frameworks folder of your Xcode project.

4. In Project -> Target -> Build Phases:

   1. Ensure Link Binary with Libraries contains Skype.framework
   2. Add a new build phase (Editor -> Add Build Phase -> New Copy Files Build Phase), set the destination to Frameworks, the Subpath blank, and drag Skype.framework here from your Frameworks group.

5. In the relevant source file add

1

#include <Skype/Skype.h>

1. Code your app. You might want to start with the following:

1
2
3

if ([SkypeAPI isSkypeRunning]) {
  NSLog(@"Skype is running.");
}

Hopefully that’ll get you started a bit faster than I did! API documentation can be found here: https://developer.skype.com/resources/public_api_ref.zip

So you can’t use MigrationAssistant because you already have an account (Alice) on the new machine (New) and don’t have the disk space on the old one (Old) to turn FV off? Fear not! Let’s assume the username you want to move is steve:

1. As Alice, create (admin) account on New with same username, steve, and same password, and enable FileVault.
2. Replace /Users/steve/steve.sparsebundle on New with /Users/.steve/steve.sparsebundle from Old. (you could use scp, rsync, an external hard drive, or any other method to transfer these files) (I did this with steve logged in, but you can do it logged out though the path may be different)
3. As Alice, fix the permissions on New: chown -R steve /Users/steve
4. Log in as steve on New (unlocks the sparsebundle and mounts it at /Users/steve)
5. Fix permissions again: either as Alice or steve, sudo chown -R steve /Users/steve
6. Log out an in again Voila!

If this helps you, please drop me a comment :)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

// Root class
function MyClass() {
  return this;
}

// Create first subclass
function MyChildClass1() {
  return this;
}
MyChildClass1.prototype = new MyClass();
MyChildClass1.prototype.parent = MyClass.prototype;
MyChildClass1.prototype.constructor = MyChildClass1;

// Create second subclass
function MyChildClass2() {
  return this;
}
MyChildClass2.prototype = new MyClass();
MyChildClass2.prototype.parent = MyClass.prototype;
MyChildClass2.prototype.constructor = MyChildClass2;

// Create an instance
var instance = new MyChildClass2();

Shorter way (gives wrong output on console.log()):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

// Automation function
function NS_Extend(parentClass,childClassName) {
  var childClass = function(){return this;}
  childClass.prototype = new parentClass();
  childClass.prototype.parent = parentClass.prototype;
  childClass.prototype.constructor = childClass;
  window[childClassName] = childClass;
}

// Root Class
function MyClass() {
  return this;
}

// First subclass
NS_Extend(MyClass,'MyChildClass1');

// Second subclass
NS_Extend(MyClass,'MyChildClass2');

// Create an instance
var instance = new MyChildClass2();

Simple, but when I console.log(instance) it gives me NS_Extend.childClass as the name of the class, rather than MyChildClass2 as was done previously. It makes no difference to the functionality of the app, but it makes my debugging a bit harder. And I really don’t want to have to keep defining function ClassName(){return this;} everywhere - waste of time typing all that repeated code! So I figured I’d define the function through eval(). However that didn’t work, until I realised that eval() is scoped to the current context. So to define a window level function we’d need to change the scope to the window object. Easy enough with JavaScript’s .call() method. The solution:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

function MyClass() {
  return this;
}
function NS_Extend(parentClass,childClassName) {
  var js = 'function '+childClassName+'(){return this;}';
  eval.call(window,js);
  var childClass = window[childClassName];
  childClass.prototype = new parentClass();
  childClass.prototype.parent = parentClass.prototype;
  childClass.prototype.constructor = childClass;
}
NS_Extend(MyClass,'MyChildClass1');
NS_Extend(MyClass,'MyChildClass2');
var instance = new MyChildClass2();

Now when I console.log(instance) it tells me MyChildClass2 - WIN!

[I use Chrome under Linux, by the way.]

Seems like a simple problem, there must be a simple solution? Here are the existing options:

• syslog-ng or rsyslogd logging to a central location.
  • Pros: centralized logs, easy to analyse, near to real time
  • Cons: Requires extra dedicated server (costs). Have to extend storage or upload to S3 periodically. PITA setup from a security POV unless you know the IP addresses in advance (you can set up your own CA and generate a number of SSL keys and distribute these to the servers on startup and use TLS encrypted communications, but I really CBA with all that hassle - not to mention that last time I checked Amazon Linux didn’t support these servers OOTB so I’d have to install them from a CentOS RPM or similar)
• Message queuing
  • (I don’t think this is a perfect match)
• Hadoop cluster
  • Pros: awesome data crunching ability
  • Cons: we don’t have 10million users yet, I think this is a bit heavy handed (not to mention expensive)
• Scribe from Facebook
  • Cons: too much setup, requires specific server heirarchy.
• logg.ly and splunk
  • Cons: too expensive/untested/unknown for now
• Something else…

I chose “something else” - a custom solution using logrotate to upload the logs to S3 on a per server basis. Here’s how I did it (and how you can do it too) You’ll need:

• /usr/local/bin/s3uploadfile download - a script that uploads a file to S3…
• /etc/init.d/s3logrotate download - an init script to trigger logrotate to be called (twice) on server shutdown
• /usr/local/bin/s3logrotated download - a callback script to call when a log has been rotated (process the logrotate message and calls s3uploadfile to upload the log to S3)
• /root/.s3env - a simple bash script to export your s3 credentials for s3uploadfile to work

1
2
3
4

#!/bin/bash
export S3BUCKETNAME='MyBucketNameHere'
export AWS_ACCESS_KEY_ID='MyKeyHere'
export AWS_SECRET_ACCESS_KEY='MySecretHere'

Step 1: Create your dedicated server log S3 bucket (if you haven’t already)

Step 2: Download and install the scripts above to the locations above, including /root/.s3env and then make them all executable (chmod +x [file])

Step 3: Modify the relevant logrotate scripts to use s3logrotated callback and add s3logrotate to the server rc*.d scripts. When configuring logrotate: don’t use dateext, do use compress, do use delaycompress, do use sharedscripts. Here’s what I use:

1
2
3
4
5
6
7

# This is what I used, you will need to customize this for your own servers
sed -i /etc/logrotate.conf -e 's/^dateext$/#dateext/;s/^weekly$/daily/;s/^rotate 4$/rotate 15/;s/^#compress$/compress/'
sed -i /etc/logrotate.d/httpd -e 's/^\([ \t]*endscript\)/\t\/usr\/local\/bin\/s3logrotated ".2.gz" "\$1"\n\1/'
sed -i /etc/logrotate.d/syslog -e 's/^\([ \t]*sharedscripts\)/\1\n    delaycompress/;s/^\([ \t]*endscript\)/\t\/usr\/local\/bin\/s3logrotated ".2.gz" "\$1"\n\1/'
# This is the centos command to add the s3logrotate script (it contains the levels configuration within it) - 
# you'll want to update-rc.d or similar on debian/ubuntu hosts.
chkconfig --add s3logrotate

Note that the ".2.gz" is because of compress and delaycompress - it is the first compressed log file. This is why we need /etc/init.d/s3logrotate to run ”logrotate -f” twice on server shutdown.

Step 4: Analytics I suppose, but I leave that as an exercise for the reader. I hope this helps someone - if it does, let me know in the comments. If you want any help setting it up (or understanding how it works) then just shoot me an email or leave me a comment.

The cable comes out of the wall:

And then via a splitter:

To the V+HD box:

Note that the “online” LED is lit up, suggesting the system might be online? And the modem:

(it’s not clear here because my flash drowned it out, but the “enet”, “sync” and “ready” indicators are all green, and the “recv” and “send” indicators flash when I load the website below.)

(It’s an ambit 256 modem)

(Note that cable plugs in the back.) Here’s what happens when I connect my laptop to the modem:

1. I am given an IP address of 213.107.40.198 (I have proof this is an NTL/Virgin Media IP address at the bottom of the post)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

jem@jem-laptop:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:22:15:8f:a5:50  
          inet addr:<strong>213.107.40.198</strong>  Bcast:213.107.40.255  Mask:255.255.255.0
          inet6 addr: fe80::222:15ff:fe8f:a550/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:10 errors:0 dropped:0 overruns:0 frame:0
          TX packets:43 errors:0 dropped:0 overruns:0 carrier:2
          collisions:0 txqueuelen:1000 
          RX bytes:2606 (2.6 KB)  TX bytes:9440 (9.4 KB)
          Interrupt:28 
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:12 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:800 (800.0 B)  TX bytes:800 (800.0 B)

1. I am taken to the following screen, asking me to activate broadband
2. When I click the link, it asks me to choose between Virgin Media and ntl/telewest

3. I choose virginmedia and the page asks me to phone you:

   Here’s what I see from the V+HD box:

   

Here’s a video of the V+HD box in action, showing the terrestrial channels delivered over cable, but the premium channels cannot be viewed because I don’t have a subscription: The radio channels also work for me through the V+HD box as I guess they are unprotected.

In conclusion: please believe me now that I do in fact already have fibre optic at this house and let me sign up!

Proof that the IP address I was given is a Virgin Media (NTL) address:

$ whois 213.107.40.198
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: This output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '213.107.40.0 - 213.107.45.255'

inetnum:        213.107.40.0 - 213.107.45.255
netname:        NTL-BIA-LONDON-BROMLEY
descr:          NTL Infrastructure - Southampton
country:        GB
admin-c:        NNMC1-RIPE
tech-c:         NNMC1-RIPE
status:         ASSIGNED PA
mnt-by:         AS5089-MNT
source:         RIPE # Filtered

role:           NTLI Network Management Centre
address:        NTL Internet
address:        Crawley Court
address:        Winchester
address:        Hampshire
address:        SO21 2QA
phone:          +44 1633710142
admin-c:        MH22007-RIPE
admin-c:        NR731-RIPE
admin-c:        CW1083-RIPE
tech-c:         MH22007-RIPE
tech-c:         CW1083-RIPE
nic-hdl:        NNMC1-RIPE
mnt-by:         AS5089-MNT
e-mail:         pim@virginmedia.co.uk
source:         RIPE # Filtered

% Information related to '213.104.0.0/14AS5089'

route:        213.104.0.0/14
descr:        NTL-UK-IP-BLOCK-4
origin:       AS5089
mnt-by:       AS5089-MNT
source:       RIPE # Filtered

% Information related to '213.107.0.0/16AS5089'

route:        213.107.0.0/16
descr:        NTL-UK-IP-BLOCK
origin:       AS5089
mnt-by:       AS5089-MNT
source:       RIPE # Filtered

% Information related to '213.107.0.0/17AS5089'

route:        213.107.0.0/17
descr:        NTL-UK-IP-BLOCK
origin:       AS5089
mnt-by:       AS5089-MNT
source:       RIPE # Filtered