Over the past year, I’ve contributed a handful of readings to the wonderful Starship Sofa science fiction podcast. Ciaran O’Carroll and Tony C. Smith began the show in 2006, as an in depth discussion of the life and works of a variety of New Wave and Golden Age Science Fiction authors. Ciaran left the show last year, but far from this being the harbinger of podfade, it spurred Tony on to new heights of fevered podcasting activity. The Sofa began to acquire the audio rights to a host of science fiction stories, poetry and factual articles, and started soliciting it’s own fictional content in the form of flash fiction.

Today’s Sofa has grown from a single weekly podcast to a pot-pourri of science-fictional productions, regularly releasing a variety of shows under the banner StarShipSofa: Science Fiction Audio Magazine.

Tony manages to regularly crank out the Round Table (a discussion of SF stories with some of the Sofa’s dedicated crew of readers and writers), Engine Room (a peek behind the production curtain), Aural Delights (short stories, essays and the like, read by a variety of professional and amateur micsmiths), and Sanatorium (an audio diary of his own eccentric doings).

The importance of the Star Ship Sofa, in this era of struggling old media, and horrendous science fiction magazine sales in particular; can’t be overestimated. The show is introducing a new generation of listeners to science fiction as it was meant to be enjoyed. Not merely as an alternative setting for a samurai or cowboy adventure, but as a fertile ground for new and strange ideas about mankind’s place in the universe, our relationship to technology, and the nature of consciousness itself. Great science fiction stories exude what’s become known as sensawunda, a glorious feeling that – like deja vu – has the quality of a unique and singular emotion. It’s clear that Tony and his fellow ‘Sofanauts’ share an enthusiastic familiarity with this addictive state.

So far I’ve done six readings for the Sofa (including two not yet released), you can download four of them (as part of Starship Sofa ‘Aural Delight’ shows) below…

Billy And The Wizard by Terry Bisson
The Spenser Inheritance by Michael Moorecock
Passion Ploy by Matthew Hughes
Gigantic by Steve Aylett
The Serial Murders by Kim Newman – Part 1, Part 2, Part 3 & 4.

I’ve enjoyed these readings enormously; even when I haven’t liked the stories, which has been quite often! Producing over the top characterisations, coming up with unique, appropriate (and often awful) accents for each character, and gaining skill in intonation and production, has been a whole lot of fun. Thankfully Tony has been heroically lenient with his narrators, allowing a great deal of leeway for creative interpretation. This is a good part of the reason I’ve just equipped myself with a new podcasting rig, as I’d previously recorded readings in TCD’s radio station. The show was also one of the main reason I began writing science fiction again, and hence sold my first story last year, to Irish magazine Albedo One.

You can check out everything Sofa related at Starshipsofa.com, including the sites excellent forum.

Synopsis:

I detail my experiences with a WordPress hack across multiple shared hosting sites, the steps taken to recover WordPress and secure against future attacks.

Introduction:

On the December 15th I discovered a number of my WordPress installations had been compromised. Rather than a concerted attack, this was likely the result of widely available scripting tools that allow ‘crackers’ to exploit known vulnerabilities in out of date WordPress installs. Due to the large number of WordPress installs on my server, and a reluctance to run bleeding edge software, I’d been a little remiss in updating WordPress. While none of my installs were more than a couple of months out of date, 5 of my 8 WordPress installs were infected.

Viewing the source code of my sites in Firefox confirmed that advertising links had been inserted directly into the wordpress PHP code – behind a DIV visibility tag – presumably to increase the Google ranking of trojan or spam sites. Additionally, the SQL databases in which WordPress stores it’s data had, in several cases, had additional users added. These fictitious users were most often called simply “WordPress”.

I’m going to outline in this post the steps I took to deal with this infection and to reduce the chance of future attacks. I’m not a security professional, or even a programmer, so my advice is provided ‘as is’. Implementing these measures won’t protect you from a dedicated hacker – if someone wants to crack your website in particular, especially on shared hosting, no amount of effort will stop them. However, these techniques should recover your site, and make WordPress a little less vulnerable to automated scripting attacks.

Option 1 – The Nuclear Option

After you’ve been hacked, the best way to ensure that WordPress is completely free of nefarious links (which can be hidden in obscure lines of code in your theme, WordPress itself, or even plugins), is to use WordPress’s export feature (Tools -> Export) to download an XML file of all of your posts. You can then set up a new directory on your server, install WordPress from scratch – using a new database, new passwords etc, and import all of your previous posts. Finally you can delete your original WordPress install and database.

This method however, has an enormous number of drawbacks. Not least is that you’ll have to manually copy your uploaded media files to your new wordpress folder – either by downloading them and reuploading them via FTP or SFTP, or by copying them using a shell command like scp. [Aside: If you have shell access to your server, you can login using terminal on OSX, or a programme like Putty on Windows]. You’ll also need to confirm that your .htaccess file rules and permalinks structure are replicated in the new install. Likely too, you’ll have to manually update the location of images in each of your old posts, or run an SQL command from phpMyAdmin to do this. Any customisations you’ve made – and failed to back up – to your WordPress theme, will have to be manually redone. Finally, you’ll need to manually rebuild all of your blogroll and other sidebar links, as these are not included in WordPress’s export.

I was forced to do the above for one of my WordPress installs (actually this site) which refused to work after an update to the latest version (at the time of writing 2.7). This method should be reserved for situations when (the user front end of) WordPress has broken down after a hack and attempted repair, or when you have reason to believe that changes have been made to the SQL database underlying WordPress: specifically, changes you may not be able to find or safely remove.

Don’t forget, you’ll still need to check the content of individual posts, as it’s possible – though unlikely – an attack may have pasted spam links into posts themselves – see ‘Checking for Damage’, below.

Option 2 – The Conservative Option

When it came to most of my WordPress installs, cleaning the infection was less arduous. First of all I upgraded (via SFTP, FTP’s encrypted brother) to the latest version of WordPress. SFTP is available in most FTP clients (E.G.: Filezilla, Fetch), although your host may not support it.

Next I deleted any unusual WordPress users, and changed the passwords of my existing users. You can do this pretty safely in phpMyAdmin, which you should be able to access from your hosts admin panel. Instructions on changing the passwords are available here – wordpress allows quite strong passwords, including symbols. You can generate new passwords using a variety of online services, like this one.

It’s also advisable to change the wordpress ‘admin’ user name to something else, you can do this in phpMyAdmin when you are changing the password – just remember to keep a (secure) copy of your new login names and passwords!

To delete dodgy users simply delete their fields in the ‘browse’ view of the wp-users section of your WordPress database (again in phpMyAdmin). Hint: If you can’t remember your database login and password information, you can find it stored (unwisely) in plain text, in your wp-config.php file, which you can get to via SFTP or Shell.

Next I changed the user passwords of my SQL databases to the strongest allowed by my host. In Dreamhost this is done in the admin panel, by going to Goodies -> Manage my SQL -> Users with Access, and clicking on the user name listed in your wp-config.php file.

Just to be clear, these are not the WordPress passwords stored in my databases (the ones I use to login to wordpress), but rather the passwords of the databases themselves (the ones I recovered from the wp-config.php files). After I’d changed my SQL user passwords, I also need to update my wp-config.php files with the new passwords (so my WordPress installs could login to their respective databases).

So to recap. So far – for each of my wordpress installs – I had 1) reinstalled WordPress, 2) deleted strange users, 3) changed my database user login name and password, both in my host’s database management panel and in WordPress’s wp-config.php.

Checking for Damage

Now I needed to check that there were no remaining stolen advertisements or malware scripts hanging around WordPress. I did this in two ways. First I checked my sites themselves, as the world see them, by surfing to each site in turn, and checking the outgoing links on a variety of pages. Firefox used to have a feature which did this automatically, but this has been deprecated in recent releases. Fortunately you can reinstall it via this plugin. After you’ve installed the plugin and restarted Firefox, just go to Tools -> Page Info -> Links, in Firefox, to see the links present on any page. It’s an excellent way to quickly scan for links that shouldn’t be there.

Second, I checked (as best I could) that there weren’t any strange scripts or backdoors left in any of the WordPress themes or files left over from my update. As this can include an enormous number of files when plugins are taken into account, I completely removed all plugins from my WordPresses, and reinstalled each of them.

Getting plugins set up is now much easier – as the latest version of WordPress allows you to install them directly from the dashboard, without having to download, unpack and upload manually.

Once I’d done this, I checked the remaining files (i.e.: all the contents of my wp-content folder, except my new plugins) in a text editor via SFTP. This can take a while, and it can be difficult to know exactly what your looking for, especially if like me your knowledge of PHP comes from playing around with WordPress. As of right now, I’m not aware of any automatic way to confirm the integrity of these files.

Ok – so at this point I’d updated my WordPresses and cleared out any dross left over from the attack. What could I do to protect from this happening again?

Protecting WordPress From Future Attacks

Backups

Database backups won’t protect WordPress, but they could make it a great deal easier to restore my databases after any future attacks.

A variety of plugins have been designed to optimise this process. I use WP-DB-Backup, however backups created via WP-DBManager might be easier to ultimately restore (however this plugin has not been updated for the latest version of WordPress, so use at your own discretion).

With DB Backup, I scheduling an automated emailed backup once a week. I set up a new Gmail account created for this purpose, with a strong password – as these backups include all of my WordPress passwords and content (although passwords are MD5 hashed).

I also created a Gmail filter which lets through all emails containing “WordPress Backup” (the subject in emails created by WP-DB-Backup), as some hosts (including my host Dreamhost) have made Google’s greylist, and emails can get automatically discarded to the spam folder.

Next, I manually backed up my customised themes, something I’d neglected to do as often as I should have.

NB: It wasn’t enough to secure just WordPress. I needed to update and secure other CMS installs on my server, as it only takes one compromised service to infect your server with hideous nasties. For me, these included Gallery 2, MediaWiki, and Moveable Type installs. I also used this opportunity to delete older unused CMS’s and databases which were creating a risk on my server.

NB2: I learnt not to rely on Google to tell me if my sites have been compromised. While, Google do automatically scan pages, advising surfers and webmasters if pages have begun to carry a payload, it’s worth noting that despite 5 of my installs becoming infected Google only notified me of 2 of these penetrations.

In other words, if you haven’t updated WordPress for a couple of months, and or if you’ve discovered an infection on one of your sites / installs, it’s highly advisable to manually check all of your sites.

Securing Access To My Host

The ability to recover after a hack is reliant on access to your host’s (in my case Dreamhost) admin panel. I strengthened my password, and confirmed that my account details – phone number, security questions, associated email accounts etc, were up to date.

I chose to turn FTP off altogether, in favour of SFTP (FTP with SSH encryption). On Dreamhost, I did this in manage users -> edit -> tick ‘disallow FTP’.

Your host may also provide specific security functionality to help you lock down access to administrative functions. For the rest of this section I’m going to list Dreamhost specific security options.

Dreamhost Specific Security

I enabled Dreamhosts enhanced user security, by going to manage users -> edit -> and ticking ‘Enhanced Security’.

I also manually backed up my entire account.

I checked that no one else had access to my account, in account privileges.

Dreamhost allow you to link your login to their admin panel to a specific IP. As I’ve got a static IP address and only need to access my admin functions from a limited set of locations, I did this in Edit Profile section of the Dreamhost admin panel. In Preferences -> tick ‘ Require email confirmation…’.

NB: If you’re planning this, make sure that your correct email address is saved here, and that it is working. Once you enable this option, you’ll be logged out of your web panel till you confirm access to your IP address by email.

WordPress Security Plugins

A variety of plugins are available to improve the (pretty awful) default level of security in WordPress. I’ll list the ones I found useful, and briefly outline some of their functionality.

1 – WP Security Scan – Scans WordPress directories for appropriate permissions, automates production of strong WordPress passwords. The makers promise new features, including WordPress admin protection and one click folder permissions change, in future releases.

2 – WP DB Backup – See backups section above.

3 – Login Lockdown – protects WordPress against brute force / dictionary attacks. It’s something that you might assume WordPress does by default. It doesn’t.

4 – Bluetrait Event Veiwer (BTEV) – This fantastic plugin provides a widget (which I dragged to the top of the dashboard for instant visibility), letting you know who has tried to login to WordPress (and whether or not they’ve been successful), what emails have been sent from your install, what plugins have been activated or deactived, and lots of other useful stuff.

5 – AsakApache Password Protect – I actually haven’t been successful in enabling this plugin in a dreamhost shared hosting environment, but it may be useful for you. The plugin protects a variety of WordPress folders from third party access. If the PHP environment ever crashes or is horribly misconfigured on your server, as happened to Facebook, this could prevent the outputting of raw code, passwords etc, to anyone who loads the right address.

NB: Be careful with this plugin – if it locks you out of wordpress folders incorrectly, you may find your wordpress install inoperative until you manually remove (via SFTP) both the plugin and all of the .htaccess files it has created.

6 – UTF8 Sanitise – After I updated wordpress across my sites I found a huge number of older posts containing odd characters where accents, umlouts and punctuation should be. What happened is that more recent WordPress versions display characters using UTF8 rather than latin characters. So anywhere I’d cut and pasted from word, the web or acrobat into WordPress, I found that extra characters had been added. UTF8 Sanitise lets you manually resave older posts, repairing their encoding in the process.

You can also do this for all your posts at once using UTF8 Database Converter, but again this hasn’t been updated to work with WordPress 2.7, so I’d be cautious using it (considering the database modifications).

Conclusion:

With such an enormous number of blogs being attacked, and such a short time between vulnerabilities being discovered and exploited, it seems the perfect time for the equivalent of anti-virus to emerge – software to actively scan for vulnerabilities and nefarious changes to WordPress, and indeed such scripts are becoming available. One such tool is WP Scanner, which uses a combination of wordpress plugin and website to check for vulnerabilities.

NB: Once again, be careful uploading .htaccess files to wordpress subfolders. On my server, using .htaccess to block outside access of wp-content or wp-admin folders severely impairs wordpress’s functioning.

For more information on securing wordpress, check out BlogSecurity’s whitepaper on the topic.

The most important thing I’ve learned is to update WordPress, as soon as possible after a new update – ideally within days. This is fortunately now a lot easier with WordPress’s automatic update functionality.

There’s a profound fragmentation going on right now in online identity. The change from isolated web sites (with technical, and in the early days, financial barriers to entry) to readily updatable, easily subscribable, often free blogs, acted to more tightly integrate the public web. An unfortunate side-effect of newer forms of blogging, more sophisticated social networks, and the surfeit of emerging web services, has been to splinter identity across multiple platforms. Ultimately this problem may be solved by more open and intercommunicative social networks using something like Open ID, by an ur-MySpace aggregating all the disparate services, or by next generation life stream scrapping utilities. Until then here’s where where I am..

Del.icio.us – makes bookmarking easy, available anywhere, and subscribable like a blog. It’s also a great way to keep informed of your friends projects.

Twitter, Jaiku – moblogging platforms, which allow you to create a mobile social network of sorts, create and receive updates on the go.

Facebook – my social network of choice. I’m on Bebo, Myspace, Live Journal, Orkut, and now Pownce; but Facebook plays host to my primary ‘online identity’. Facebook’s newsfeed provides a great way to keep up with the goings on of real friends; and it’s open API and platform are providing an evolutionary paradigm, an environment in which shared ideas, intense competition and rapid development are accelerating the improvement of social applications.

Tumblr – I use as a meta blog, adding all my feeds, and posting photos, videos, songs, and cool articles. I made the decision a while ago to keep the focus of dbspin.com on original articles and occasional media projects, which reduces hits, but hopefully increases quality. Additionally, I’m finding less and less time to blog proper, and in turn, feeling that each blog post must attain some arbitrary level of quality and originality. I believe quality is becoming more important relative to timeliness in the blogosphere; as rapidly updated information moves to micro and mobile blogging platforms, and as more people begin to suffer from feed aggregation overload. Tumblr’s also just a great way to give a technophobic friend a really user friendly blog they can update online, or via their phone.

An interesting point to note, is that for non developers there’s really no need to buy webspace any more; given that most features from web pages, to blogs, video hosting, to podcasting, are possible and easier to implement via free web services.

Right now I’m involved in three major projects. One is a prospective web2.0 / mobile service company, two is a documentary film, and three is co-writing a satirical (and partly parodic) novel. I can’t talk about the first two right now, but I’m more than happy to suggest you check out the book, which is being updated in real time on the web (please avoid if even moderately easily offended), or the main satirical review site – with over 100 movie, music and other reviews (again offensive); not too frequently updated right now, but that should change in October.

As Twitter leaves the realms of ‘joiner geek’ social network, and (partly due to it’s integration with platforms like Facebook), becomes a more popular and diverse service, its utility is being more rigorously critiqued. What is Twitter for? Its a cogent question. In a world of blogs, microblogs, and social networks, what’s the use of twitter? Is it merely a loose knit social network, or a (nano) blogging platform? Is it just the latest fad? Lets look at some real world uses of Twitter, how its utility differs from more traditional blogging platforms, and some scenarios in which it could be more effectively used.

Real world use

Right now, twitters use is primarily as a loosely coupled, mobile social network. The barriers to membership and use are low – as the platform is quick to join and simple to use; and adding ‘friends’ can be done easily and quickly. Perhaps too easily, since no confirmation is required to ‘subscribe’ to a user. Mass adding of contacts within twitter is a quick way of creating an audience – as some of those added will tend to add back the mass connector, creating potentially unwanted one way connections.

Connectivity

Twitter has strong but limited communication features. Users can receive updates (‘tweets’) from their ‘friends’ via web, RSS, IM or (free) text message. They can send their own tweets, through Twitter or a host of other websites, IM or paid text messages.
Tweets are either undirected, direct, or aimed at a user, but visible to the general audience (signified by the @ symbol). A general audience in this context can be either the public at large, or a users entire friend group, depending on privacy setting. Blocking is available – and users can only directly message their friends.
However no fine grained control is available – users cannot for example, chose to receive text message notifications of tweets from one specific user, or group of users. In fact no user groups or gradations of connection exist within twitter. Aspects of this functionality may ultimately emerge from third party applications.

Moblogging

With built in text messaging functionality, Twitter has the capacity to function as a ‘short message’ (140 character) moblogging platform. While it lacks MMS picture support, it has the advantage of immediate one to many connectivity. As such, it can be used to provide news updates and reports of ongoing events.

Public Conversation

A conversation emerges in the ‘friends’ feed of a users twitter; visible from their profile on Twitter.com or via RSS. Although direct messages are not visible to all, as previously stated, frequently publicly visible messages are directed at a user, through use of the @ sign. This allows a users friends or readers to engage in conversation, and allows a user to publicly flag the existence of their communication with high status users. In effect this serves to pull users on the periphery of the network into more direct conversation with more centrally connected users. In this role Twitter can provide a form of ad hoc business social networking.

Linkblogging

Blogging grew in part, out of link logging, and the low time cost and mobile nature of Tweets, makes Twitter ideal for sharing links with a directed audience. Due to the character limit of ‘tweets’, links are often truncated using services like slink.in.

Future Use 1 – Group management

As previously mentioned, creating user groups is not yet a feature offered by Twitter, but their addition would allow significant additional emergent uses of the platform. Work teams, clubs, and institutions could use Twitters push messaging to notify members of meetings and updates. Rather than acting as a replacement for traditional systems of notification (e.g.: email memo’s), Twitters immediacy would make it ideal for notification of last minute changes of plan, or time sensitive communications – especially with large or loosely affiliated groups.

Future Use 2 – Mobile applications

Applications are already being built around twitter, but none have yet taken advantage of the powerful utility of the mobile communication capabilities built into the platform. Twitter applications should in theory proliferate virally, due to the public nature of visible ‘@’ directed communications – in a similar fashion to the recent viral growth of applications on the Facebook platform. However this can only take place if an applications utility necessitates two way communication with its users. ‘Mashups’, which utilise the API’s of two or more external services to create a new service providing additional utility, hold great potential in this regard. Combine low cost two way mobile communication, with social application proliferation, API sourced data, and multicasting, and a variety of potential services become obvious. The viability of businesses which utilise Twitter in this way will be dependent on the future reliability of the platform, and on Twitters tolerance of providing host to an emergent ‘ecosystem’ (positive construal) or ‘parasite infestation’ (negative construal) of applications.

Social Network ‘Facebook’, has made an enormous splash this week with the release of the ‘Facebook Platform‘, an opening up of the mature Facebook API to internal widgets with access to Facebook’s ‘core functions’. Whilst this move has been criticised by some influential members of the syndication community, it places Facebook at the forefront of mashup’s and the read-write web. In one fell swoop, Facebook has become a socially enabled aggregation platform.

‘Zuckerberg describes the Facebook core function that the new third-party applications can tap into as a “social graph,” the network of connections and relationships between people on the service’.
- Dan Farber on ZDNet

Social Aggregation

Let’s look at some ways that Facebook collates user information, and serves this summated information back to a users social network.

Facebook offers users three direct methods of adding to the ‘News Feed’ activity stream of their connections on the Facebook network.
‘Status Updates’ are brief, 160 character messages, updatable by text message or from within Facebook.
‘Posted Items’ are smart links which retrieve a brief descriptive paragraph and photo from a provided link; updatable from within Facebook, and through browser and web based bookmarklets.
‘Notes’ are text, images and links – essentially blog posts; updatable from within facebook itself, and via imported RSS feed. Notes contain an additional feature, a social equivalent to pingbacks, allowing posters to ‘tag’ friends mentioned in, or related to, a post.

Controversially, Facebook also provides an aggregated feed of indirect socially relevant user actions – profile changes, photo uploads and contact addition etc.

Finally, with the addition of ‘Facebook Platform’, Facebook can now socially aggregates the information flow to and from users and their installed applications – for example, tweets updated via and integrated Twitter application.

All of this aggregation is done parsimoniously and noninvasively, with an emphasis on usability and integration with the social map of a users Facebook network – for example, if a user updates a number of notes in quick succession, rather than each note appearing in the news feed of his connections, a lists of titles will appear.

Fine Grained Control

The aggregated ‘Posted Items’, ‘Notes’, and ‘Status Updates’ from a user’s connections, can each be exported as an RSS feed. In addition to the feeds available from applications within the ‘Facebook Platform’, this means that users can now (or will soon be able to, with third party developer support) use Facebook to export feeds of updates to the attention streams, social bookmarkings, and blogs, of their connections.

By allowing users fine grained control over which aspects of their social activity on the site are published to the ‘news feed’ of their connections – for example a user can choose to de-list notification of their new connections completely, or on a case by case basis – and control over the sources and quantity of the information they aggregate from their connections; Facebook have built the beginnings of a social write application to compliment feed reading, on the Read/Write web. Together with the social elements of next generation browsers, this could provide the template for how such services work in the future.

Maturation is inevitable and necessary, and Facebook are only at the beginning of the development of their internal services – for example ‘Posted Items’ can function as social bookmarks, but without tags or folder utility, cannot replace a dedicated social bookmark application.

Balancing the desire of application providers for greater access to the Facebook API, with the privacy of users and the overall usability of the platform will be a difficult challenge. Right now it seems that Facebook are erring on the side of caution – for example, Twitter integration seems for the moment hampered by a lack of write access to user ‘Status Updates’ through the Facebook API.

What to do with all this information?

Beyond hyperbole, what does all this mean? Right now, it means I can import the ‘Status Updates’, ‘Notes’, and ‘Posted Items’ of my Facebook connections; right into Google Reader. I can also export my own aggregated updates, and construct a feed blog, or metafeed – increasing the utility, and decreasing the exclusivity, of my updates to Facebook.

This flow of information will grow richer as more useful applications are added to the ‘Facebook Platform’, as Facebook continues the roll-out of its third generation of internal services (like Market Place and Video); providing a portable, rapidly updated aggregation of social events and conversation – a friends feed, which comprises a deepening, dynamic, and semi-public conversation.

The Future

What’s next? Increased portability of Facebook mobile updates (currently the limited SMS notifications available are restricted the the US), and the easier establishment of networks, would greatly increase the utility of such powerful information aggregation; as would RSS feeds of the ‘News Feed’ activity stream itself. Right now, Facebook is a fantastic tool for large, loosely connected, public social networks, but increased privacy options and network building flexibility could make it a more useful tool for work groups, businesses, and families. Although it’s important to note that company networks do already exit, each new network must be suggested directly to Facebook – with little direction as to the amount or type of requests needed before such a network will be created.

Look out soon, for applications leveraging the social aspects of the Facebook platform in innovative ways – enabling collaborative video editing or games for example.

Open source and information portability advocates would no doubt like to see greater portability of the (user generated) networks with give Facebook its value. Such portability may become ever more difficult, as users become more locked into the services provided by a specific social network, and the social groupings which exist there.

Caveats

You should be aware before you start that posting remixed video online (if you don’t own the copyright to your source video) could theoretically get you into legal trouble.

These instructions are for Windows. Here are some simpler instructions for the Mac. Let’s face it, video stuff is faster and easier with a Mac, if you can afford one.

It’s also important to note before you begin, that uncompressed video files are enormous. You’ll likely need at least 1 free Gigabyte per 5 minutes of video you plan to convert, and much more to do editing later.

If anything goes wrong, I disclaim all responsibility. These instructions are provided as is.

All that said, here’s how to do it..

Download the video

There are lots of ways to download a video from YouTube. Here are a couple.

Throw the address of the video you want into one of these sites
- Keep It Simple, Video Downloader 2.0, KeepVid.

or

Install Firefox, GreaseMonkey, and one of the these scripts.

Once the file has downloaded, you’ll have to convert it before Movie Maker or Adobe Premier Pro 1.5 (haven’t tried this with more recent versions) will open it.

Download Super

The free program ‘Super’ will convert almost any multimedia file to almost any format.

Download Super. The link is difficult to find on the horrendously designed site, but keep looking, it is there!

Convert the File

1. Install and Run Super
2. Find the file you’ve downloaded, and drag it into Super.
3. Along the top of Super, set the settings like so [Image]
   • Output Container: avi
   • Output Video Codec: huffYUV
   • Output Audio Codec: WAV -(pcm U8)
4. Right click anywhere in Super, and click ‘Specify the Output Folder Destination’. [Image]
5. Select the folder where you’d like to put your finished file and click ‘Save Changes’
6. You may wish to increase the size of the output video (by default Youtube’s resolution is 320*240). To do this simply change the Video Scale Size setting (e.g.: 640*480) [Image]
7. Your finished settings should looks something like this [Image]
8. Now click Encode. After a few seconds the video should start to encode [Image].
   In a few minutes (depending on video length), the process will finish.

That’s it!

You should now have a video that most video editors can import and edit without glitches. Happy remixing!

An article by Mike Elgan in Computer World Magazine, laying the boot into e-books, has sparked a surprisingly intelligent discussion on Digg. According to Elgan, e-books are bound to fail because..

1. They aren’t cheaper – both the hardware and content are more expensive
2. Content is available on other platforms (e.g.: PC)
3. People love paper books

Throughout his article Elgan conflates the e-book format and electronic book devices, in a way that confuses the issue of uptake. Perhaps the reason he fails to differentiate between medium and it’s media, is that there are so many kinds of things that can be described as a e-book. Wikipedia for example, lists twenty five e-book formats, including both document types and readers.

Elgan’s article might have been written twenty ago, about digital music..

‘Companies like Sony, Panasonic, Hitachi and Fujitsu have devoted millions of dollars over the past couple of decades developing what they hope will be a device that replaces the..’ [Record player]
‘The hardware costs hundreds of dollars’ [CD Player]
‘everyone already has alternatives’ [Vinal, Tape, 8 Track]
‘do people want to “curl up” with a battery-operated..’ [iPod]

For a new format or device to succeed, what matters is not how much people like an existing product, but how much they would enjoy an alternative with greater function. Currently e-book readers (like all digital display technologies), are in their infancy. We know this because their development is occurring so rapidly. The best digital display available in 1987 was a 16 inch VGA CRT, boasting 256 colors at a 320×200 pixel resolution. Twenty years later, digital images can be displayed on a variety of output media from 63 inch flat screen HDTV’s at 1920 × 1080 pixel resolution, in 281 trillion colors; to high contrast, monochromatic 800*600 e-ink ‘powerless displays’.

We each carry a variety of devices capable of displaying digital books; from laptops, to MP3 players, to mobile phones. Digital displays are becoming ever more versatile, ubiquitous and cheap, with increasing contrast, fidelity and resolution. Fujitsu have recently announced the first prosumer colour e-ink display. So why haven’t e-books already taken off?

Existing efforts are crippled by Digital Restrictions Malware, and available in a bewildering variety of incompatible proprietary formats, from Adobe, Sony, Microsoft, Mobipocket, eReader and others. Publishers fear they will experience the same growth in copyright infringement that the record industry claims has negatively effected sales. It may already be too late. In the absence of reasonably priced, DRM free alternatives, consumers are turning to unlicensed downloads, just as happened with music and film. A quick search of isoHunt, a top bittorrent index, for the term ‘DVD’ returns over 26,000 active downloads. A similar search for ‘Book’, returns over 4,000.

A great majority of these files are posted without their authors consent, but some publishers and authors are embracing digital distribution. Blogger and award winning science fiction author Cory Doctorow, has distributed all his novels online for free; releasing digital versions simultaneously with their paper equivalents. A few publishers, like Baen Books, have adapted to the new marketplace, making available older content for free, and selling reasonably priced, DRM free, multi-format e-books, with subscription options. Initiatives like Project Gutenberg, seek to make digital copies of public domain books universally available. Whether publishers eventually embrace consumer friendly formats, or continue to ignore them, digital e-book content will continue to grow in availability.

With e-book readers, the costs of adoption are still high, as dedicated devices or high resolution PDA’s still cost hundreds of euro. Similarly, while common devices like iPod’s can technically display e-books, such uses often require a degree of technical knowledge, and force users to struggle with unfriendly user interfaces. This should soon change, as devices like Apple’s iPhone usher in a new generation of high resolution, high contrast digital display devices. While Apple seems likely to restrict the iPhone’s use, their competitors will be more than happy to capitalise on more open platforms, whilst learning from Apple’s user interface innovations.

Digital books provide a variety of predictable advantages, as well as many which will not emerge until they become more evolved. Right now groups like The Institute for the future of the book, are hard at work ‘inventing new forms of discourse for the network age’, and their efforts provide an insight into just some of the potential benefits of e-books..

1. Collaborative writing / revision / comment / annotation
2. Effectively free wireless distribution
3. Smaller form factor – potentially infinite books in one networked device
4. Environmentally friendly
5. Text search
6. Updateable
7. Rapid universal publication
8. Dynamic user interfaces
9. Flexibility of format
10. Interactivity

Whether e-book’s are ultimately consumed on laptops, dedicated palmtop devices with flexible screens, enhanced newsprint, heads up displays, or by all these and other means, is impossible to predict. Right now paper books are far more durable, resilient, and user friendly than any of their alternatives; but as an analogue medium, their development is slow and expensive. E-book’s by contrast, benefit fully from the brakeneck pace of accelerating technological change, and offer so many potential advantages in cost, portability and capability that their adoption is all but inevitable. Witness the publication and consumption of scientific articles, which though nominally tied to peer reviewed magazines, increasingly occurs initially online – increasing the speed, penetration, and availability of research.

Digital consumption will affect the format of books, as it has already affected the format of articles published on the web. There will always be a market for traditional ‘dead tree’ editions; but ‘the book’ will likely morph and splinter into a variety of forms, and the nature of authorship will change with it. This is as an evolution of discourse as significant the creation of written language, or the invention of the printing press. It’s an exciting time to be a reader, and an even more exciting time to be a writer.

Just back from Barcamp Ireland 3. What a day, so packed I couldn’t possibly get to half the talks (which ran concurrently), but I managed three lectures and the panel discussion. The event was held in the beautiful Digital Hub, off Thomas St in Dublin. The building is fantastic, with bare brick walls and natural lighting throughout, and would make a fantastic billionaires studio apartment.

I’ve posted some wikified notes. I grabbed several mini interviews, on TFM‘s sweet but pricey Roland wav recorder (check out the uber cheesy website), not enough material for a full blown podcast, but I’ve thrown them up, below. Also attempted to moblog throughout the day, with mixed results.

Mini Interviews

Sean O Sullivan of Rococo.
Robin Blandford creator of Comment Casting.
Darren Barefoot of Capulet Communications.

Update: Fixed the wiki link!

In other news, my wonderfully creative friend Andrew and I were having so much fun producing astoundingly erudite and informed reviews for TFM, that we decided to break out a site dedicated to our own elitist brand of music criticism. Check out the new site, Gil-Martin Writes.

Retitled in mid January to ‘Joost‘, TVP is an IPTV application. That is, a program designed to carry high resolution video direct to consumers via the internet, rather than through satellite, cable or terrestrial broadcasts.

On the technical end, Joost uses both the UDP (to stream video direct to viewers) and TCP/IP (to share shows between users) protocols to create a hybrid Peer-to-Peer and Streaming, MPEG 4 H.264, (currently) free, on-demand TV network.

While Joost does live up to its promise to deliver full screen, uninterrupted streaming video at a watchable quality, a variety of potentially insurmountable challenges stand between the company and its goal of subverting broadcast television.

1. No Premium Content

Whilst Joost’s founders perhaps aim to ultimately provide licensed ‘A Plus’ broadcast content, current offerings are thin indeed, and there is little to suggest that the company has the industry connections or necessary expertise to lure rights holders into making such content available.

“Watch for sci-fi shows, rock videos, sports, comedy — anything with a testosterone angle. Deals are in the works with the three music majors, plus top US broadcasters and cable channels. For the rest of the world, there’s a modified PBS strategy: classic reruns, documentaries, and independent dramas.”

- Wired Magazine

While there’s clearly a market for cheap, low quality television, creating a new distribution channel for low value content is unlikely to bring about the paradigm shift in multi-hour-a-day TV watching articulated by Joost’s founders in the latest issue of Wired Magazine.

As Zennström points out..

“You have to put together a whole consumer offering, a great instantaneous experience. A simple service that fills an obvious need and can be offered for free.”

Right now there’s no compelling reason to think that Joost can deliver on the ‘obvious need’ element of such aspirations. Kazaa and Skype were technological and infrastructural achievements, not content deals.

2. No User Content

It may be too early to tell, but early indications are that Joost, like the iPhone, will be a closed platform. Although the developers, unlike Mr. Jobs, welcome independent extension development, it appears they will avoid distribution of user uploaded content entirely. With online only content, and content production companies, from the Revision 3, TWIT, and Podshow networks to Channel101, growing in quality and diversity all the time, this has become the golden age of high quality, short format, web friendly media. At the other end of the scale, YouTube’s survival in the wake of massive takedown notices and competition from far more piracy friendly alternatives, signifies a huge unanticipated interest in the lower end of ‘user’ generated media; from inventive indy band videos, to post modern soap operas, to emerging comedians. According to Wired, the Swedes have no desire to tap this particular well spring of talent.

“Content that few people want to see — what Leiden engineers call “the too-long tail” — crimps a P2P network’s advantage.”

Whether you accept the sincerity of this reasoning or not (in fact, shows like ‘Diggnation‘ and ‘This Week in Tech‘ regularly beat the viewership of much cable television), if Joost’s founders follow their stated plan, they will fail to do for syndicated internet video what apple managed for the podcast, and in the process potentially waste their biggest advantage over ‘content providers’ like Apple, Microsoft and the major vertically integrated media corporations – rich, freely generated ‘sticky’ media.

3. Computers don’t deliver the TV experience

In contrast to active clip grazing or movie watching, computers are ill suited to the casual background parsing of TV. Theres a missing piece of the IPTV puzzle that Joost cannot in its current form solve. A link between the expanding, thinning, television and the computer. The key here is that connectivity must be bidirectional. It’s no use connecting your laptop to your Plasma via a composite cable (and in the process distracting your computer from any useful task), if changing channels necessitates a return to the keyboard. Technical solutions to this problem abound these days, with devices from companies like Slingmedia and Cisco prepared to carry high resolution audio and video to next generation HDTV’s. These are however, likely to be niche products when compared to the omnipresence of media centers in the form of the Microsofts Xbox 360, and Apple TV, both of which offer the potential to bring pay (and pay per view) programming to the television; in the form of all important big media licensed content. Arguably these players are in a far better position to ‘migrate broadcast television’s mass audience to the Web‘.

4. User Bandwidth

With grave doubts proliferating, as to the ability of the internet as it now exists to manage the load created by the growth curve of streaming video, and considering the effective and explicit bandwidth limits placed by ISP’s on their customers, it is by no means obvious that Joost can succeed in the mass market.

According to Joost’s detailed FAQ (beta customers only), in one hour’s use of the service “approximately 320Mb data will be downloaded and 105Mb uploaded”.

In Ireland, the broadband penetration poor man of Europe, common download and upload ‘allowances’ start out at around 10 GB download / 1 GB upload per month. That’s less than 10 possible hours of Joost watching, hardly enough for the program to replace broadcast television. Whilst this represents the worst peak of corporate bilking (leading to one of the worst broadband takeups in the EU), bandwidth limits are an uncomfortable reality throughout Europe and daytime and application targeted bandwidth throttling common in the US and UK.

5. Too Complicated

The Joost interface itself is a model of parsimony and slickness. It is also however, neither familiar nor obvious. Control of the program is via a video recorder / PVR interface metaphor, but in it’s current form more complex and ‘mystery meat‘ than either. Elements of the interface, like the channel library, used to manage channel subscriptions, and the ‘mychannel’ link used to open specific program items, remain unintuitive. A higher level interface design may prove necessary to ease the public into a world of fullscreen online television.

6. Advertising Specificity

Whilst Joost will offer advertisers a variety of viewer demographics – geographic, temporal, and viewing profile – this is not the kind of information which is necessarily most valuable to mass market advertisers. Whilst broadcast television ratings provided by companies like Nielson are deduced from representative samples rather than IP headcounts, they provide rich demographic information that includes income, employment and interests, for each data point. The difference, crudely put, is between ‘what kind of person’, an ‘what time and place’. Additionally, while Joost may allow advertisers to know which programming is viewed when, and which ads are ‘flicked’, this is a capability shared by existing PVRs, and one Zennström and Friis may well choose to leave disabled.

7. Visual Quality

Thanks to James Corbett, I’ve had a chance to sample the Venice Project over the past few weeks, and I can report that under XP at least, the visual quality is not up to the DVD standard claimed. Whilst the resolution may be as high, some channels have a washed out muddy look, more often associated with Flash video.

Check out a screen shot here (NOTE: This is a 2meg uncompressed BMP file!), a static camera, full resolution shot from the Green Day documentary ‘Bullet in a Bible’, you’ll notice it’s much lower quality than the reference images provided by Joost (check out a second screen grab here, from a different video, exhibiting higher quality). While this may be a function of the low end laptop I’m using, its a noticeable reduction in quality compared to watching a DVD or other H.264 content on the same machine.

For comparison purposes here’s another screenshot, a frame from an apple trailer played fullscreen in 480p, roughly equivalent to DVD resolution (again 2meg!). Perhaps image quality is increased on a higher speed connection? Our home machine is connected to Digiweb DSL XTRA (theoretical 3 meg down, 384k up), and about two miles from our exchange.

In summary, picture quality is certainly more impressive than rival streaming web formats, but not quite up to DVD. Does this matter? Well that depends a lot on point three. Sitting two feet from the screen its O.K., but I wouldn’t want to see a movie this way.

8. Not a Magic Bullet

Much like Zennström and Friis’s previous projects, Joost is not a radical advance in the state of the art, but rather a more reliable implementation. The technology to distribute high quality video via IP already exists, but is currently poorly implimented. What governs future uptake in this area is likely a mixture of legal (IP and regional distribution restrictions), economic (distribution costs), and psychological (ownership preference, tolerance of DRM) issues, rather than a best of breed technological race.

All in all there are many things to like about Joost, it’s interface, while likely too complex for casual users, is efficient and geek friendly; its expandability and image quality, almost instantaneous channel streaming, and the promise of ads as short as ‘one minute per hour’, are all commendable. However it’s ability to compete against free, burnable downloadable content, and branded services with a hook to the television, remains to be seen. There do exist however, at least two scenarios which could spell huge success for the upstart company.

1. Pay per view

If Joost can snag studio support, their service could provide a terrific (if lower resolution) alternative to ‘legal’ movie downloads, which if marketed correctly (read rented cheaply) could create a whole new market for lower resolution video – or cannibalize existing DVD sales.

2. Selling Out

The rapid skipless streaming technology, and piracy resistant (sic) encryption behind Joost, could make it an attractive purchase for use by a worried major TV network, suffering media giant, or net enabled set top box maker. Perhaps licenses to ala carte device or service tailored versions of the software and the backend network which supports it could be sold to multiple providers?

In the absence of either circumstance, Joost may well end up the Betamax of streaming video.