The Fail Whale

The popular micro-status service Twitter has a huge problem: itself. With the explosion of startups and sites dedicated to aggregating Twitter, the Rails-based monolith has been feeling the pain architecturally as it struggles to keep up with its (frankly appalling) 100-API-request-per-hour limit.

As a micro-industry grows around it, Twitter is subject to evermore-creative uses such as real-time emergency broadcasting and the aid of disaster recovery, with the assumption that its architecture is strong enough to disseminate critical information in real-time, reliably. This is simply not the case, and serves as a perfect segue into our next question: can lives be trusted to Twitter?

And there was Light

When Twitter was first created, I doubt that the designers ever thought that it would evolve to become what it has today, nor do I think that they really understand what they’ve created, and the responsibilities they have to their creation. Today, over three million messages will be posted to Twitter. Yes, the majority of them may be stupid (my personal favorite example: “I’m eating a salad.” … “Who gives a fuck!?”) but there will be a great deal of them that affect the way people live, from meeting arrangements to flood warnings. What happens if the messages don’t get where they’re supposed to?

It is absolutely critical for Twitter to realize that it is no longer a fancy-free garden-salad-variety website — Twitter is now a public utility, and in my opinion it is time that they accept that responsibility and start behaving as such.

Walking the Talk

Twitter has a long way to go if it is to become a utility. Their basic goals should be:

1. Run from Ruby and Rails - I love Ruby. I think it is the best language out there, but it and (especially) Rails are not the tools for this job. Ruby’s interpreter is significantly slower than those of many other languages, and Rails was designed for the agile development crowd. Twitter: You aren’t really adding new features anymore, therefore you don’t need the framework that was designed for that subset of rapid development.
2. 99.9999% uptime (the real definition, not your definition) - First of all, you’re lying about your uptime. I see the fail-whale on a regular basis. In fact, we’ve become rather good friends. His name is Mitch. Not Mitchell. He doesn’t like that. We sit around and drink Mojitos on Friday nights, and play racquetball on Sundays. That’s how often I freaking see him. If your definition of “uptime” applies to your homepage only, then yes, you are up the 99.98% that you claim. People rely on you — it’s time to rethink your topology/architecture. Rewrite the whole goddamn thing. Just do it. Don’t argue.
3. Be API-centric - According to Twitter co-founder Biz Stone, your API traffic is an order of magnitude larger than the browser traffic to your site. If that’s true, then why the hell am I limited to 100 requests an hour? Your savior is going to be the industry you create around yourself, NOT advertising or anything like that. The little industry you’re creating demands data, so give it to them — as much as they can handle.
4. Ensure predictable delivery speed - I’m not saying it has to be instantaneous (although it could easily be) — I’m just saying that the variable wait time is a problem because you give the impression that it’s instantaneous. You’ve effectively become the world’s most delayed IM service.

Competition Fail

I don’t really know much about Twitter’s competitors, but I can tell you that they’ve already lost. FriendFeed and Identica are wonderful services, but they don’t appeal to the masses. They appeal to the geeks — and on Twitter the masses are no longer the geeks.

That said, I have a ton of respect for Identica. The idea and execution are great, but the marketing required to make it mainstream just isn’t there. This is one of those annoying illogical cases where it really is almost all in the name!

Conclusion

Twitter’s greatest success will come when it realizes that the best companies do one thing, and do it very well. They should be done adding features, and instead focus on redeveloping the entire system and monetizing it. They need to begin charging your biggest API users, and they need to do it fast. Most of all, Twitter must remember its responsibility to its users, and know that someday someone’s life may rest in their hands.

Yep, it’s live!

Watch streaming online at: http://www.mogulus.com/newsinnovation_portland The conference is currently going on. You can tune in at the link above to watch and chat. You can also find me on twitter for tiny updates at: http://twitter.com/kenkeiter/

Live Content

11:00 AM: We’re currently discussing bullet-points from our individual discussions on the news business models. Emphasis includes: new data visualization tools, collaboration with the audience, requiring better technology skills in journalists. We’ve also noticed that production in the real-world and production in print face the challenge of a lack of timeline on the web. One of the concerns is that people who are producing for the web lose their scope - we need virtual newsrooms? Journalists need the back-and-forth between peers to gather information? Many disagreements about this. Should consumers provide the primary critique!

11:09 AM: Consumer driven media versus media outlets as gatekeepers — deciding what stories the public sees. Big topic of contention.

11:15 AM: What is the threshold between when consumers want content to be forced to them versus when they gather it themselves. Someone made the point that human interest stories are dugg down? First thing I’ve ever heard about that… How does geographic location affect human interest stories.

11:19 AM: Let’s take a break!

12:02 PM: Talking about marketing and community. How do we generate community and what role should community play in news. Empower users to spread content for you. Talking about EveryBlock. User generated content is advertising — seems to be the concern here. Is it journalism? Or is is just geo-twitter? Static information versus dynamic information. What is the definition of journalism? Is it the provision of data (no matter how small) to empower your community? Or is it stories that are vetted and go through a real process. CRM in news! Managing the relationship with consumers that GENERATE media. This may be an entirely new application… Consumers that generate media are critical in an industry that is being killed by teh interwebz.

12:58 PM: Talked about http://www.mapwith.us/ briefly. We’re talking about geo-relevance for stories and micro-stories. How to tackle social burnout and get people to continue producing content.

Let’s face it: TV is inconvenient. Don’t you hate it when you’re rushing home to catch LOST and you spill your coffee all over your lap, scalding yourself in an area that was only meant to be treated nicely, while simultaneously ruining all possibility of hanky-panky with that cute (read: slightly geeky) girl (or guy)? That was painful.. never happened to me. Anyhoo, with the release of the PyAVC module, you Python developers out there can cross another fear off your list: the trade of LOST for loins! If you’ve got a Mac and a cable box with a Firewire (IEE1394) port you can roll your own simple DVR in Python.

A comcastic fantastic discovery.

As a resident of Portland (Oregon) my cable provider is Comcast.

No, it’s not Comcastic in any possible way, shape, or form. It never will be. Even the well-meaning gay man that initially coined the phrase “comcastic” while sipping his sixth appletini at “the club” is currently crying in a corner somewhere, wishing that — even through the haze of his Asprin/NyQuil-induced coma — he’d kept his mouth shut in that marketing meeting the next day.

Ignoring the disparities in service quality and plummeting level of perceived “comcasticness”, I was delighted to find that, upon moving to a new apartment, Comcast provided me with an awesome new Motorola DCH-3200 cable box — a huge improvement over any of their previous models. As I unpacked my new cable box, I took a glance across the back and discovered not only one, but two Firewire ports! I anxiously found a cable, plugged it into my Macbook Pro and… nothing happened. No compatible applications. The equivalent of technological blue balls. After a brief bout of sobbing on the phone with my bff Jill and the consumption of a gallon of ice cream on the couch while watching Sixteen Candles, I began to google.

I discovered that my box speaks AVC (Advanced Video Coding) and is actually streaming MPEG-2 TS-encapsulated 1080i video over that Firewire cable! I quickly set to work finding out how to use the protocol under OS X and was delighted to find that Apple provided example code in their Firewire SDK. I wrote an Objective-C framework that provides an object-oriented interface to the device (or a set of devices) and wrapped it in Python (thanks, pyobjc).

Get it. Got it? Good.

At this point, if you haven’t already downloaded PyAVC from the projects page, you should do so now. Alternately, you can download the latest version from http://cdn.blog.kenkeiter.com/packages/pyavc_latest.dmg. It’s tiny (much like my bff Jill) so you shouldn’t have long to wait.

Once you have it, you can go ahead and run the installer package that it contains. Please note that it will request an administrator password along the way. This is so it can do two things: install the necessary frameworks in /Library/Frameworks, and run the PyAVC setup.py script.

You said you were going to… fondle your sweaters.

If there’s anything that the movies have taught me (especially Wet Hot American Summer — an 80’s themed film set at a summer camp), it is that life is full of little misunderstandings. Some really weird, awkward shit… but mostly little misunderstandings. Therefore, I like to document everything carefully before someone gets the wrong impression.

The PyAVC module (imported as avc) is very simple to use. It contains three relevant classes: AVCCaptureInterface, AVCCaptureDevice, and AVCCaptureException. AVCCaptureInterface instances act as factories for AVCCaptureDevices. You begin by creating an AVCCaptureInterface instance, and accessing it’s devices property:

The resulting list is comprised of AVCCaptureDevice instances, upon which you can call the following methods:

AVCCaptureDevice
    |- tune_channel(self, channel, channel_minor = None)
    |- begin_recording(self, file_handle)
    |- finish_recording(self)

and access the following properties:

AVCCaptureDevice
    |- @property name
    |- @property vendor
    |- @property guid
    |- @property deviceState

If for some reason something breaks, an AVCCaptureException will be raised. If it’s my fault, it’ll raise an AVCCaptureException. If it’s your fault… you’re on your own.

For more detailed documentation, you can visit the PyAVC API page or run import avc; help(avc.AVCCaptureInterface) or import avc; help(avc.AVCCaptureDevice) in your Python interactive interpreter for more information.

So, let’s begin with a simple example: recording ten seconds of video from the first device on whatever channel the cable box is currently tuned to:

from time import sleep
from avc import AVCCaptureInterface

interface = AVCCaptureInterface() # create a new AVC interface
my_device = interface.devices[0] # select the first attached device

fh = open('/Users/yourusername/TestVideo.m2t', 'wb') # open the output file
my_device.begin_recording(fh) # begin recording to the output file

sleep(10) # wait for 10 seconds (begin_recording() is a non-blocking method)

my_device.finish_recording() # close it all up.
fh.close()

Oh my God, that was so freakin easy! You should now be able to play your newly-created MPEG2-TS file using something like VLC Player for OS X.

Since PyAVC acts as a wrapper for a Cocoa/Objective-C++ framework, I figured I’d tell you a little bit about how that all works, as well — feel free to skip over this (most people won’t care about this part). The framework (installed under /Libraries/Frameworks/AVCCapture.framework) uses Apple’s AVCVideoService framework to enumerate, connect to, and stream from AVC devices. The main class, CaptureController, is a singleton that handles detection of AVC devices, and wraps them in an AVCDeviceContainer instance so that they can be handled properly by Python/PyObjC. The AVCDeviceContainer abstracts away a ton of ugly code so that AVCDevices can be managed using simple methods such as - (void) beginRecordingWithFileHandle:(FILE *)theFH.

Kick it up a notch!

Ok, so ten-second video clips are getting boring and your channel-flipping finger is starting to weaken. Let’s add a little more flexibility and make it so that we can a) call this from the terminal, b) change the channel, and c) record for a set duration. We want our command line syntax to look something like this:

The -o (or --output) parameter should allow us to provide a destination to record to, the -c (or --channel) parameter should allow us to specify a channel (or leave the parameter out to record on the current channel) and the -d (or --duration) parameter should allow us to specify a duration to record for (in hh:mm:ss format).

Let’s get coding! First of all, we need a clean, simple way to parse input (paths, switches) from the command line. We’ll use the bundled optparse module because it is both flexible and fast. If you haven’t checked out optparse yet, I highly recommend it. We’ll also be using the sys, os, time, and avc modules. It’s like a module party — and you’re invited!

#!/usr/bin/python

import sys
import os
import time
import optparse
import avc

def parse_duration_to_seconds(d):
    """
    Parse a human-readable duration in the format hh:mm:ss
    to seconds. Accepts: hh:mm:ss, mm:ss, ss
    """
    particles = [int(p) for p in d.split(':')]
    particles.reverse()
    if len(particles) > 3:
        raise ValueError('Too many colons in duration.')
    return sum([p * pow(60, i) for i, p in enumerate(particles)])

def main():
    # parse command line options
    parser = optparse.OptionParser()
    parser.add_option('-o', '--output', dest="output_path", default="~/Desktop/video.m2t")
    parser.add_option('-c', '--channel', dest="channel", default=0, type="int")
    parser.add_option('-d', '--duration', dest="duration", default="00:01:00")
    options, remainder = parser.parse_args() # parse command line arguments

    try:
        # initialize the capture interface and select the first device
        interface = avc.AVCCaptureInterface()
        my_device = interface.devices[0]

        # if we defined a channel to record to (other than 0), tune to it
        if options.channel != 0:
            my_device.tune_channel(options.channel)

        # open an output file for recording and begin doing so
        fh = open(os.path.expanduser(options.output_path), 'wb')
        my_device.begin_recording(fh) # begin recording

        # loop while updating a status until we've recorded for the desired duration
        total_time = parse_duration_to_seconds(options.duration)
        elapsed_time = 0
        while 1:
            # we'll make ourselves a status line!
            sys.stdout.write('\rRecording Ch. %d - Elapsed: %ds, Remaining: %ds%s' % \
                (options.channel, elapsed_time, total_time - elapsed_time, ' ' * 8))
            sys.stdout.flush()
            time.sleep(1)
            elapsed_time += 1
            if(elapsed_time >= total_time):
                break

        # clean up
        fh.close()
        sys.stdout.write('\rRecording complete -> %s\n\n' % options.output_path)

    except avc.AVCCaptureException, e:
        print 'An error occurred:', e.message

if __name__ == '__main__':
    main() # if we're calling from the command line, then run it.

Now we have a pretty little script to start a recording! If you’ve got Unix’s ‘at’ command enabled you can now schedule recordings by running something like this:

It’s all a matter of expansion and contraction…

Following the theme of movie-based lessons, we learn from Out Cold’s flawlessly executed hot-tub/genital -separation that it’s all “a matter of expansion and contraction”. We also learn that the Eskimos have nine different words for “hell mooch stuck in a hot tub”.

As you use PyAVC, you’ll discover that the files it outputs are gigantic. Two hours of 1080i MPEG2 TS-encapsulated video is around 12 GB. Therefore, you’ll most likely want to transcode your video! The versatile VLC Media Player has wonderful transcoding facilities, and a helpful community. Assuming you have VLC Player installed, you can use a command like this:

to transcode your video into a more comfortable MPEG4 format.

That’s great. Give me a goddamn GUI.

I’ve been working on a FrontRow DVR appliance for this for a while and am almost satisfied with what I’ve come up with — however I’m not yet prepared to release it (needs to be refactored before I’ll put my name on it). In the meantime, you’ll be happy to know that functionality in PyAVC is implemented in a minimally-blocking way, and is natively threaded. This means that you can do any number of things including: using Django to build a guide-viewer + scheduler + kitchen sink, or using something like wxpython or PyObjC to build a Cocoa GUI.

Ideas and Resources for Improvement

To get you started with building a EPG (Electronic Program Guide) DVR, you should check out TMSDataDirect. They have an awesome SOAP API that you can access for a nominal fee (I think I paid around $20 for the year). The data is provided in XTVD format (XML based), and (of course) there’s a Python package called xdtv-tools to parse it.

Also, keep in mind that if you have multiple cable boxes you can record multiple streams simultaneously! Devices are accessed individually and the only limitation is Firewire bandwidth.

Disclaimers

I’m 99.99% sure that none of this will jack up your machine — that said, this is alpha software, so use it at your own risk. Also, if any part of this article was offensive to you, please bury your head in the sand and don’t visit my site. A lot of what I write is offensive, but it’s always meant in good fun! If you can’t understand that, you don’t belong here.

Special Thanks

The illustration for this article was created by the exceedingly dynamic Jeff Patterson. Look for a new illustration from a different artist for every major post.