jtolds.com - Programming

Pubsubhubbub and Superfeedr

JT Olds — Thu, 9 Jul 2009 20:53:30 +0000

Occasionally there is a huge hole on the internet that needs to be filled. One such hole is a core part of how feeds on the internet work. Feeds are currently Polling-based (though I suppose Pull-based is a similar enough name). It's the same problem that Twitter currently has. Basically all clients have to continually sit and ask if there's any new information, instead of being notified. Notification delivery is how it should work.

Two developments today will help fill this hole, and instead of just sharing them on my Google Reader Shared Items feed, I thought I would make an explicit newsletter entry.

First, PubSubHubbub. This holds the promise to fix, for the long term, how feeds work. I'm about as excited about this as I am for Google Wave.

More immediately, though, is Superfeedr. I was just notified via Friendfeed of this new startup that does exactly what I've been hoping someone would do for a while now. So, yes! Now my stealth SnewsFlash project will be way easier. Thanks, Superfeedr!

Update: Well, I'll be. Gnip, too. Yay!

TextMarks

JT Olds — Mon, 25 Aug 2008 05:42:46 +0000

Thank heavens. TextMarks is the developer service I have been looking for. I am so glad someone made this. I figured it was only a matter of time.

I've recently decided that it's too bad SMS short codes are so prohibitively expensive for small-time developers to SMS-enable their (potentially infrequently used) applications. TextMarks solves this by essentially multiplexing SMS services over one short code, requiring users of each service to prefix their messages with a "textmark" service identifier, thereby cutting costs for everyone. TextMarks pays for the one short code and covers the service fees, allowing small-time developers to use SMS as an application interface. Hooray!

Can you make (10, 10, 9, 9, 1, +, /, -, *) = 4?

JT Olds — Sun, 24 Aug 2008 22:35:48 +0000

Last night at around 10pm I got a text from my friend asking how you could combine the numbers 10, 10, 9, 9, and 1, to make 4, using only addition, subtraction, multiplication, and/or division, provided you use the numbers once each.

I didn't start thinking about it until later (sorry Mike), but at 2 am I got frustrated and, as I seem wont to do, ignored reasonable priorities (such as sleeping, or the rest of my todo list) and sat down to write a program to enumerate all of the possible solutions, instead of just figuring it out myself.

Below is a general solution to the problem. The basic idea is it is simple enough to consider all the valid possible combinations of the numbers in Reverse Polish Notation (postfix), and evaluate them.

#!/usr/bin/python2.5





def add(x, y): return x + y


def sub(x, y): return x - y


def mult(x, y): return x * y


def div(x, y): return float(x) / y


OPERATIONS = [add, sub, mult, div]


REPRESENTATIONS = {add: '+', sub: '-', mult: '*', div: '/'}





def new_counter(counter, item):


  if counter is None: counter = 0


  if type(item) == type(add): return counter - 1


  else: return counter + 1





def valid_perms(lst, counter=None):


  if counter is not None and counter <= 0: return


  if len(lst) == 1:


    if new_counter(counter, lst[0]) != 1: return


    yield lst


  else:


    for i in xrange(len(lst)):


      for perm in valid_perms(lst[:i]+lst[i+1:], new_counter(counter, lst[i])):


        yield [lst[i]] + perm





def all_combos(lst, n):


  if n == 0: yield []


  else:


    for i in xrange(len(lst)):


      for combo in all_combos(lst, n - 1):


        yield [lst[i]] + combo





def uniq(generator):


  seen_things=set([])


  for thing in generator:


    if tuple(thing) not in seen_things:


      seen_things.add(tuple(thing))


      yield thing





def valid_postfix_stacks(nums):


  for op_combo in uniq(all_combos(OPERATIONS, len(nums) - 1)):


    for perm in uniq(valid_perms(nums + op_combo)):


      yield perm





def compute(stack):


  s = []


  for item in stack:


    if type(item) != type(add):


      s.append(item)


    else:


      s.append(item(*reversed([s.pop(), s.pop()])))


  assert len(s) == 1


  return s[0]





def infix(stack):


  s = []


  for item in stack:


    if type(item) != type(add):


      s.append(str(item))


    else:


      s.append('(' + ' '.join(reversed([s.pop(), REPRESENTATIONS[item], s.pop()])) + ')')


  assert len(s) == 1


  return s[0]





def main(nums, answer, tol=.0001, find_all=False):


  print "searching..."


  found = False


  for postfix_stack in uniq(valid_postfix_stacks(nums)):


    try: val = compute(postfix_stack)


    except ZeroDivisionError, e: continue


    if (float(val) + float(tol)/2 >= float(answer) and float(val) - float(tol)/2 <= float(answer)):


      print infix(postfix_stack), '=', val


      found = True


      if not find_all: break


  if not found: print "no solutions found"


 


if __name__ == "__main__":


  main([10,10,9,9,1], 4)

How to get V4L2 devices to work with Flash

JT Olds — Sun, 27 Jul 2008 23:14:06 +0000

Background

Recently, I wanted to get a webcam I have working with TokBox. The allure of TokBox is great, since it's videochatting with no downloadable software, as long as you have Flash set up correctly. Unfortunately, however, Flash currently supports Video4Linux version 1 devices, and most new webcam drivers for Linux are Video4Linux version 2.

As of this writing, Flash 10 Beta 2 is the most recent release of Adobe Flash, and it now has support for V4L2 devices! However, not many devices (help out here), and if you're not the sort of person that wants to uninstall your comforting Flash Player deb package for some beta tarball, Flash 10 Beta 2 isn't actually the solution yet.

So, I started looking for another way. Well, Gstreamer supports my webcam, and most other V4L2 devices, so if there was a way of converting a Gstreamer pipeline to a V4L (v1) device, then I'd be set. Turns out, there is! gstfakevideo was written so that Skype users could get better webcam support in Linux.

So now, if you want to get your Gstreamer-supported V4L2 video device to work with Flash (even pre-V4L2 supporting builds of Flash), follow these instructions. I admit they're a bit high-level.

Implementation

First, grab gstfakevideo. I did everything with the repository's revision 3 (the latest at the time of this writing), so if the gstfakevideo people change something, these instructions will work with codetree version 3.

Stupidly, gstfakevideo is hardcoded to intercept any attempts to grab /dev/video0. You'll probably want to change that so applications can see all of your video devices, including the fake one we're about to make. I plan on submitting a patch to the gstfakevideo people soon so this is configurable, but haven't yet.

To fix this for now, you'll need to edit both the gstfakevideo shell script and gstfakevideo.c source before compilation. Just pick an unused video device name and change all instances in both files of /dev/video0 to your new video device. I just used /dev/video1. Then compile.

Next, find in the gstfakevideo script where it says "exec skype". Change that to exec firefox, or whatever webbrowser you use. Make sure you close your running webbrowser instances. Then, on your command line, try "./gstfakevideo v4l2src ! ffmpegcolorspace ! videoscale". All we're doing here is setting up a Gstreamer pipeline that the gstfakevideo library will convert to a V4L device. This should launch your webbrowser. Navigate to a website that has Flash and right click the Flash applet. Go to settings, and select the webcam tab. Change the input device to the gstfakevideo device and click the webcam icon box to see if it works. If it does, you're set.

If it doesn't work, there's a chance that figuring out a different Gstreamer pipeline would work better (the pipeline is the "v4l2src ! ffmpegcolorspace ! videoscale" part). If you can get the pipeline "videotestsrc is-live=true ! ffmpegcolorspace ! videoscale" to work, then Gstreamer is probably not the problem. Play around with pipelines (help elsewhere). You can test pipelines by using gst-launch (gst-launch-0.10 on Ubuntu Hardy). You'll need to add ! xvimagesink at the end of any pipeline given to gst-launch.

Once you have a working system, you can make the whole process a little easier to start up the next time around by replacing the line that says 'GST_PIPE="$*"' in the gstfakevideo script with 'GST_PIPE="v4l2src ! ffmpegcolorspace ! videoscale"' (or whatever pipeline you figured out), moving the libgstfakevideo.so file to /usr/local/lib, and renaming gstfakevideo to a name that makes more sense, like firefox-video. Then, move the script to somewhere more useful like ~/bin.
Every time you run that script, you'll start your browser with Flash support for your V4L2 Gstreamer default input device (configurable with gstreamer-properties). Since all you need is your compiled libgstfakevideo.so library and the script you just moved, you can delete the source directory.

Hooray!

Mox

JT Olds — Thu, 17 Jul 2008 00:06:45 +0000

In working on Viricide, I spent a good deal of time writing unit tests for the server portion. The majority of my previous Python unit testing experience being at Google, I looked around on the internet for potentially similar solutions to write my unit tests in.

I was surprised to discover that mock-object unit test frameworks for Python essentially boiled down to the following three solutions:

The Python Mock Module (http://python-mock.sourceforge.net/)
pMock (http://pmock.sourceforge.net/)
Python Mocker (http://labix.org/mocker)

I didn't like any of the solutions, really, as they all seemed to have various glaring problems or inconveniences (the specifics of which I have forgotten), but I settled on Mocker as it was closest to the Google mock-object system, right after sending an email to a friend at Google requesting that they open source what I was used to.

It appears that my friend totally pulled through, sent an email to the right people, and Mox was released on Google Code in mid-June.

Though I first noticed that Mox was available for download the day Protocol Buffers were released on Google Code (Mox is included in the Protocol Buffer source distribution as it is required for unit tests), the Google blog is now announcing Mox for general availability.

I guess I'm not surprised the announcement references other former Googlers requesting it, as it's a pretty sweet mock-object system. But hooray!

Also, do check out Protocol Buffers. I'm pretty excited about them, too.

My OLPC software is actually useful!

JT Olds — Fri, 9 May 2008 21:16:35 +0000

In February, I received an OLPC laptop as part of their Give1Get1 program. One of the things it was missing was a way of launching native Linux applications and commands without opening a terminal, so I wrote a simple OLPC Activity that provided this ability to some extent. The project page is here.

Early this week I found out that my little project is of some use to the Teaching Matters non-profit, and they are now rolling out my app on XO laptops in NYC.

Here's the blog post where I'm mentioned.

Holy crap: Google App Engine!

JT Olds — Tue, 8 Apr 2008 08:43:50 +0000

Update: whoa, check out one of the first demo apps: HuddleChat
Update 2: as usual, Google does a better job of explaining their own products than I do.

Probably unsurprisingly, there are a number of unlaunched Google projects I learned about while I worked there last summer. Unfortunately, I've had to keep my lips completely sealed due to the NDA I signed. So, I've had some amount of quiet and unshared (at least, outside of Google) anticipation for a number of different things I happen to know Google has been working on.

Few of these things, however, even come close to matching my anticipation for the service that was finally released today.

Even though I knew Google planned to release a web-services platform, I didn't fully understand the details. I was sort of expecting a me-too competitor to Amazon Web Services, and figured Amazon had simply beaten Google to the punch here. Hey, it could happen.

However, with the public release of Google App Engine, I am again blown away by the details with which Google nailed this product.

TechCrunch just covered the launch, and although their initial article simply reported the news, I expected more thought and speculation on how much this is a game-changer. Every crappy part of AWS is fixed. There are some limitations that Google App Engine has that AWS doesn't (only Python is supported, say), but these are silly when compared to the limitations of AWS*:

cost - App Engine is free under certain very reasonable limits, AWS is not.
elasticity - load balancing and replication are automatic with App Engine, not AWS
simplicity - the only work involved with App Engine is writing some Python code; you don't have to manage your own virtual machine images

* I haven't actually used AWS, and I've only used internal Google technologies (basically App Engine, but not exactly), so this feature comparison is based on my loose reading of news clippings.

Here's what I see happening:

The barrier to entry for scalable website design is totally lowered. Tons of developers see this and go "ooh, neat, I can write that personal project pretty easily now" and do so.
Some of these projects become financially viable from Google Advertising, but where the costs would usually go straight into maintenance for the growing user demand, Google manages it all very cheaply and efficiently. Instead of developers having to choose between killing their project due to too much demand or quitting their day job just to support it, Google eases the maintenance of the site considerably. Futhermore, the cost-effectiveness of running your own service is now much better due to the low over-head costs. Developers actually could quit their day jobs.
Many new, innovative services start springing up like wildfire, and don't be surprised to find that they're all only one or two man shops.

I've already been convinced that Android is going to completely revolutionize the cellphone market, and I feel the same way about Google App Engine. Even though AWS really started this product space, App Engine seems like enough done right that this will open the floodgates on cloud application hosting. I do not expect Google App Engine to simply be a footnote in the history of the internet.

If you're interested in developing for App Engine, make sure you sign up soon, as there's only a limited number of trial accounts during the preview release.

We could totally rewrite Chipmark on Google App Engine. That would be sweet.