Skip to content

The system I'm currently working on integrates with several external systems, over HTTP, using simple (RESTish) web-services. I really don't want to involve those external systems while testing my own, though; I want to stub 'em out.

My first attempt involved stubbing out HTTP calls using my mocking framework of choice. I'm using RestClient, which I like a lot, and stubbing out RestClient API calls worked quite well. It kept on working quite well for several hours, until I decided to refactor a little, using RestClient in a slightly differently way, at which point it broke completely. Bother. I really don't like having tests coupled to implementation details, so went searching for another way.

FakeWeb looked pretty good, in that it stubs things out at the Net::HTTP layer, which I'm unlikely to refactor out of the picture. In the end, though, it's not really what I wanted. I wanted to be able to do things like:

  • verify the body (and mime-type) of a POST/PUT request
  • dynamically generate responses, based on some aspect of the request (e.g. query parameters)

In short, I wanted a Fake Object, rather than a simple stub.

It occurred to me around about then that we already have plenty of tools for describing the behaviour of web-applications: they're called web-application frameworks! Many of them are too heavy-weight for my purposes, but Sinatra is nicely minimal. So, 60 lines of Ruby code later, I had a little web-app that mimicked one of those external web-services sufficiently for my testing. Win!

But waitaminut. I really don't want to have to start a separate process running my fake web-service, and talk to it using HTTP. That's going to be slow: network I/O isn't cheap. Isn't there some way I can use something like Sinatra but still keep everything in-process?

There is now. ShamRack plumbs Net::HTTP directly into applications built to run on Rack. Which includes all Sinatra apps, as well as Rails, Merb, etc.

Using ShamRack, I avoid the network traffic, making the tests a whole lot faster (about 25 times faster, in my case). Plus, I avoid the complication of having to start and stop an external web-server. Finally, because my fake web-service app is in-process, I get a handy back-channel I can use to setup or inspect it's state during tests.

If you find ShamRack handy, or have ideas about how it could improve, let me know!

I am not a UX professional, but I've interacted with a number of them over the last few years, in the context of agile development. The interaction hasn't always been smooth - the prevailing mood has been that agile development was asking UX professionals to do a bad job, but I've never felt that way myself. My feeling is that UX professionals and software developers have been on similar paths in regard to agile development, but the UX people are starting a few years later and have had less leadership so their challenge has been more difficult.

Why do I think the paths for both roles are similar? Because once-upon-a-time many software developers would have said that agile was asking them to do a bad job. After all, how could you do a good job with no upfront design, no long architecture phase, no detailed requirements? But what developers realised (either quickly or slowly, or perhaps not yet) was that agile values didn't mean doing a bad job, they meant doing a good job in a different way. Part of that was learning new skills and new ways to work. The software/programming community was lucky because they had early leadership in these skills and practices.

The feedback I've gotten in the past from UX folk was essentially "when we're in an agile project we can't work the way we work now", to which my answer was a resounding YES! You (as a profession) need to find new ways to work that produce good results faster. It's not your fault that those techniques don't exist yet, or that they may be hard to discover, but that doesn't invalidate the search. But I haven't done very well at explaining that.

However, practices for UX professionals on agile projects are emerging now, and Jeff Patton has done a good job of describing the practices that he sees working in this context. Moving to agile development means developing new approaches for everyone involved, and it's good that this is now a little easier for UX.
It occurred to me recently that I have this notion of programming as a process that involves breaking a problem down into a sets of smaller and smaller problems until I have something I know how to solve. (I mentioned this to Steve yesterday which reminded him of a joke about an engineer and a mathematician.)

I have previously just assumed that I therefore follow this process when I'm actually problem solving however, on reflection, I'n not so sure. More specifically, I'm either not doing it at all or, at the very least, I'm doing it intuitively.

I wonder how many people do (or have done) this as an explicit part of their own problem solving and if so, what effects they've noticed as a consequence.
I've heard the phrase "Excuse my poor code" (or words to that effect) a number of times recently. I've said it to myself about my own code (if not to others), I've heard two of my work colleagues say it, and I've had a customer's sole developer apologise ad nauseum about his code.

I'm not here to make any determination as to the veracity of these claims but the thing that I find interesting is that in all cases, the majority of the code in question was written by a single person.

Now this may be difficult for some to believe but I inherently don't trust my own opinion. I do tend to confidently put ideas forward as bold assertions to be shot down when I'm wrong. The confidence comes from knowing I'll defend my ideas to the hilt and not take it personally when my argument is proven flawed.

I hate working alone because I don't trust decisions that weren't arrived at through furious debate. I don't like developers working alone no matter how good they are (or think they are) because I don't trust they can be objective enough by themselves. In fact I don't care if they're working in pairs of developers or not, I just want their decisions to be scrutinised by smart people as early as possible.

The idea that the majority of a developer's work involves going off to think on their own is totally nonsensical to me. It feels like a lot of hocus pocus going on behind closed doors and then *poof* a few days later something is divined. Not only is the process opaque, but the possibility for smart people to scrutinise is left as late as possible. Sure, I like to sit and ponder without the hullabaloo of every man and his dog trying to give me their 2c worth as much as the next but the lack of transparency and scrutiny in this as a process is something I find very difficult to accept.

Smart people working alone is even worse. They're often implicitly promoted to the position of grand wizard, the seer and knower of all things. They go away, think about a problem and come back with much fanfare (trumpets playing, drums beating) to bestow upon the people their creation who will wonder in amazement at the design, so simple and yet, so complex, that only they can truly grasp the significance of what they have achieved.

When I write code on my own, I feel personally, individually, responsible. When I write code on my own, the pressures I feel are, mostly, self imposed. They make me lie awake at night. They make me code into the wee hours of the morning, on the train home. I worry that I've missed something. I start to believe my own hype. That I'm good enough to do this on my own. That because I did most of it on my own, I should fix it on my own lest someone else realise how crap my code is. In a vacuum my ideas have no predators. As importantly (if not more so), my priorities have no predators. I race towards a goal without stopping to re-think that goal because I'm stuck in a vicious cycle of self doubt and self confidence. Moreover, when I work alone, the production of code becomes the focus and not, as I believe it should be, thinking!

When we write code together we have shared code ownership. I feel like someone has my back. That even if we got the decision "wrong", we decided together and if two smart people can't get it right then maybe it's good enough, maybe there are diminishing returns for adding more people to solving the problem. When I go home at night, sure I might tinker but I know there's not much point to spending a lot of effort because I'll just be re-doing it tomorrow anyway. And if there is something we missed, someone has my back if need be.

I speak as one who has been, done and experienced others doing all of the above and decided once and for all that it's just not worth working alone. Having more than one person working on something WILL cost more but I assert that the result will be better and will be achieved more quickly and with less churn. Anyone who thinks they're smart enough to do otherwise needs to learn to keep it in their pants.

I have this mental model of software development that is akin to the way I've heard Abraham Lincoln's cabinet described. Apparently he got a whole bunch of really smart people together, even those from opposite sides of politics, and let them argue it out. His (Lincoln's) role was to decide when to stop the debate and which idea(s) to act on.

I think that's what collaborative development should be.

YMMV :)
David Allen is fond of saying that GTD is not a system, it's a systematic approach. Step 1 is the idea of systematically collecting or capturing whatever's on your mind and getting it into a trusted system in some kind of inbox for dealing with in Step 2, called Processing or Clarifying. The first myth I want to dispell is that you can have one inbox to rule them all. Would that it were true. I'll take you on a cook's tour of my current array of inboxes in my systematic approach to life and all its inputs, but be forewarned that you might be horrified at just how many there are! It's a non-trivial task to keep them all empty, let alone actually getting around to doing anything. The second myth is that you can computerise your system entirely. Sorry, that's not going to happen either, at least not in this decade. We'll see why soon enough.

Let's start with a few of the obvious ones.

Email

If you're like most geeks, you'll have multiple email addresses and multiple ways of accessing them. I won't go much into this here because it's well covered all over the interwebs. Suffice to say I'm a huge fan of the whole process of actually emptying my inboxes on a regular (every day or so works for me) basis. I'm also a huge fan of - gasp! - email-free days. I know this may be just about the weirdest thing most geeks can imagine, but I dare you to try it. I have empirical evidence that the world does not end even when I do not open anything remotely resembling an email client for 24 and even 48 hours. If you try it and don't achieve a Zen like state of focused productivity resulting in an immediate pay rise, please see the usher for a full refund.

Voice Mail

I've heard it said that twenty years ago only really important people had mobile phones but now only really important people don't. There's a Freud's worth of therapy research tied up in our addiction to the ubiquitous electronic erstwhile status symbol. All I'll bother to say here is that if you're an iPhone user, double-clicking the power button on the top sends the caller straight through to voice mail. Ah, that feels much better. Extra points if you don't even look at who's calling. Like David Allen often says about your mind, your phone is a fantastic servant, but a terrible master, so I encourage you to work out who's the boss of who.

Paper

Wasn't the world suposed to be paperless by now? While we're waiting, we of course need a systematic approach to handling the paper input in our lives. You probably have a letterbox on the street where your favourite magazine gets delivered. If you don't live by yourself, there's probably an informal ritual of grabbing the mail and distributing it (mine is unopened most of the time) to its various recipients. There'll be an equivalent process at the office, but for our purposes we'll assume that paper that's personally relevant to you gets collected somewhere.

I have two main paper inboxes (three if you count the entire surface of my desk at work). The first is one of those little plastic tray things at home that lives in a drawer because they can get pretty ugly. Here's what it looked like the other day:

My very kind wife will put my mail there as well as receipts for things she's helped me acquire, as in this example there are some tickets to a stage production of Lazytown at which my daughter will be dressed like so:

I also have a portable manila folder called "IN" that is always with me, or at least it's always in my backpack. It's done a lot of miles so it's a little the worse for wear as you can see:

This comes out of my backpack and onto my desk at work each morning and comes back home at night. If I take notes during a meeting in my (always present) paper notebook, I'll rip out the pages at the end of the meeting and put them in here. If someone hands me a business card, it goes in here. You get the idea. It gets emptied every few days.

Computer Files

I have a directory named "gtd" on my computer and one of the first things I do after my sexiest GUI on the planet launches is to open up a 1960's style shell in there and leave it open. That's a whole topic in itself that I'll leave for another day, but I'll just point out a few things about this very simple directory. On Mac OS, you can drag it to your sidebar like so:

This means that it will be available as a convenient place to save and load things from in every application. You can also drag it to your dock like so:


This gives you a convenient drop location for files from every app as well. Emails that are going to take some time to read go here in my system. Windows users can probably do something similar but I'll leave that as an exercise for those less fortunate.

By the way, one of the simplest and most commonly used commands in that shell is called "collect". This is simply for capturing a random thought that I'll need to process later. All it does is fire up my favourite editor and put a timestamped file in my gtd inbox. If you want it, it looks like this:

#!/bin/sh

# The collect command is used simply to collect thoughts and place them in the inbox
# The command creates a dated file in inbox and lets you edit it with whatever content you wish.

date=`date +%F-%H%M`
file="inbox/collected-$date"

$EDITOR "$file"

This won't work on Windows without some help from an operating system simulator like Cygwin, but you get the idea.

Now for some of the more geeky ones.

Voice Notes

I used to carry around a pocket sized notebook and pen for jotting down random thoughts while out and about. In the interests of having fewer things to put in my various pockets every time I leave the house, I've long since abandoned that in favour of taking voice notes on my phone. I currently use an app called Recorder on the iPhone and it's highly recommended. I keep it on the dock so it's always handy. When I get around to updating to iPhone OS 3.0 I'll take a look at the built in one but I'm not a day one kind of guy for that sort of thing. I used to use a Motorola Razr V3i and for anyone still using one, here's a tip: the built in voice notes app has some lame memory limit that means you can only take about ten voice notes. I got into the habit of using the video recorder for voice notes because it's only limited by the total memory on the device. I would still hold it to my ear like I was talking on the phone while recording a note, so you get to watch weird sideways and slightly slanted videos of random street scenes while draining your voice notes inbox. Try it and see if you can pick the location where you had the thought. Any non-geeks reading this should just go away now.

BTW, if you can type fast enough on a pokey little device to keep up with the stream of thoughts in your head, it's not your typing that's fast; it's your brain that's slow. Go the voice notes.

Receipts

If you're at all anally retentive financially, you'll collect little bits of paper while you're out on the town that you need to do something with later. There are two basic types of these in my world: ones that are mine and ones my boss is going to reimburse me for. These don't go into the aforementioned paper inbox like you might expect. These go into separate sections of my wallet like so:

This feeds into my weekly habit of balancing the books using the recently discontinued Microsoft Money. Perhaps the fact that I'm still using MS Money 99 has something to do with them being unable to make money at that game. Sometimes software just doesn't need any more features and all us geeks could maybe learn something from that. I'll leave that topic for another day because I'm trying to limit this to the "collect" phase of GTD and it's long enough already.

So called GTD systems

Whoulda thunk we'd have gotten this far without talking about Things, Omnifocus, RunwayorRemember the Milk? Isn't "geeking things done" about using a new and cooler GTD software app each week and re-dumping your entire life into a completely new system as a way of endlessly procrastinating on the things you're in denial about needing to actually do? That certainly is half the fun, so we'd better take a look at them, at least from a collection standpoint for now. Religious wars on the merits of these apps bore me and I make no assertion that I know them all equally well, so if your favourite app is not mentioned here, I believe there are still 2^lots-1 domain names under which you can start yet another GTD blog and talk about it to your heart's content.

The reason we've made it this far without talking about so-called GTD software apps is that life is just more complex than any one of them can cover. It might not always be this way, but right now it is for me. Either the technology will have to get better or my life will have to get simpler for one of these apps to be all I need. That being said, I can't live without one of them. My two current favourites are OmniFocus and Runway, so I'll describe their collection mechanisms here. BTW, my experience with OmniFocus on the ipHone is that it is just too darn slow to start up to be useful at all.

OmniFocus has a ubiquitous capture input box that looks like this:

You can hook it up to a custom keystroke but it only works if the app is actually running. The cool thing about it is that if the thought you have is well formed into an action, you can fill out the project and context, etc, and have a new action in your system in seconds. If not, just type a name for it and it will end up in OmniFocus' own Inbox for later processing.

Runway is a web app, so it's a little different. Thankfully, there's Twitter integration, so you can use Tweetie (you do use Tweetie, right?) as an equivalent to the OmniFocus capture box to create an action using the Runway super-quick action syntax like so:

Setting up Twitter integration is an extra step for Runway users described here. Runway also supports the idea of forwarding emails to it with the subject being parsed in the Runway action syntax, so that gives you another generic capture tool too.

What's the point?

In geek speak, what inbox philosophy comes down to for me is asynchronous batch processing and *not* synchronous event handling. Don't leap to respond to emails as soon as they arrive; batch them up until you get at least a dozen or so and can use a ten minute break from whatever you're doing to drain them. Same for phone calls; make a conscious decision about answering the phone, not a knee jerk (or elbow jerk) reaction to pick up the call. Multitasking is just plain evil, especially for geeks because a lot of the context switching we have to do involves large and complex mental models that are held together in our minds in a remarkably fragile and transient way. The cost of reassembling them after an off topic phone call or email, even assuming you *can* reassemble them, is significant. You're wasting your boss' money if you keep letting your mental model collapse and have to keep rebuilding it just because a trivial email arrived. Think about it that way and you might find any guilt you feel about your less than optimal responsiveness goes away. Just as in software design, sometimes scalability is more iportant than responsiveness :)

If you've made it this far, you really should get back to work. Good luck with getting yourself set up to never lose an important thought again without wasting that brain the size of a planet on trivia. Get it out of your head asap so you can focus on something more worthy of your wetware.

In reponse to my previous blog post Robert said...


'I've always wondered if the classic small-law firm model would work with IT shops. You know - newcomers start as "associates" and get paid a salary (possibly with bonuses), but the ultimate goal is to get recognised as a partner, in which case you shift to profit sharing.

The associate-level allows the firm to recruit new talent at a fixed risk; the partner-level allows for the co-operative mode. As long as the ratio of partner-associate is good (the case in small law firms, not large) the competition between associates isn't likely to be unhealthy - particular if the promotion to partner is merit based rather than "we've got 1 slot open this year".'



It turns out that we have some of these elements at Cogent, but we didn't have that model in mind when we created them.

We do have two groups for allocating profit share. One is like "partners" - it's myself, Marty and Simon. 50% of the profit pool goes to the partners and the remaining 50% is split evenly over all the employees. We needed that arrangement to make it sensible for the partners to stop doing personal consulting and switch to a salary-plus-profit-share arrangement, and even then we're only just now at the point where the partners would nominally break even. It's also rather abstract because as long as we're pushing consulting profits into product development there aren't any cash profits to share anyway!

So far this is very similar to the small-law firm model. The difference comes when we consider what will happen over time. It seems to me that the small-law firm model is predicated on growth, or at least turn-over - on the expectation that when you get promoted to partner there will be a supply of associates to generate profits. I'm biased against models that encourage growth (though our existing model has that bias at the moment). Over time at Cogent I'm actually expecting to reduce the partners' percentage of the profit share pool - keeping it enough to keep them interested in staying at Cogent, but gradually making more and more of the profits available to the general employees.

There are flaws in both approaches, so it will be interesting to see how the profit share approach evolves at Cogent over time.
At Cogent Consulting we compensate people in two ways - we have healthy base salaries and we offer profit share that is not linked to performance or to salary. It's close to equal shares (there are some details that mean that's not true overall). We debated this arrangement when we created Cogent and I've discussed it with colleagues back as far as 2001, but it's based on our gut feel of what would minimise conflict and maximise co-operation. It avoids the debate about relative performance that bonus systems create, that are normally circumvented by secrecy (secrecy isn't an option at Cogent). So it was interesting to read this in No Contest: The Case Against Competition by Alfie Kohn:



"In recent years, Deutsch and his associates have investigated not only the way tasks are set up but the way rewards are distributed. Among the possibilities are a winner-take-all system (which is what many contests amount to), a distribution proportional to accomplishment, and an equal distribution. Much as we tend to assume that competing boosts performance, so it if often taken for granted that the first two arrangements provide a crucial incentive for working hard: reserving a desirable reward for the winner is thought to promote excellence. A series of six experiments with Columbia University students, involving tasks that ranged from decoding Japanese poetry to estimating the number of jellybeans in a jar, was devised to test this assumption. The results: When tasks could be performed independently - that is, when there was low means interdependence - the system of distributing rewards had no effect on how good a job they did. There was absolutely no evidence to suggest that people work more productively when rewards are tied to performance than when everyone gets the same reward. But for those tasks where success depends on working together, there was a clear difference. A system of equal rewards, Deutsch discovered, "gives the best results and the competitive winner-take-all system gives the poorest results".



Notice that this is also a profit-sharing scheme, not a bonus system tied to some particular behaviours. We hope that our behaviours are driven by our principles, and that long-term this will generate profits. We wouldn't want people to change their behaviours to maximise single year's profit. If we ever detected that we'd need to review the profit-sharing scheme.
Cogent has made a couple of steps forward this week. We'll tell you what those steps are in a minute, but first a little bit of background. Cogent Consulting really started two years ago when Marty and Steve merged their consulting revenue into one pool - now we have 11 employees, and we're going to come close to $2m in sales for this financial year. We generate enough margin to have two people on product development, we've released Runway, and we're about to start on another product venture. We still keep all our financials open and we operate reasonably democratically. Things are tracking fairly well.

So here are our two big steps. Our 11th employee was a business development manager, Rachel Morley. Having a business development manager increases our overheads, but it also gives us a more active, structured approach to business development that lets us (and indeed encourages us to) support more employees. As a result, we're looking for people who would like to become Cogent employees.

Cogent isn't your typical employer. We're interested in the usual things - people who have deep technical knowledge, people who can represent us well on client sites, people who like to learn. You've heard all that before. We're quantitatively different in that regard, in that our standards are very high, but there are some things that are qualitatively different about Cogent. If you're looking for someone to tell you what to do, in detail, then we're not the place for you. We're much more likely to give you some rather vague goal and ask you to figure out how to achieve it - and a lot of the things you need to do won't involve programming. They may involve marketing, or copy writing, or business analysis, or something as mundane as going and buying some stationery.

We truly need you to be collaborative and collegial - we don't have any places for people who want to work alone or who don't want to subject their programming thoughts to critique. Bob Sutton's "The No Asshole Rule" captures our philosophy quite well. In exchange we'll be completely transparent to you - you'll know our clients, our rates, our revenue, our costs, and everyone's salaries. You'll have the chance to be involved in decisions that you probably don't even know happen at your current employer.

We need you to be passionate. While we pay good salaries, we need you to be the sort of person who would program for enjoyment even if you weren't getting paid for it. We need you to be highly ethical - to be more concerned with doing the right thing than the profitable thing if there's a conflict.

We're interested in programmers, web developers, business analysts and possibly testers. So if you're interested in undertaking a variety of consulting work, off-site development, and product development at a place like Cogent, please drop us an email.

Updated June 18



Along the Cogent Consulting journey we've documented a few different sets of values, principles and goals. Our manifesto was created quite early. Recently we've supplemented this with a simple list of values




  • Openness

  • Integrity

  • Respect

  • Craftsmanship

  • Collaboration



and goals


  • Develop non-consulting sources of income

  • Build some kick-ass software

  • Work with good people

  • Have flexibility in lifestyle and the amount of time I want to work



We should probably rationalise all this (and I just added that task to Runway), but first I want to add some more philosophical statements that occurred to me after I read Small Giants by Bo Burlingham.


  • We need to make a profit, otherwise we can't achieve our other goals. That's a constraint

  • Doing the right thing for the customer is more important than maximising profit

  • Our families are more important than our customers

  • For everything we do we should ask not 'is it profitable', but 'is it the right thing to do?'.

  • Right now, product development depends on consulting, and consulting depends on business development. We need to bear that in mind when we're setting priorities.

When I started to write I thought there would be more points, but combined with our existing manifesto that seems to say most of what I need to say.
Or at least until all those Twitter client developers have fixed their Twitpocalypse bugs. In case you didn't know, a few days ago the ID range used for Twitter's messages exceeded 2^31 (approximately 2 billion) causing any apps that stored them as 32-bit integers to think they were really small negative numbers.

It's usual--and I say usual because I don't always adhere to it--policy for storing external identifiers is to treat them as text, even when I know they are numbers. Why? Essentially because I consider it a coincidence that they're numbers. That identifier, number though it may be, has no special significance to me over and above being an opaque handle to some entity in another system. As such, I like to treat them as text.

Discussing this with a good friend and colleague of mine, the question of column width came up. Ie. if you're going to make it text, how long should the column be? If you're lucky enough to be using a database such as PostgreSQL, then the answer is: it doesn't matter--there's no performance benefit to artificially limiting the size of the column. For other databases, the common practice is to use something like VARCHAR(255). Think about it, even if it is a number that's 10^255!

Twitter claims that its API is RESTful. And if to you, REST means nice, predictable URLs with some semantic path possibly followed by a numeric id and returning numeric ids in search results, then yes, it's RESTful. Want to see the most recent messages for a user? There's a simple HTTP request you can make to a nice, semantic (if you speak English) URL that returns a list of them and their identifiers. And, as expected, our Twitter clients have been dutifully squirrelling these ids away in integer fields (probably because that's the default) and all has well until 2 days ago.

Now, without going into too much of an ideological rant, I happen subscribe to the principle that RESTful URLs should be opaque. That is, a URL is a URL is a URL. No slicing, no dicing, no assembling, no joining. If I have a URL to a resource then that's what I use. Period. End of story. (You can find plenty of discussion on this by Roy Fielding using Google.)

So, back to our column widths. Assuming we have a text field in our database large enough to accommodate a URL, we could go one step further. Rather than treat the identifier as text and storing that, why not go the whole hog and store the URL instead?

As far as I can tell, the only reason is that Twitter's API, RESTful though they may claim, sends back numeric identifiers rather than URLs which in turn leads developers to incorrectly assume that they should be storing them as numbers.

On their own, identifiers are meaningless and in fact, useless. To utilise an identifier requires us to know the system in which it is stored and the collection in which it belongs. If instead each piece of information was identified by a URL we get all that context for free and the power to share information grows phenomenally.

To me, the beauty and power of the internet is the ability to link together disparate systems in ways no one had previously imagined. More specifically, in ways the publishers of the information never considered.

Opaque URLs combined with idiomatic use of HTTP verbs can help reduce the coupling between producers and consumers by giving back control to producers in how and where they store information and at the same time increasing the freedom for others to share and use that information.

(That last paragraph reads like an Amnesty International commercial!)