Red Squirrel Reflections
Dave Hoover explores the psychology of software development
Thu, 04 Dec 2008
In late 2006 and early 2007 I was doing a lot of "selling" for Obtiva. I was trying to win a project big enough to allow me to roll off my current on-site consulting gig. I wanted to start a new practice at Obtiva I called "The Studio". This practice would allow us to work from our office in Wheaton, Illinois, USA and hopefully allow us to create an environment conducive to apprenticeship. I had already found some Rails projects for us to work on. One of these projects allowed us to hire Ryan Platte, Obtiva's fifth full-timer. The rest were done in our off-hours or with sub-contractors, none of them were big enough to justify me leaving a steady, long-term gig a few miles from home. And then in March 2007, I was contacted by Gary Levitt about a project called Mad Mimi:
Hey Dave, I'm a bass player, came to NY from South Africa to contribute to the Paul Simon Group. I set up a music company and am building a hip application for musicians.
He went on to say that he had all the "groundwork" for the project completed, meaning the requirements were ready to be executed over the course of a few months. Anyone who has ever been involved with a software project for more than a few months would find that claim pretty funny, but it was great to hear that Gary had already started laying the "groundwork" (coached by Jeff Patton) and had a delivery date in mind. I was thrilled to receive that email from Gary, and ecstatic when a week later I ended up winning the project despite stiff competition against some people and companies I have a ton of respect for.
I rolled off my on-site gig a month later and showed up in "The Studio", a low-budget, 3 room office about a mile from my home in Wheaton, IL, USA, that smelled a bit like the dentist office down the hall (which meant that kids would occaisionaly run down the hallway screaming as they tried to escape their impending doom). There I was, all by myself, with a new client in New York and some fixed-price, fixed-scope 1 week milestones to churn through. And churn we did. In those first weeks Mad Mimi morphed her identity more than once and a few weeks later we had something we could play with. Mad Mimi was all about creating Promotions, making it easy for people to compose beautiful emails in as few steps as possible. Unfortunately I'm not good (yet) at making things visually beautiful, and so Gary (yes, the customer) nominated himself to become the CSS ninja, and over the course of the coming months, Gary learned how to navigate a Rails code base, run the app locally, use Subversion, and develop beautiful CSS designs. Eventually he even learned how to use Capistrano and deploy the app to the server, but I'm getting ahead of myself.
Not surprisingly, it took longer than a few months to create the product that Gary wanted, partly because Gary discovered what he wanted along the way, and partly because I was still learning how to do what Gary wanted. I had to implement Ajax image uploads along with image drag-and-drop to allow people to have a WYSIWYG-ish experience creating their Promotions. I had to pick and integrate a Rails-friendly background task processor (circa mid-2007) to allow users to pause, cancel, or observe the progress of their Promotion mailings. My drag-and-drop code eventually got rewritten by Tobie Langel when he joined the project in early 2008. The background task processor that I chose, BackgrounDRb 0.2.1, seemed like the only viable option at the time, so I chose it and we're still using it (though I wouldn't say we're raving fans). There was also the issue of dealing with large (to me) data sets triggered at any moment by a user action, such as a user uploading 50,000 contacts or sending a Promotion to 70,000 people. I had some previous (bad) experiences with handling lots of data with ActiveRecord, so at least I knew what not to do, and I got something working with data sets of less than 100,000 with a mostly empty database.
Along the way the Studio practice was growing. I quickly brought Brian Tatnall on board as our first apprentice in May 2007 to help with Mimi and some smaller Rails projects. Brian had worked for a small design shop previously and his big client followed him to Obtiva, so instead of helping me reduce my workload, he actually increased the Studio's workload by bringing a huge (for the Studio) ecommerce project with him. By the end of 2007, the Studio team had grown to 5 people, 3 of whom were apprentices, and Brian had rolled out of the Studio onto an on-site, hedge fund client.
Mimi's development gradually slowed over time due to my growing Studio responsibilities, Gary's burn rate, and waiting for Tobie's brilliant re-write of Mad Mimi's Promotion Composer. We silently launched MadMimi.com in February 2008, where it lay dormant for several months as Gary tweaked it and allowed a small number of initial customers to try it out and put it through its paces. At this point we were still on the same low-end RailsMachine server that the project had started with. As the initial customers started working with more and more data, and sending more and more emails, we found that running 6 Mongrels and BackgrounDRb and MySQL all on the same little server wasn't working. We worked with the guys at RailsMachine to split the web front-end from the BackgrounDRb back-end onto 2 low-end servers. Once we started getting significantly more load, we went to one of their higher-end "8 core" accounts. I must say I am a very happy RailsMachine customer, and we have just finished migrating our large e-commerce client onto RailsMachine. RailsMachine's people are incredibly responsive and have a knack for teaching you how to fish rather than just handing you the fish (they've taught me a lot about system administration).
Gary eventually started promoting Mad Mimi in May 2008. This was were the real fun began. Working with a customer to create something that only exists in their imagination can be an exciting process, particularly when you have the rapid feedback that Rails provides. But watching your code go out into the real-world and begin to generate revenue for your customer and feel the load of real-world data is not something I've been so close to before. Early on in May or June, I wrote a Capistrano task that would spit out Mimi's monthly revenue from the command line. By this point we had trained Gary to use Capistrano, so he was able to watch these numbers closely (as did we). I remember the week that Mimi started generating enough revenue to pay for her server costs ($300 per month). That must have been in June. Then Mimi was generating $1000 per month in July and we got excited. Gary wasn't paying for any advertising, just blogging and getting mentioned on Ajaxian and 37Signals Product Blog. And people were loving Mimi. Gary had created a certain vibe around the product, from the look-and-feel of the site to the Campfire chat support that Gary and his brother Dean expertly manage. Mimi's revenue was growing steadily and I was falling in love with the subscription-based web service model popularized by Basecamp.
Not surprisingly, Mad Mimi's traffic was growing too. We handled our first 100,000 person mailing with a couple hiccups. I needed to do some database optimizations to help with some non-performant scenarios and we needed to increase the speed of the mailings. We had a BackgrounDRb worker responsible for each mailing and it would simply run through the email list, sending one at a time. For our bigger mailings, this was taking too long, so I split up the big mailings to be handled by multiple concurrent workers, and worked on some caching strategies to improve our mailing speed. Our fastest mailings now hover around 20 emails per second. We're going to need to improve that even more, but for now it's sufficient. Gary learned all about how to get emails to render correctly in all the different web clients. Brian and I wrote TamTam so that Gary could create HTML emails with CSS and not have to use inline styles. Colin and I learned about DomainKeys and got those working to keep some of the pickier email providers happy. And by now we've sent out mailings as big as 350,000 and have sent several million emails within a 48 hour period. I've had to write my first couple MySQL stored procedures in order to optimize certain SQL queries and imports, with some guidance from a friend of mine. Needless to say, I've learned a lot in the process.
I presented my experiences with Mad Mimi at Windy City Rails in September 2008. At that point Mimi had just over 150 paying customers generating just over $5000 per month in revenue. I predicted that Mimi would hit $10,000 per month by December. And I'm excited to announce that this prediction was accurate. Mimi now has just over 300 paying customers generating over $10,000 per month. It has been an awesome experience watching Mad Mimi come from Gary's vision to a viable product to a contender against the established email marketing platforms. After seeing the way Gary learned CSS and was adamant about the product's personality, Mimi has forever changed the way I see my customers. After experiencing success with rapid feedback and leveraging Rails to iterate quickly with Gary, it has forever changed the way I approach my projects. And after seeing the simplicity of Mimi's revenue model and watching Gary turn away one buyout offer after another, it has changed the way I think about start-ups.
Nowadays, the Studio is 10 people and still growing. We've split into two teams and will be splitting into a third in the next month or two. I'm proud of the Studio and how much we've improved it since the chaotic, early days when we were chronically over-sold and our first few apprentices didn't get enough mentoring. We've had a lot of trial-and-error, but we've slowly, steadily improved our practices, discipline, and processes over the last 20 months.