Red Squirrel Reflections
Dave Hoover explores the psychology of software development
Thu, 04 Dec 2008
In late 2006 and early 2007 I was doing a lot of "selling" for Obtiva. I was trying to win a project big enough to allow me to roll off my current on-site consulting gig. I wanted to start a new practice at Obtiva I called "The Studio". This practice would allow us to work from our office in Wheaton, Illinois, USA and hopefully allow us to create an environment conducive to apprenticeship. I had already found some Rails projects for us to work on. One of these projects allowed us to hire Ryan Platte, Obtiva's fifth full-timer. The rest were done in our off-hours or with sub-contractors, none of them were big enough to justify me leaving a steady, long-term gig a few miles from home. And then in March 2007, I was contacted by Gary Levitt about a project called Mad Mimi:
Hey Dave, I'm a bass player, came to NY from South Africa to contribute to the Paul Simon Group. I set up a music company and am building a hip application for musicians.
He went on to say that he had all the "groundwork" for the project completed, meaning the requirements were ready to be executed over the course of a few months. Anyone who has ever been involved with a software project for more than a few months would find that claim pretty funny, but it was great to hear that Gary had already started laying the "groundwork" (coached by Jeff Patton) and had a delivery date in mind. I was thrilled to receive that email from Gary, and ecstatic when a week later I ended up winning the project despite stiff competition against some people and companies I have a ton of respect for.
I rolled off my on-site gig a month later and showed up in "The Studio", a low-budget, 3 room office about a mile from my home in Wheaton, IL, USA, that smelled a bit like the dentist office down the hall (which meant that kids would occaisionaly run down the hallway screaming as they tried to escape their impending doom). There I was, all by myself, with a new client in New York and some fixed-price, fixed-scope 1 week milestones to churn through. And churn we did. In those first weeks Mad Mimi morphed her identity more than once and a few weeks later we had something we could play with. Mad Mimi was all about creating Promotions, making it easy for people to compose beautiful emails in as few steps as possible. Unfortunately I'm not good (yet) at making things visually beautiful, and so Gary (yes, the customer) nominated himself to become the CSS ninja, and over the course of the coming months, Gary learned how to navigate a Rails code base, run the app locally, use Subversion, and develop beautiful CSS designs. Eventually he even learned how to use Capistrano and deploy the app to the server, but I'm getting ahead of myself.
Not surprisingly, it took longer than a few months to create the product that Gary wanted, partly because Gary discovered what he wanted along the way, and partly because I was still learning how to do what Gary wanted. I had to implement Ajax image uploads along with image drag-and-drop to allow people to have a WYSIWYG-ish experience creating their Promotions. I had to pick and integrate a Rails-friendly background task processor (circa mid-2007) to allow users to pause, cancel, or observe the progress of their Promotion mailings. My drag-and-drop code eventually got rewritten by Tobie Langel when he joined the project in early 2008. The background task processor that I chose, BackgrounDRb 0.2.1, seemed like the only viable option at the time, so I chose it and we're still using it (though I wouldn't say we're raving fans). There was also the issue of dealing with large (to me) data sets triggered at any moment by a user action, such as a user uploading 50,000 contacts or sending a Promotion to 70,000 people. I had some previous (bad) experiences with handling lots of data with ActiveRecord, so at least I knew what not to do, and I got something working with data sets of less than 100,000 with a mostly empty database.
Along the way the Studio practice was growing. I quickly brought Brian Tatnall on board as our first apprentice in May 2007 to help with Mimi and some smaller Rails projects. Brian had worked for a small design shop previously and his big client followed him to Obtiva, so instead of helping me reduce my workload, he actually increased the Studio's workload by bringing a huge (for the Studio) ecommerce project with him. By the end of 2007, the Studio team had grown to 5 people, 3 of whom were apprentices, and Brian had rolled out of the Studio onto an on-site, hedge fund client.
Mimi's development gradually slowed over time due to my growing Studio responsibilities, Gary's burn rate, and waiting for Tobie's brilliant re-write of Mad Mimi's Promotion Composer. We silently launched MadMimi.com in February 2008, where it lay dormant for several months as Gary tweaked it and allowed a small number of initial customers to try it out and put it through its paces. At this point we were still on the same low-end RailsMachine server that the project had started with. As the initial customers started working with more and more data, and sending more and more emails, we found that running 6 Mongrels and BackgrounDRb and MySQL all on the same little server wasn't working. We worked with the guys at RailsMachine to split the web front-end from the BackgrounDRb back-end onto 2 low-end servers. Once we started getting significantly more load, we went to one of their higher-end "8 core" accounts. I must say I am a very happy RailsMachine customer, and we have just finished migrating our large e-commerce client onto RailsMachine. RailsMachine's people are incredibly responsive and have a knack for teaching you how to fish rather than just handing you the fish (they've taught me a lot about system administration).
Gary eventually started promoting Mad Mimi in May 2008. This was were the real fun began. Working with a customer to create something that only exists in their imagination can be an exciting process, particularly when you have the rapid feedback that Rails provides. But watching your code go out into the real-world and begin to generate revenue for your customer and feel the load of real-world data is not something I've been so close to before. Early on in May or June, I wrote a Capistrano task that would spit out Mimi's monthly revenue from the command line. By this point we had trained Gary to use Capistrano, so he was able to watch these numbers closely (as did we). I remember the week that Mimi started generating enough revenue to pay for her server costs ($300 per month). That must have been in June. Then Mimi was generating $1000 per month in July and we got excited. Gary wasn't paying for any advertising, just blogging and getting mentioned on Ajaxian and 37Signals Product Blog. And people were loving Mimi. Gary had created a certain vibe around the product, from the look-and-feel of the site to the Campfire chat support that Gary and his brother Dean expertly manage. Mimi's revenue was growing steadily and I was falling in love with the subscription-based web service model popularized by Basecamp.
Not surprisingly, Mad Mimi's traffic was growing too. We handled our first 100,000 person mailing with a couple hiccups. I needed to do some database optimizations to help with some non-performant scenarios and we needed to increase the speed of the mailings. We had a BackgrounDRb worker responsible for each mailing and it would simply run through the email list, sending one at a time. For our bigger mailings, this was taking too long, so I split up the big mailings to be handled by multiple concurrent workers, and worked on some caching strategies to improve our mailing speed. Our fastest mailings now hover around 20 emails per second. We're going to need to improve that even more, but for now it's sufficient. Gary learned all about how to get emails to render correctly in all the different web clients. Brian and I wrote TamTam so that Gary could create HTML emails with CSS and not have to use inline styles. Colin and I learned about DomainKeys and got those working to keep some of the pickier email providers happy. And by now we've sent out mailings as big as 350,000 and have sent several million emails within a 48 hour period. I've had to write my first couple MySQL stored procedures in order to optimize certain SQL queries and imports, with some guidance from a friend of mine. Needless to say, I've learned a lot in the process.
I presented my experiences with Mad Mimi at Windy City Rails in September 2008. At that point Mimi had just over 150 paying customers generating just over $5000 per month in revenue. I predicted that Mimi would hit $10,000 per month by December. And I'm excited to announce that this prediction was accurate. Mimi now has just over 300 paying customers generating over $10,000 per month. It has been an awesome experience watching Mad Mimi come from Gary's vision to a viable product to a contender against the established email marketing platforms. After seeing the way Gary learned CSS and was adamant about the product's personality, Mimi has forever changed the way I see my customers. After experiencing success with rapid feedback and leveraging Rails to iterate quickly with Gary, it has forever changed the way I approach my projects. And after seeing the simplicity of Mimi's revenue model and watching Gary turn away one buyout offer after another, it has changed the way I think about start-ups.
Nowadays, the Studio is 10 people and still growing. We've split into two teams and will be splitting into a third in the next month or two. I'm proud of the Studio and how much we've improved it since the chaotic, early days when we were chronically over-sold and our first few apprentices didn't get enough mentoring. We've had a lot of trial-and-error, but we've slowly, steadily improved our practices, discipline, and processes over the last 20 months.
Mon, 29 Sep 2008
Looking forward to a fun weekend with Leah, Fred, and Dan.
Sun, 23 Mar 2008
One year ago I stopped doing full-time on-site consulting. I started on-site consulting in 2004 when I joined ThoughtWorks, and continued through my first client at Obtiva. I have done a few multi-day, local stints since Spring 2007, but the vast majority of my days have been spent in a smallish office about a mile from my house in Wheaton, Illinois, USA. It was a risk to start Obtiva's Craftsmanship Studio and subsequent Apprenticeship Program, but after a ton of hard work, frequent mistakes and mismanagement, I can confidently say these last 12 months have paid off and the future is brilliant. Let me try to explain what I'm talking about, and why I'm confident about our first year's success. (And thank you to Michael Hunger for asking for more information on this topic.)
What is a Craftsmanship Studio? I should rephrase that to "What is Obtiva's Craftsmanship Studio?" because I can only speak with authority on that. First, what does "Craftsmanship" mean in this context? My best answer to that question is to advise you to go read Part 3 of "Software Craftsmanship: The New Imperative". Being a self-taught programmer, and coming from a right-brained background, the concept of craftsmanship immediately resonated with me. It should come as no surprise that when I had the opportunity to create my own practice within Obtiva, I tried to model it after the ideals that inspired me in Pete McBreen's book.
Second, what is a "Studio" in this context? The dictionary tells us that a studio is "an artist's workroom" or "an establishment where an art is taught or studied". That sounded right to me and it is supported by the fact that programming as art has been a strong theme among the leaders in our field for a long time.
The above ideas basically summed up the vision for our Craftsmanship Studio: a place where programming newcomers can come to learn the craft of software development on real-world projects working closely with experienced developers. Reality was much messier than our vision, and there were too many times when apprentices were isolated or working together without much oversight. This messiness can be attributed to the fact that I am a journeyman, not a master craftsman, and therefore this year was filled with mistakes as I learned (by trial-and-error) about project management, customer relations, capacity planning, and recruiting. Thankfully we have had about 50 retrospectives during that time and have adapted our agile principles into a process that continues to improve each week.
So if our Craftsmanship Studio is where newcomers and old-timers work, learn, and mentor, what is an Apprentice? The above reference to Part 3 of Pete's book will quickly introduce you to the concept of apprenticeship...
A central tenet of the craft model is that it is hard to pick up a skill just by being told about it. You actually have to practice the skill under realistic conditions and under the watchful eye of an experienced practitioner who is providing feedback.
This describes the need and the relationship, but in reality, who are these people, these supposed newcomers? My working definition for someone in an apprenticeship is: a person who is looking to maximize their learning opportunities, even at the cost of other opportunities. Often this means purposely putting yourself into a Be the Worst situation, which is exactly what I did when I joined ThoughtWorks. For Obtiva, this means we're looking at potential and attitude rather than credentials. These people could probably make more money somewhere else in the short-term but they are making an investment in learning that will pay off in the long-term.
I would like to see our Apprenticeship Program mature into a more formal apprenticeship with better feedback mechanisms and milestones. That said, I can say that the four apprentices who have participated in our Craftsmanship Studio have been very successful despite my shortcomings as a manager. This year reinforced for me what I learned when I was an apprentice: your apprenticeship is what you make of it. We had a Perl web developer come to us, learn Ruby and Rails and Java, and leave us writing a multi-threaded Ruby/JRuby/Java/JNI application that leveraged a 16 core machine for a local hedge fund. We had someone come to us to reboot his career, learn Unix, MySQL, Perl, Ruby and Rails and is now both managing and developing e-commerce deliveries for the Studio's largest client. We had someone come to us from a local Rails sweatshop, learn technologies like Sphinx, rSpec, god, ActiveMerchant, CruiseControl.rb, along with Perl, who is now introducing Git into the team and will soon be technical lead on his third Rails e-commerce project. We had a network administrator come to us, who after rapidly delivering several different Rails projects, is now wrangling a large, chaotic Rails codebase under control via rSpec better than many experienced developers I know.
What do I attribute our success to?
Tue, 10 Jul 2007
I returned from Manhatten last month after my third running of Obtiva's Ruby on Rails TDD Boot Camp. And for the first time, I wasn't completely exhausted! Although I'm not a natural-born public speaker, practice does help a lot, and hopefully it showed this time.
It's hard to believe that it was only a year ago that Obtiva was scrambling to meet the Rails training demand as we constructed our curriculum in preparation for our first course. Since that time the course has run 7 times with Tyler Jennings, Obie Fernandez, and myself as trainers in Illinois (Wheaton, Lombard, Chicago and Evanston), California (San Jose), and New York (Manhatten) for a wide variety of businesses (Cisco Systems, Northwestern University, Stony Brook University, Nielsen Media Research to name a few). It's been an awesome experience to provide these trainings and adapt the course with the changes in the Rails landscape.
I am pleased to announce that our 8th running of this course will be for our newest client Google. And I am even more excited to announce that the instructor for this course will be Obtiva's newest team member Dave Astels.
Sun, 24 Jun 2007
This post was going to be an email to a friend who is interested in joining Obtiva, but most of what I was going to tell him were things I've been wanting to blog about, so I figured I'd make it a public-facing message...
After I had been with Obtiva for six months, I experienced a change in perspective that has changed my career, and hopefully, the careers of many other future Obtivians. At that point I had already delivered my first training course, and with the help of my fellow Obtivians, I had introduced Ruby and Rails at the J2EE shop of one of our main clients. I was excited to make an impact, but it began to occur to me that a company as small and as financially stable as Obtiva is more than just a place to work hard and make an impact, it is a vehicle that can be driven. The way that the company is structured provides us with incentive to not only do great work, but to go and create the work we want to do. And that is exactly what I did. I subscribed to a bunch of feeds like the 37signals Gig Board and started winning business. These wins allowed us to hire 4 of our 11 employees and provided us with the first wave of apprentices for our Craftsmanship Studio in Wheaton.
Nine months later, I am living a dream. I am riding my bike to work and spending my days with Brian Tatnall, Victoria Wang, and Joseph Leddy in the Studio, helping them along their paths toward mastery as we deliver Rails projects to a variety of customers using whatever tools we want. While I certainly cannot take all of the credit for my good fortune, I can say that I have maximized the situation, and we have room for others to do the same: Gareth sees the same potential and Obtiva has become a vehicle for him to grow our consulting business into the world of business and finance.
Friend, you will likely have many opportunities to join Obtiva in the coming years, but I am uncertain whether the company will be as malleable as it is today. You are a multi-talented person, a world-class software developer, to be sure, but you have many other talents that could be leveraged to allow you to use this opportunity as a vehicle to drive you toward exactly the sort of work that you want to be doing, and where you want to be doing it.
Thu, 24 May 2007
As we continue to expand our Craftsmanship Studio, I am becoming increasingly aware of our dependencies. In order to grow better as we grow bigger and maintain a healthy mix of apprentices and journeymen (no masters yet), we are absolutely dependent on the essential attributes of a software developer. As Andy said in one of his long-lost podcasts, the two most important attributes of a software developer are the ability to communicate well and to learn quickly. I would add passion to that list.
Mentoring apprentices comes in many forms. Some of it is indirect ... they overhear your conversations ... they read your code ... they read your blog (!). But some of it needs to be direct, and in those moments, I am sacrificing my productivity on some other activity in order to answer a question, provide guidance, or sketch a domain model. That sacrifice is an investment. I believe that 10 minutes now is going to be paid back tenfold in future productivity gains. And I believe that because it's obvious that our apprentices Victoria and Brian have the attributes that Andy was talking about.
A couple of examples ... Victoria's ability to communicate well lightens my workload because as she is implementing features, she can interact directly with our customer, allowing me to stay focused on my development tasks rather than diving into Basecamp or GMail. Brian's ability to learn quickly was apparent after I reviewed our Rails portfolio with him in his first few days and a week later he had single-handedly retrofitted ActiveRecords (with advanced associations) onto a legacy database on his first Rails project.
Fri, 11 May 2007
Working in the Wheaton office has been a most excellent transition. While there are some mundane details to take care of, it's all in the name of establishing a Software Craftsmanship Studio we can call our own. Like any software company that does consulting and training, there will always be some of us at client sites or running trainings in exotic locations. But there will also always be a sub-set of us in the office delivering full-fledged projects from our Studio.
And that is what Brian and I (and soon Victoria will be nearly full-time with us!) are establishing. A place where apprenticeships flourish, where footwear is optional, where collaboration is dialed up to 11, and where we deliver top-notch software to our customers on a daily basis. It's been intense thus far, and I anticipate some great times ahead.
Mon, 26 Mar 2007
Kevin introduced me to Gareth Reeves almost a year ago, and Gareth and I quickly hit it off as we carpooled to the first annual RailsConf. I was excited to meet Gareth because I remember reading some of his papers on XP back when I couldn't get enough of the XP kool-aid. Ever since then we've been working on getting Gareth into Obtiva and I'm thrilled to announce that we finally have him!
Gareth has been involved with XP from the early days, and was mentored by none other than Kent Beck. He brings a mountain of agile and Java and (more recently) Ruby experience with him and we're excited to see where his expertise takes us. Gareth's first gig will be joining Ryan for a Rails coaching gig at a nearby hedge fund.
Wed, 21 Mar 2007
I'm excited to announce that my primary goal for 2007 will come to fruition next month. Since we opened our Wheaton office in December, I've been splitting time between working on-site at one of our main local clients and leading our Rails projects out of the office. Obtiva just landed a project (yet another coastal gig) that will allow me to work full-time out of our Wheaton office building our Rails practice and further establishing our Craftsmanship Studio. Life is very good.
While the project itself is interesting in its own right, one aspect that greatly excites me is that Jeff Patton, agile product design pundit, is heavily involved. I am excited to collaborate with Jeff and experience his approach to agile software development on an ambitious Rails project like this one.
Thu, 22 Feb 2007Ryan with some routing issues in which a single character (a period) was the cause of confusion. I nearly fell off my chair laughing after this little interchange...
Ryan: yeah, the breakthrough was the dot. thanks for the help :-) Dave: it's always one stupid character Ryan: yes, and that character is often me
Sun, 11 Feb 2007
Working for a small software company excites me for many reasons. One of the most rewarding aspects of the 10 months that I've been working for Obtiva has been working with Tyler and watching him mature as a Rubyist. Tyler was already hacking Ruby when I met him, but since guiding him toward Rails and Ajax, Tyler has delivered some excellent work on a soon-to-be-announced Obtivian web offering. One of my favorite aspects of working with a small group of people vs. going independent is that our weaknesses and strengths are complimentary: our whole is greater than the sum of our parts. Although I did a lot of speaking and training in 2005 and 2006, one of my biggest weaknesses is (still) monologue. After road-tripping to GLSEC 2006 with Tyler last year and attending his talk, it was immediately apparent that Tyler is a gifted speaker. I'm pleased to announce that Tyler will be talking about Ruby and Rails at this month's Chicago Uniforum meeting and will be teaching our Ruby on Rails TDD Bootcamp in early March.
Even cooler still, Obtiva has just hired our first apprentice (and female), Victoria Wang. This is another important step toward establishing our software studio in Wheaton. Apprentices bring enthusiasm and an appetite for learning that inspires. On top of this, Victoria brings some other important qualities with her: excellent visual design abilities (that's her RailsConf-inspired drawing) and a non-male perspective. Welcome Victoria!
Wed, 24 Jan 2007
Like many small consulting companies, Obtiva was built upon a longstanding relationship with a large, local client. A client like this provides a strong foundation on which to build a business. As I said in my 2006 Retrospective, my primary goal for 2007 is to spend an increasing amount of time in our Wheaton office leading our Rails projects. To achieve this goal, we will need to bring in a steady stream of new projects. I'm happy to say that January has proven to be a good start. We kicked off two new Rails projects: negotiations software for a large bail bonds company and (yet another) social networking application for a major television network.
While these projects are wildly different, they have one thing in common. Both clients are located in California. Why is that significant? Go read Mike Karlesky's insightful post on Inshoring. While neither Obtiva nor Atomic Object can compete with the prices of our offshore brethren, it is possible for us to undercut our coastal colleagues.
Moving toward distinct, shorter-term projects creates a need for a steady stream of new business, which is exacerbated by our expertise in Ruby on Rails and agile practices. We have found that by coupling a killer technology like Rails with the disciplines of frequent releases and test-driven development, we are able to deliver our projects ahead of schedule. This puts more pressure on us to bring in more projects, but it also increases our chances for repeat business, not to mention the morale boost our team feels. It's a positive feedback loop that I hope we'll still find ourselves in when summer arrives.
Fri, 03 Nov 2006Ryan Platte will be joining Obtiva in a few weeks. Ryan is the fearless leader of Chirb and will add significant depth to our dynamic language expertise.