How To Measure Things

I made a bit of personal history this past Thursday. I got up early and got to the pool before work, as is my daily habit, and logged another 3,000 yard workout. I've been spending some time with "toys" these days: 3x200 free w/ pull buoy; 3x200 free w/ snorkel; 3x200 kick w/ fins; 3x200 free w/ fins; 3x200 IM on 4:00. I felt great when I was done, as usual. I usually do 2,000 yards in the morning, but I cranked it up to 3,000 for three days to meet a goal. I've logged 520,000 yards for 2008. That's 10,000 yards each and every week, or 2,000 yards for each and every working day. My previous best total was 440,000 yards back in 2006. I've been keeping track since 1994. My average over all that time is 340,000 yards per year.

This isn't an extraordinary total for a really good swimmer. I think high school athletes can put in 6,000 yards per day, and Olympic athletes will do my weekly total in a single day. But it's something special for me.

How did this happen? There are a few important factors:

1. Good health. I've blogged about the bulging disk in my neck that ruined my 2007. It helps that I've had no injuries or sick time.
   
2. Availability of a pool. I've been fortunate for my entire career, because I've always been able to find a pool nearby wherever I've worked. And they've generally been staffed with people who make the opening of the doors and the arrival of a lifeguard as dependable as the rising of the sun.
3. Consistency. Woody Allen said "Eighty percent of success is showing up." That's certainly true here. I've always managed to weave swimming into the day in such a way that it's relatively transparent to my wife.
   
4. Swimming with others. For the longest time my ritual was to swim a mile of freestyle at lunchtime and get out. I never swam competitively, so I didn't know about workouts or intervals. I managed to figure out how to do butterfly on my own so I could swim the individual medley, but that was my only concession to trying anything new. In 2000 I finally worked up the courage to try Masters swimming. I was nervous. I didn't know if I could finish a longer workout, or keep up with the intensity. I found out that swimming with others makes all the difference in the world. The people I swim with are now among my best friends. The social interaction and better workouts have contributed greatly to my enjoyment of swimming.
   
5. Measurement. My tracking system is simple: I have an Excel spreadsheet that acts as the database and does all the lovely plotting for me. I roll everything up by the week. I have "best month" totals. I can tell every week exactly how I'm doing. I derive a great deal of motivation knowing that I'm close to a goal of some kind. Sometimes that's the one thing that gets me out of bed early in the morning.
   

I'm already thinking about how I can top it. Just 10% more each day will do the trick. One more 200 to make my habitual daily total 2200 yards. They add up. I find that the snorkel is helping my lung capacity, so I'll work that in every day. And I'd love to learn how to tolerate butterfly for longer stretches. I can endure 100 yards of fly in relative comfort now. I'd like to work that up to 200 yards at a time in 2009.

I've been thinking a lot about that last bit, measurement. Tracking makes a big difference. I've been working on a star schema for a relational database to house my swimming data. I've learned more about data warehousing from that simple problem than I ever knew before. If it ever reaches a more finished state I may blog about that as well.

But it makes me wonder: How can I use measurement, metrics, and tracking to motivate the other important things in my life? My last blog talked about my frustration over lack of progress in learning Lisp and other things. How can I start tracking my progress through "Practical Common Lisp" to ensure that I end up at the desired destination?

What about measuring software development progress? If I could figure out how to do that, I'd be a wealthy man. Everyone in the software development industry laments the sorry state we're in. A large percentage of projects are never completed or fail to deliver the desired functionality. If we only had enough management, oversight, governance, and measurement!

It's not that those things haven't been tried before. You can't throw a rock without hitting a methodology that promises better results than the scorned "waterfall". Project managers have been around for as long as there have been projects. (You can see them in Egyptian hieroglyphs, holding Gantt charts on papyrus.) Analogies abound: "Software development is like building a building" - hence the love of the title "architect". "Software is like manufacturing" - hence the term "widgets".

Where I work, the conclusion is that if some hasn't worked in the past, more in the future will make things better.

Why hasn't any of this worked? What's so different about software development?

I think my swim tracking has a couple of key features that software lacks.

First, it has a well-defined time scale built in. I know exactly how long it takes me to swim those 2000 yards every morning. I have a metronome in my head for each lap, each 200 yards. (Unfortunately, it's a very slow timepiece.) The week closes every Sunday, and the workouts pile up in a steady drip, drip, drip as the year goes on. I can easily extrapolate to the goal at the end of the year: "Still averaging 10K per week, with two in the bank to anticipate that week's vacation in the summertime."

Second, the metrics are well defined. There's nothing fuzzy to account for, like "quality" or "maintainability" or any other "-ility". It's distance and time and frequency.

Third, the metric are additive. I swim in a pool that's 25 yards long. Laps roll up into sets, sets into workouts, workouts into weekly, monthly, and yearly totals. If I look at the plot of yards aggregated at the end of the month I can clearly see where I'm leading or lagging. I got up early to do 3,000 yard workouts last week because I wanted to cross the 520K line without having to worry about winter weather keeping me out of the water.

Fourth, it's just one individual. I'd imagine that it's harder for a coach to keep track of an entire team. There's an art to cultivating each individual to achieve their best and orchestrating a team to an end of year peak for a big meet.

Manufacturing has employed statistics to drive continuous improvement since Deming and Shewhart showed us out. There are any number of Six Sigma black belts running around applying these techniques.

So why does software development resist such improvements? I believe there's something different about it. Like Knuth said, there's an art to it. The social aspects of doing it in large, far-flung groups don't lend themselves to statistical measure. We've all heard about Dilbert-esque measures that reward those who find bugs ("Look at that! Found another one!"). We've griped about the scourge of unintended consequences caused by bad measurements. Measure success and progress by KLOC or number of classes checked in and you're likely to see your code base expand in size.

I'm going to spend some time seeing if I can extend my success at measurement in swimming to other things. If I make any progress, I'm sure I'll post it here.

Overcoming Writer's Block

I've been suffering from terrible writer's block, as evidenced by this blog. It's not that there's nothing to say. Sometimes there's TOO MUCH to talk about. I'm stymied by a kind of shyness: "Why would anyone be interested in what I have to say?" I'm self-conscious about putting my opinions in public. The Web is wonderful, but I worry about how it breaks down our privacy.

I downloaded TiddlerWiki today, just to try it. It's supposed to have some nice features for linking and searching that might be useful.

I'm thinking about two applications for it. First, it'd be a great way to centralize little tidbits about software development (e.g., the stuff I have for setting up a MySQL database buried in my daily diary). Second, it might be a web-based alternative to my personal journal.

One thing I like about it is that I've downloaded an empty.html file to my hard drive. It's in my control. I'm still wary about putting stuff out there, "in the cloud", living on someone else's servers.

I also looked at a site called GoalsOnTrack.com. It's a nicely conceived and implemented site for establishing and tracking goals: simple, graphical, and elegant. Why didn't I write that? I signed up for a username and password, but first I read the terms of use. It was a real eye-opener. It's free now, but they actually have extensive language about billing and monthly charges and cancellation fees. The currency amount just happens to be $0.0, but it'd be easy to change. If the site is free, why add all that unless you plan to monetize it as soon as you reach a threshold number of users?

There's also language that says your data no longer belongs to you. The people who own the site can do whatever they bloody well want with it once you upload it. Woe be to you if you happen to have a goal of working through all the positions listed in the Kama Sutra by your next birthday! It could be dicey if your partner happens to be a co-worker or someone else who isn't your wife.

Anyway, both technologies are right up my alley. I've been keeping an electronic journal in Word since 1994, and I've been tracking my swimming yardage in an Excel spreadsheet since 1996. I've been looking for ways to improve both. The Word docs are nice, but I can't search them very easily. I thought about using Lucene to index and search the documents, but then I learned that my swim tracking got in the way. I started by embedding a spreadsheet into the Word doc at the start of every month. The POI library that I used to read the Word docs blew up over the Excel spreadsheets. So I had to move the Excel data out of the Word documents and into something better.

I became interested in data warehousing and dimensional modeling. I'm designing a data warehouse for my swimming data now. It's a surprisingly good demonstration for the problems of data warehouses and ETL. I'm guilty of "premature roll-up" (don't tell my wife). I aggregated my data by week, and now I'll have to work really hard to get back the finer grained daily results.

GoalsOnTrack seems like a great way to track all the other things that I need to get done. I've been pretty good about doing yoga on my own since I started taking a weekly class fifteen months ago, but actually seeing the hours spent in downward dog accumulate would be a great motivator. I'd love to brush up on the mathematics I've forgotten and learn the news things I need to take a new direction at work, but I'm a lazy man who tends to watch the Boston Celtics on television too much. Having a measure, like my swimming yardage totals, seems to motivate me.

How much tracking does one person need? After a while, the tracking activity consumes a great deal of your time. How anal.

Bloggers like Seth Godin and Jeff Atwood are my latest heroes. I don't know how they manage to be so insightful. Seth Godin posts every day. That old adage about writers having to write every day seems to be true. If I'm going to find a voice, I'll just have to start getting used to the idea of writing every day.