I spent most of my twenties managing direct marketing campaigns for the financial services industry, where we would release several multi-million dollar programs per year in the hope that they would generate enough customers to justify our salaries. Measuring and predicting the performance of these campaigns was a simple statistics problem – given X expectation and Y performance Z effective days since delivery into the campaign, estimate the difference between actual results and expected results.
The tricky part was getting Z (effective days since delivery) right early in the campaign, since you were trying to predict how fast a slug of bulk mail moves through the postal service system. This was subject to all kinds of minor disturbances such as holidays, weekend timing, and potentially even how your letter shop sorted the file for production. This was before the Internet-based tracking systems we have today, so we were very old school and did a lot of estimation.
As the program manager, I cultivated this Zen state for the first couple weeks of a major program. We knew there was a zone of uncertainty: the most rational response to any developments was “wait and see what happens next”. While we could make good predictions several weeks into a campaign about the outcome, it was actually better not to look at anything from a strategic perspective for the first couple of weeks.. There was just too much noise to get a good read on the signal.
Flashing forward to the present, we put up our first website (a little AJAX based word game solver) a couple of months ago and had to keep tabs on what was happening.
Naturally, as an analytics guy, I have my own preferences about how I wanted to see my data, so I wrote a couple of small scripts using the SL4A Python scripting package and the Google Analytics API to provide me with the ability to get some additional real time analytics from my Android phone.
This wasn’t anything very complicated – just a few slices of additional data delivered my way, available upon request from a touch screen icon on the phone. I started checking our site statistics several times a day – and potentially several additional times a day if we seemed to be lagging.
Once we had this river of real time feedback flowing, I naturally wanted some tools to actually *do something* about the information. The first step was adding Connectbot (open source SSH client for Android), which was followed by building a couple of other admin features into our sites. While I didn’t have the speed of doing tweaking and development from a desktop, I did have 24-7 access to the site and could tweak some key elements of my SEO program, regardless of where / what I was doing at the time.
The truth sunk in this past weekend, when I flipped a setting on the site based on few hours of results from a change. Several hours of results isn’t enough of a sample size to get a reasonable read on an issue for most questions. I wasn’t really GETTING anything from real time access to my performance data, except the stress of being aware of every time we had a temporary dip below the median performance (which, statistically, occurs uh… 50% of the time).
Compounding my error, I actually now have the ability to ACT on this half-baked version of the truth. While theoretically useful, in practice, I’m losing the “cooling off” period that my desktop-based process required (eg. go home, fire up development box, open up the project and do an appropriate change/promote cycle). Using my hyper kinetic real time system, I could shoot myself in the foot at warp speed.
I’m reminded of a news story comparing the performance of online stock trading accounts with conventional brokerage relationships. The online traders actually did worse than the “old school” accounts. The root cause was identified as overtrading (more trades per year, higher cost associated with the trades). This is ironic given that most of the online traders were more “sophisticated” in terms of investment knowledge. The real-time trade interface removed the friction of the original process, which provided a moderating influence on their decision making process.
So this is worth thinking about next time somebody pitches you on a cool real time analytics tool. Sure..it’s nice to have up to the minute data – and yes, there are certain operational applications (process management, fraud detection, etc) where it helps to have the pager go off… but are you losing your cooling off time for strategic decisions? Are you getting so close to your process that you cannot see the forest from the trees?