ummsomething.com

I posted this to the context-driven software testing list I’m on. If you want a quick primer on where the people I am talking with are coming from, go here. I’ll comment anything interesting that comes back from it. This was going to be a response to (a) request for industry standard metrics, but then a distillation of what I think about discussing software development and testing with managers fell out.

It’s very easy and true to say that the application of universal/typical/broadly applicable measurements for software quality is a bad and dangerous idea. Manufacturing measurements of quality don’t work, every “quantitative” measurement is based on qualitative data points and collection, etc.

The management theory we are engaging with believes that all business activities can and should be measured, that there is always a way to measure success, progress towards goals, and see the impacts of changes to process. Whether it is volume/cost/productivity/calls per unit/worker/mile/day, baseball statistics, etc, management of virtually every industry and occupation is believed to have been improved by this type of analysis.

It’s easy enough to see how someone approaching software development from this context grasps at bug counts, reporting dates, lines of code, or anything that smells like something that can be used to get a handle on what is happening and what to expect. Learning a business and developing a good enough “feel” (or bundle of heuristics, if you like) is hard and time-consuming anyways; for those without the background, even if they do have the intellectual horsepower, they are not going to learn fast enough from the insular, insecure, and often brilliant people they find in our field.

In the Power-Point(y-Headed boss) world most of us work in, managers want easy and universally applicable ways to get their bearings. These people need some idea whether they need to start shaking things up or stay the course, they need to be able to demonstrate their own positive impact, and so forth. How do we help these people feel warm and fuzzy about progress towards goals, measuring quality, success, etc? What can they do to help, and how will they know?

We need to not only be able to say what It Depends on, but to educate them about how to think about software quality and productivity in testing. How can we positively reframe the context? I’ve thought about what other contexts we might substitute to replace the manufacturing quality approach. Here’s three I’ve tried so far.

Since writing software is really more of a creative exercise, I have had some limited success asking people to think of it like other writing; focus on the verb and reflect on what it means. I’ve seen many writers say that they never finished a article/story/novel/etc, they simply ran out of time or patience (usually someone else’s) to keep improving it. This gets some nods sometimes, but doesn’t end any “But how do we measure it?” conversations.

Another approach I’ve tried is to say that writing software is like building a house while having to fabricate all the materials. Maybe you have a really skilled 2×6 developer and a really crappy electrical box developer, or he’s the same guy. This seems even farther away though, and lends itself too easily to people trying to participate in the metaphor incorrectly and missing the message that the process is not as simple as proper component assembly.

The other context I could compare it to is managing a research lab. How do you measure breakthroughs? How do you compare the impact of one breakthrough to another? How to you measure the impact of a breakthrough coming one month, week, or day earlier than it otherwise would have? This may be satisfying when you consider that managing a research lab consists of creating a quiet environment with sufficient resources and reliable experiment facilities for best thinking results, but still doesn’t address the issue of measurement and management. Also, this usually translates as “just stay out of the way”, which no one really wants to hear.

None of these frame the problem of the testing context. Testing metrics would be a measurement of how good we are at finding a percentage of a variable quantity dependent on multiple factors. I can probably list a lot of factors to talk about what “It Depends” on here, but still not something anywhere near enough to coherent for 8th grade reading levels and three sentence attention spans.

I’m thinking we may need to solve the software development context problem before we get to the testing one. That requires explaining and selling qualitative methodologies as intellectually rigorous and somewhat reproducible. Measurement? Er…

What do you think? Can you help me reframe the problem statement?

It might seem that we are winning, or at least not losing the net neutrality war right now. Or are we?

By we, I mean everyone that is not in the telecommunications business, and by winning, I mean that we are not dealing with our network providers deciding whose content and programs we can or can not run across the “last mile” of our internet connections. Web pages, cable tv, voice…it’s all bits. VOIP, streaming media, the democratization of media, the connection of the free world and even the explosion of what we used to call e-commerce back in the Nineteen Hundred and Nineties all happened because the network providers were not allowed/didn’t think of/were too incompetent to restrict access to the Internet.

Since many of our network providers (Time Warner, etc) are also content providers, they must be watched carefully so they do not abuse their position to advantage their other businesses. This is hugely important for fostering innovation. The last 15 years changed how we live, and simply couldn’t have happened if new ideas needed sponsorship and funding in order for an audience to see them.

If you are opposed to Network Neutrality, please issue yourself a generous dividend from the telecommunications provider you must have an ownership stake in and start considering your exit strategy. If you do not have any such financial position and still think that cable and telcos have your interests in mind, please shoot yourself.

I believe that our present and future is best served when we operate in the kind of market where new ideas can thrive, and the open Internet has shown itself to be an incredible laboratory for this. Net Neutrality was the plank in Obama’s platform that let me know that we were moving past “A Series of Tubes” in Washington, and that the good guys might even win this one. So far, so good, and it is a victory for all of us and our future that so far, anyone can set up a website and/or web service that anyone can get to.

However, another front in this fight has been ignored…up until now. Here is fresh evidence that some people in Washington are paying attention.

Previously, mobile providers have locked down on what we can do and how we can do it, and we’ve let them. The multiple handset makers and carriers provided at least the appearance of competition. Then, the breakthrough: Apple’s App Store. In less than a year and a half: 65,00 applications, 1.5 billion downloads, and from what I can tell, a lead as a mobile platform that is going to be really hard for anyone to catch up to.

A typical life cycle for a technology is for it to appear as innovation, to go through a generation or two of refinement, and then to become commoditized. Everyone enjoys the benefit of competition to provide the commodity, whether it is cars, groceries, or anything else where we have choice and open competition. Apple’s dark genius is that they only appear to have provided a platform where mobile apps can be commoditized and innovated on…as long as you support their hardware on their exclusive carrier.

In reality, they control the hardware, the operating system, and even the developer kit. This is similar to their personal computing model, except adding the idea that they get to evaluate and decide which applications can be run. Everyone craps on Microsoft about their competitive practices, with plenty of justification. Developers can still write whatever software they want for Windows, and don’t need to get Microsoft’s approval to release it to consumers. Microsoft doesn’t control the hardware AND pick your ISP, either.

There is a great deal of change and innovation to come over the next couple decades until we get to ubiquitous portable computing. Maybe application virtualization could make our mobile devices simple interfaces, provided wireless platforms and carriers don’t prevent it.

If we could go back to 1990, would we have let Windows become the standard for at least the next two decades? What will we think about the App Store in 2029?