Get rid of manual quality control

March 29, 2010 – 17:55

After reading recent Eric Ries piece in HBR, one thought was bouncing my head…

What are such rules for software development service companies, like mine 42 Coffee Cups, that does Python, Django web development?

Some things in Eric’s article resonate quite closely to me. For example, when a stupid bug creeps into the code we deliver to our customers for review, at least half of time I’m being asked – aren’t your QC people watching it?

I had difficult time answering this, because my gut feeling from 20yr of software development is that dedicated QC slows things down while hardly compensating with increased quality.

You know, if debugging is the process that removes bugs from a code, then programming must be an activity that implants the bugs there.. :)

In my reality, when this bug is drilled down to find the cause, it’s almost always something like

  • the particular piece of code wasn’t covered by unit tests
  • whoever integrated the code didn’t run the tests before shipping (yup, it happens occasionally – we’re all humans)
  • whoever integrated, hasn’t bothered to eye-check the result and html, being clean, gives awful UI experience
  • there was a deadly misunderstanding between customer and developer

Each and every item on the list could be fixed by either

  • automating more work
  • adding more automated checks, like running CI tests on every checkin or measuring the test coverage
  • delivering to customer faster and ask for his feedback

During early stages or if done wrong, this gives awful results – a customer has to spend a day just listing defects :(

Yet…

Isn’t a reliably shorter round-trip time (time from a feature request submitted and the code shipped to you) and solid process of “making errors only once” much better than have to wait an additional day (on average) for QC review, pay for its costs and still have errors shipped sometimes?

42 Coffee Cups still have a long way to walk to achieve the ideal sanity, yet I believe this is a way to go

What do you think?

  • http://tilarids.blogspot.com/ tilarids

    I completely agree with you that using QC can slow down the development. And in many cases using dedicated QA engineers for testing doesn't lead to significant improvements in the code quality. But in any case I think that exploratory testing is a must.
    First, a lot of bugs with unit and functional tests passing OK can still be here. You already mentioned the messed UI, but also what about tests with different input data? If you have a complex system, you can't cover all the input choices by automated tests. And I don't mention race conditions, environment problems, etc.
    Second, using the automated tests (even with 100% code coverage) doesn't mean you'll have the 5-stars code. Even with all the input choices covered (that's in turn practically impossible) it would be impossible to cover all the side effects in case of complex non-pure system.
    Third, QA engineers are not “test cases running machines”. With QA help you can make your product better, but with help of automated tests you can only make your code to run smoothly.
    Therefore, I think the best practices are in automating the routine work and leaving exploratory/advisory tasks to the QA people.

  • http://a.khavr.com/ Andriy Khavryuchenko

    I'm not talking about automated tests and test coverage.

    I'm talking about running 5-why on every defect and missing goal religiously. And doing actions basing on decisions made.

    This could be automated exploratory testing, regular visual UI reviews and, most importantly – process changes.

    As Eric Ries quoted some time ago [1] – behind every technical problem, there is some human problem.

    That is the point I've tried to make.

    [1] http://www.startuplessonslearned.com/2009/04/bu…

  • http://i1t2b3.com/ Alex

    Sounds very good. When we start to “delivering to customer faster” we forget about the other rules…

  • http://a.khavr.com/ Andriy Khavryuchenko

    That's why you have to have

    * an established process to monitor the quality of your service and
    * a process to learn on things that where broken

    The later is best started with “5 why” technique.
    Former depends on your business model. I'll give a blog post about metrics soon..

  • http://www.revsoftware.ca/ Shane McCallum

    Hey Andrey, interesting thoughts.

    IMO if your company has a single QC person on staff or outsourced your software will be better, period. It is not the customers job to find bugs, and programmers can only automate for testing flaws they are already aware of. One person to spend a few hours actually “using” the application saves more than just client complaints it allows your team to discover logic and design flaws as well as programming bugs.

    I am all for automated testing as it takes care of most of the obvious problems and keeps your team from introducing new bugs to old code. But I would not discount QC as a valuable tool both for you and your customers.

  • http://a.khavr.com/ Andriy Khavryuchenko

    Shane, I'm tempted to add QC every time I see defects popping up in the weekplan.

    Yet, in the dynamic product development – what I should feed to QC person? Script that he has to run and do an eye-checking? Half of that's could be done automatically, other half is a job of person, integrating the feature branch into the main one. If the “other”, manual part is delegated to other guy – what he should get? The whole project knowledge that got project coordinator?

    In waterfall-style projects, the answer was “hey, let's add more documentation”. So every screen was signed off, every button action was described, et cetera… Oh, wait… That was supposed to be done and every time it really was attempted, the budget sky-rocketed…

    On “consumer” project I know how to do this – I just a/b test a single change and check if there would be any errors in my inbox and if any of key metrics (check AARRR metrics by Dave McClure [1]) would not improve.

    I'm not that sure what I have to measure in outsourced projects. Obviously, when you call me “Andriy, wtf you've delivered – I can't show this to my client” – that's bad. Even worse is that it's too late to act – we have to understand “what's an error” for you and your client and, at least, filter them earlier.

    Or, even better – stop putting them into the product from the very moment project starts.

    I'm still awaiting for “lessons learned” on the project we're doing for you and will institute them as wide as possible as soon as possible.

    So, I'm not expecting a breakthrough from any single process improvement. And I really believe that have those improvements combined, we, 42 Coffee Cups, would provide much better service than we do now.

    [1] http://500hats.typepad.com/500blogs/2007/09/sta…

Andrey Khavryuchenko himself is proudly powered by WordPress Entries (RSS) and Comments (RSS). Theme by Bob