Comments on: Get rid of manual quality control

By: Andriy Khavryuchenko

Andriy Khavryuchenko — Wed, 31 Mar 2010 23:19:07 +0000

Shane, I'm tempted to add QC every time I see defects popping up in the weekplan.

Yet, in the dynamic product development – what I should feed to QC person? Script that he has to run and do an eye-checking? Half of that's could be done automatically, other half is a job of person, integrating the feature branch into the main one. If the “other”, manual part is delegated to other guy – what he should get? The whole project knowledge that got project coordinator?

In waterfall-style projects, the answer was “hey, let's add more documentation”. So every screen was signed off, every button action was described, et cetera… Oh, wait… That was supposed to be done and every time it really was attempted, the budget sky-rocketed…

On “consumer” project I know how to do this – I just a/b test a single change and check if there would be any errors in my inbox and if any of key metrics (check AARRR metrics by Dave McClure [1]) would not improve.

I'm not that sure what I have to measure in outsourced projects. Obviously, when you call me “Andriy, wtf you've delivered – I can't show this to my client” – that's bad. Even worse is that it's too late to act – we have to understand “what's an error” for you and your client and, at least, filter them earlier.

Or, even better – stop putting them into the product from the very moment project starts.

I'm still awaiting for “lessons learned” on the project we're doing for you and will institute them as wide as possible as soon as possible.

So, I'm not expecting a breakthrough from any single process improvement. And I really believe that have those improvements combined, we, 42 Coffee Cups, would provide much better service than we do now.

[1] http://500hats.typepad.com/500blogs/2007/09/sta…

By: Shane McCallum

Shane McCallum — Wed, 31 Mar 2010 05:43:25 +0000

Hey Andrey, interesting thoughts.

IMO if your company has a single QC person on staff or outsourced your software will be better, period. It is not the customers job to find bugs, and programmers can only automate for testing flaws they are already aware of. One person to spend a few hours actually "using" the application saves more than just client complaints it allows your team to discover logic and design flaws as well as programming bugs.

I am all for automated testing as it takes care of most of the obvious problems and keeps your team from introducing new bugs to old code. But I would not discount QC as a valuable tool both for you and your customers.

By: Andriy Khavryuchenko

Andriy Khavryuchenko — Wed, 31 Mar 2010 05:14:17 +0000

That's why you have to have

* an established process to monitor the quality of your service and
* a process to learn on things that where broken

The later is best started with "5 why" technique.
Former depends on your business model. I'll give a blog post about metrics soon..

By: Alex

Alex — Wed, 31 Mar 2010 02:53:59 +0000

Sounds very good. When we start to "delivering to customer faster" we forget about the other rules...

By: Andriy Khavryuchenko

Andriy Khavryuchenko — Tue, 30 Mar 2010 01:54:13 +0000

I'm not talking about automated tests and test coverage.

I'm talking about running 5-why on every defect and missing goal religiously. And doing actions basing on decisions made.

This could be automated exploratory testing, regular visual UI reviews and, most importantly – process changes.

As Eric Ries quoted some time ago [1] – behind every technical problem, there is some human problem.

That is the point I've tried to make.

[1] http://www.startuplessonslearned.com/2009/04/bu…

By: tilarids

tilarids — Mon, 29 Mar 2010 23:27:02 +0000

I completely agree with you that using QC can slow down the development. And in many cases using dedicated QA engineers for testing doesn't lead to significant improvements in the code quality. But in any case I think that exploratory testing is a must.
First, a lot of bugs with unit and functional tests passing OK can still be here. You already mentioned the messed UI, but also what about tests with different input data? If you have a complex system, you can't cover all the input choices by automated tests. And I don't mention race conditions, environment problems, etc.
Second, using the automated tests (even with 100% code coverage) doesn't mean you'll have the 5-stars code. Even with all the input choices covered (that's in turn practically impossible) it would be impossible to cover all the side effects in case of complex non-pure system.
Third, QA engineers are not "test cases running machines". With QA help you can make your product better, but with help of automated tests you can only make your code to run smoothly.
Therefore, I think the best practices are in automating the routine work and leaving exploratory/advisory tasks to the QA people.