Sunday, April 8, 2007

Bugs per lines of code

books that reports or web site that addresses the topic of bugs per line of code

The book "Code Complete" by Steve McConnell has a brief section about error
expectations. He basically says that the range of possibilities can be as
follows:

(a) Industry Average: "about 15 - 50 errors per 1000 lines of delivered
code." He further says this is usually representative of code that has some
level of structured programming behind it, but probably includes a mix of
coding techniques.

(b) Microsoft Applications: "about 10 - 20 defects per 1000 lines of code
during in-house testing, and 0.5 defect per KLOC (KLOC IS CALLED AS 1000 lines of code) in released
product (Moore 1992)." He attributes this to a combination of code-reading
techniques and independent testing (discussed further in another chapter of
his book).

(c) "Harlan Mills pioneered 'cleanroom development', a technique that has
been able to achieve rates as low as 3 defects per 1000 lines of code during
in-house testing and 0.1 defect per 1000 lines of code in released product
(Cobb and Mills 1990). A few projects - for example, the space-shuttle
software - have achieved a level of 0 defects in 500,000 lines of code using
a system of format development methods, peer reviews, and statistical
testing."

------------------------------------------------

Vinnie Murdico
Software with Brains, Inc.
SWBTracker - Value-Priced Defect Management Software http://www.softwarewithbrains.com

6 comments:

Unknown said...

Any chance you have also collected metrics on the Cost to achieve a specific kloc level?

The reasons this matters is that it relates to the number of Security Defects in code and the cost to control such events.

Knacho said...

In the real world of in-house testing, you need to make your own expected "bugs by code line" number. Whit this you can define SLA´s for the developers or factories you use.

amar said...

i dont have any such metrics, but when the build is stable in functionality its better to initiate for security testing in parallel to the development. That can actually save time in terms of cost rather hen waiting completely for a finished product

pete keys said...

This is very interesting. The number of defects found is going to change through the stages, unit test, system integration test, acceptance test. Are there any metrics on the reduction through phases?
Also the number of defects is proportional to the test coverage /thoroughness. Are there any metrics on the number of tests per KLOC.
I am working on the first iteration of a new product with 430KLOC and expecting a defect density of less than 1. At more than 0.5 I would be disappointed and my exit criteria is 0.01: t
he defects rates quoted in this article seem very high. Perhaps the severity of a defect needs taking into account: a defect that causes a crash and data loss is different to a menu entry typO.

pete keys said...

This is very interesting. The number of defects found is going to change through the stages, unit test, system integration test, acceptance test. Are there any metrics on the reduction through phases?
Also the number of defects is proportional to the test coverage /thoroughness. Are there any metrics on the number of tests per KLOC.
I am working on the first iteration of a new product with 430KLOC and expecting a defect density of less than 1. At more than 0.5 I would be disappointed and my exit criteria is 0.01: t
he defects rates quoted in this article seem very high. Perhaps the severity of a defect needs taking into account: a defect that causes a crash and data loss is different to a menu entry typO.

Peter DeLorme said...

I, too, am looking for more depth in the analysis. For example, the defects per lines of code tested this month, so that I can trend the number and see that quality is improving toward benchmarks at say the mid-point and at the end of each test phase. Do you have any hints on where to find such metrics for .Net development?