Friday, October 31, 2008

Why software estimation is so hard

I once worked with a guy at JPL who had an uncanny ability to give extremely accurate estimates. You could give him a problem, he could sketch it on the whiteboard, then give you an estimate. He would usually use a 10% margin of error (say, 80 hours +/- 8). And he was always dead-on. I thought it was amazing, and assumed it was because he was such a skilled programmer.

I noticed something once when I was admiring some of his code. I saw some code that he'd written that I knew was available from some pretty common third party libraries. I've always been a lazy programmer--so I steal a lot. I'm always on the lookout for cool libraries and frameworks that I can use. I'd rather not write code somebody else has already written. But Scott wrote pretty much everything he needed (and only the things he needed--he never tried to build-out for a future that might not arrive).

Fast forward 15 years: I've been spending my spare time this week programming. I've decided that every manager should be required to write a certain amount of productive code every so often to keep them grounded in the reality of the technical problems they're supposed to be managing day-to-day. As I reflect on the things that have burned most of my time in the past week I realize that it's usually not my code, but my dependencies on other libraries and frameworks. The very libraries and frameworks I leverage to amplify my productively are also the most unpredictable. If it was only my code, I would be in complete control. If it's some other library that works as expected, I zoom ahead of my scheduled progress. But when I don't know I need to upgrade a third party library to get the behavior I require, I do things like spend time trying to figure out why the library doesn't work the way I expect, then spend time trying to find a solution, then spend time trying to figure out how to work around it, then spend time trying to unravel my stack to get back on track..

Today we don't write modern software without leveraging powerful and complex platforms, frameworks, and environments. This is what makes software estimation so hard--these areas outside our code are so unpredictable.

This is why I'm a big fan of the XP style estimation that uses a "load factor" and "velocity" in estimation. These techniques pragmatically adjust for the uncertainty of the factors surrounding your code.

Friday, October 24, 2008

Software Engineer in Test

Since Mike Kirby (one of our Whitebox test team leads) doesn't have a blog, I thought I'd post some of our "Software Engineer in Test" training materials.

These are the subject areas and techniques we expect our Test Engineers to understand and the skills we want them to have to effectively perform whitebox testing in our group.

Software Engineer in Test level 1:

Week 1 Introduction
Read the following chapters from _Testing Object Oriented Systems_:
Chapter 3 "Testing: A Brief Introduction"
Chapter 4 "Testing Object-oriented Software" sections 4.1.2 -4.1.3 "Fault Model"
Chapter 4 sections 4.2.1 - 4.2.5 "Effects of the Paradigm"
Chapter 4 sections 4.4 "Coverage Models"
Chapter 5 "Test Models"
Chapter 8 "A Tester's Guide to UML"
Chapter 10.1 "Class Test and Integration"
Chapter 10.2 "Preliminaries" (Code coverage, Path Sensitization, Domain Testing)
Chapter 16 "Test Automation" 4pgs
Chapter 17 sections 1-3 & 6-7 "Assertions"
Chapter 19 sections 1-3 "Test Harness Design"

Week 2 Source Control Management
Create a SVN repository on your linux box, and check out sample the sample code to work through the examples.
Read chapter 2 "Basic Usage" and chapter 4 "Branching and Merging" of the subversion book.
Test Development repository structure:
(links to our internal wiki)

You will be asked to give a live demonstration of the following repository activities:
How to create branches
How to rollback changes
How to merge changes
How to resolve conflicts
How to create and apply patches
Demonstrate a 3 way merging using KDiff3

Week 3 Application Runtime Analysis
Read Valgrind "Quick Start Guide"

Read chapters 1-3 "Intro", "Using Valgrind" , "Using Valgrind - Advanced Topics"
Valgrind -

Read chapter 4 on Valgrind-memcheck
Build, run sampeles and find the memory leaks
Create/load Valgrind-memcheck supressions files

Read chapter 6 on Valgrind-callgrind
Build, run sampeles and generate callgraphs
Convert callgraphs to images

Read chapter 5 on Valgrind-cachegrind
Generate cachegrind output files
Review cachegrind output with KCachegrind

Week 4 Quality characteristics for Whitebox

Week 5 Non-Whitebox Essentials
Syslog and logger and Syslog-ng

SNMP + Cacti
Probing the PROC filesystem

(link to our internal wiki)

Pelco Package utilities
(link to our internal wiki)

Using Packet Sniffers

Syseng + Pulse + Crucible + Fisheye + Jira + Pelcoverse + PPM
(link to our internal wiki)

Endura Developer Reference
(link to our internal wiki)

Software Engineer in Test level 2:

Read the following chapters from _Testing Object Oriented Systems_:
Chapter 10.3 "Method Scope Test Design Patterns" (Recursive,Polymorphic)
Chapter 10.4 "Class Scope Test Design Patterns" (Invariant Boundaries, non/Modal Class Test)
Chapter 10.5 "Flattend Class Scope Test Design" (Hierarch Test)
Chapter 13.1 "Integration in Object-oriented Development"
Chapter 13.2 "Integration Patterns"
Chapter 14.3 "Implementation-specific Capabilities" (Configuration,Compatability,Performance,Fault Tolerance)
Chapter 15 "Regression Testing"
Chapter 17 section 17.4.5 "Assertions - C++"
Chapter 17 section 17.5 "Assertions - Depolyment"
Chapter 17 section 17.8 "Assertions - Notes"

Presenting Quality sub-characteristic:
Time behavior
Resource utilization
Fault tolerance

Fundamental Design Principles
Maximize cohesion, minimize coupling
Subclasses should be substitutable
Favor association over inheritance
Base the design on the invariants
Encapsulate design decisions
Minimize overall complexity

High Leverage Patterns
Factory method
Abstract factory
Bridge / Strategy

How to write Evil, untestable code

Writing Testable Code

The Difference Between Mocks And Stubs

Choosing Between The Differences
(Fixture setup, Test isolateion, Coupling, Design style)

Test Driven Development


Dependency Injection | Seperate application instantiation from application logic


The Invisible Branch: Differences between statement coverage and branch coverage

Google C++ Testing Framework

Must Pass and Would be nice to Pass Tests

Code Coverage

Too Many Tests

Test Messages

Automating tests vs. test-automation

Performance Testing

Testing Applications and APIs

Copeland’s GTAC Video

Stubs Speed up Your Unit Tests

Better Stubbing in Python

Non-Whitebox Essentials
Static and Dynamic Libraries

(links to internal wiki)


Tools and Frameworks
Python + SWIG
Java UPNPLib
Gcov + Lcov

Static Analysis
Using Understand
Review complexity
Generate static analysis reports
Coupling + Cohesion

Doxygen Commands

Doxygen Call and Caller Graphs
Doxygen Dependency Graph

GNU Binary Utilities

RPM Packaging
(links to internal wiki)

Pulse personal builds
(links to internal wiki)

(links to internal wiki)

Sustaining Log Parser
(links to internal wiki)

Stream Monitor
(links to internal wiki)

Dynamically Linked Application Profiling
(links to internal wiki)

Integrating Existing Code
(links to internal wiki)

Engineering Process Improvement
(links to internal wiki)

ISO9126 Quality Characteristics
(links to internal wiki)

Multicast Tunneling
(links to internal wiki)

Test API
(links to internal wiki)