Many of the practices adopted in software development have been based on anecdote and folklore, Rally Software director of analytics Larry Maccherone told iTWire, but the adoption of big data technology allows choices to be based on what really happens.
Mr Maccherone said that when he was at Carnegie Mellon University, the Agile movement had largely rejected metrics. He subsequently joined Rally Software, which had a large quantity of development-related data.
Once that data was anonymised and its use cleared by the customers that provided it, he began looking for measures that correlated strongly with development performance.
If work in progress becomes too narrow, productivity suffers as work becomes blocked on particular tasks, for example because of the unavailability of specialist staff. When that happens, other staff are left idle.
He also found that although the received wisdom was to break up teams at the end of a project, better performance resulted from maintaining stable teams. "People learn how to work together" as a team, he said, but that process takes time.
It's also important that each person should belong to exactly only one team, otherwise switching between contexts takes its toll. It doesn't make much difference whether an individual is moved between teams daily, weekly, monthly or quarterly, "it's dramatically bad to do it at all," Mr Maccherone said.
The suspicion is that being committed and accountable to one team makes a big difference.
The original study looked at five areas of activity (and a paper describing the results can be found here [PDF]), but has now been repeated on a broader scale with 55 variables under consideration.
Mr Maccherone noted that the latest study found that Australian and New Zealand developers outperformed their peers in North America, South America, Europe, Africa, Asia and India. They were up to 60% more productive, and achieved one-seventh of the defect density.
The next best performers were in North America while India was at the bottom of the pack with "the absolute worst performance... and the worst quality," he said.
Page 2: More findings and an example
Other interesting findings were that reducing the iteration length improved responsiveness and productivity at the cost of quality (possibly because reducing the time available leads to less review and testing activity), and that the best reason for adopting Agile is "my boss told us to" (perhaps because a high-level decision means the required structural, process and cultural changes are more likely to be put into place to support the change).
"We're using big data techniques" to get at these understandings, he stressed.
Companies can get the benefit of their fellow Rally customers experiences in this regard by adopting the Rally Insights product that is currently in beta.
"Without data you're just shooting in the dark," he added, but gave the familiar warning that correlation does not necessarily mean causation. That is, just because 'width' is inversely correlated with quality, reducing width will not necessarily and automatically increase the quality of the team's work, as the current low quality could be the result of some other factor such as a inadequate skills.
One example of the way a Rally customer used data to justify a change of practices involved "a large toy manufacturer in Denmark". Using Rally software to explore productivity revealed a 107% improvement for one group over a two-month period.
What happened was that this group was working while most of the other people were on holiday. This meant the open-plan office was quieter, and there were fewer meetings.
So the company is moving its development teams to separate rooms, and instead of meeting with as many as four stakeholders, the teams now meet with just one, and he or she is responsible for disseminating the information to the other stakeholders.
The data from this change will be used to calculate the monetary value of the productivity improvements to support other similar investments. Mr Maccherone said.