This document critically evaluates the progress and obstacles of software process improvements since 1996. It addresses the shift in the SPI initiatives from “what” to “how”, the new integrated process improvements frameworks and the assessment model that can be used by a customer to assess different organizations using different process models. The evaluation is based on what process improvement frameworks used to be before 1996 and what it was after 1996 focusing on the Software Engineering Institute (SEI) and the International Organization for standardization (ISO) models.
This paper discusses the usage of current process improvement models like PSP, TSP, SW-CMM, CMMI, ISO – 15004 (SPICE) and their challenges. Also the possible SPI failures were identified and this includes: Management Commitment, Misconception of SPI, SPI being too expensive etc.
Software has become an integrated part of human nature such that it’s contribution into the way we live have risen tremendously because it is embedded in our home appliances, mobile phones, cars etc. Software complements and supplements the hardware industry, making the two a dependant of each other. Hardly will you see hardware without software, because hardware is driven by software.
There was a need for a defined, structured and continuous engineering process that guides the production of quality software product. In the software industry, to produce reliable and high quality software, a software organization must adopt a process framework that not only cater for the life cycle implementation of the software but also allows for continuous changes and the integration of other process models. Quality software is not composed of discrete processes; instead, it is composed of continual processes that guide the development activity (CRSIP 1999).
In other words, Process improvements are detailed blueprints for software organizations to meet current demand and complexity in the software market. Process helps an organization's workforce meet business objectives by helping them work smarter, not harder, and with improved consistency (Chrissis M.B et al. 2003). A process allows an organization to structure its software development activities, and a continuous process improvement,
allows an organization to improve on these activities. SPI methods are the instruments used for guiding and managing improvement activities in practice (Sirvio 2004).
Process improvement as a framework is constantly changing to meet the challenging software requirements that were faced by the software community at large. In the last decade, software process improvements have undergone several challenges and progress, these progresses introduces new standards of process models that address project management and project lifecycle to processes that encompass managerial activities, development process and people, software engineering and system engineering etc. Progress was identified and lessons learnt introduce new methods, tools and models.
The current trend in the SPI literature is the combination of SPI frameworks to achieve better improvements results. This paper will address the new ideas of compound process improvement framework that integrates software engineering with system engineering. Also, this paper will address the progress that have been made in the SPI discipline since 1996, focusing on the Software Engineering Institutes (SEI) the International Organization for standardization (ISO) models. It also critically evaluates these process models and the symptoms of process improvement failure giving the authors judgments.
The progresses discussed in this paper are:
organized as follows; what went wrong, shifting from “what” to “how”, the chaining of process models, the assessment model that fits various process improvement models.
CMM: What went wrong?
The Capability Maturity Model (CMM) or the Capability Maturity Model for software (SW-CMM) developed by the Software Engineering Institute (SEI) and its Version 1.1 published on February 1993, is a five level (Initial, Repeatable, Defined, Managed and Optimizing) based assessment model for measuring the capabilities of software organizations to producing quality software from ad hoc to mature software process. It is used to appraise organizations maturity as they progress in software development. It was originally modeled for the US Department of Defense (DoD) contractors.
Although the Capability Maturity Model® (CMM®) provides a powerful improvement framework, its focus is necessarily on “what” organizations should do and not “how” they should do it. (Humphrey 1999).
According to Sommerville (2001), the CMM should not be taken as a definitive capability model for all software processes. The model was developed to assess the capabilities of companies developing defense software.
CMM does not focus on technology used in development; this exempts the technicalities of the product development from the process. This in turn does not give room for product and quality measurement. Very few of the Key Process Areas (KPA) address the evolving results (i.e., the software product) and associated engineering artifacts (use-case models, design models, source code, or executable code) that capture the real target product (Royce 2002, p2).
It excludes risk analysis and resolution as a key process technology (Bollinger and McGowan 1991 cited in Summerville). The inculcation of risk assessment into process improvement was missing in CMM.
Software engineers are protective of their creativity and what works for them, thereby using a framework that tells them what to do, may not align with their software development passion.
Personal Software process (PSP)
Software process improvement community has identified the need to allow software engineers define a process framework for themselves having successfully tried the CMM, it was realized that a process model that shows software engineers how to do things was needed. PSP can be likened to CMM level 4 and 5 processes for a single software engineer. It’s a level based framework, which progresses from PSP0 to PSP3 for software engineers to measure there performance. PSP initiative is as a result of the identified bureaucracy in SPI processes that tell software engineers what to do instead of how to do it. Software engineer’s according to Humphrey (1999) are understandably skeptical about changes to their work habits; although they maybe willing to make a few minor changes, they will generally stick fairly closely to what has worked for them in the past until they are convinced a new method will be more effective.
Improvement requires change, and changing the behavior of software engineers is a nontrivial problem. The reasons for this explain why process improvement is difficult and illustrate the logic behind the Personal Software ProcessSM (PSPSM) (Humphrey, 1999). Organizations using PSP have reported significantly improved size and time-estimating accuracy, as well as reductions of 60–75% or more in the number of defects injected during development or found during unit test (Bill Thomas n.d).
Figure 1. Schedule estimating error
(Humphrey, 1999, p.6)
Challenges of PSP
Software engineers adopt what works for them; this is a problem because some engineers may change and others may decide not to change inline with the introduction of the PSP. Without proper management attention, many engineers gradually slip back into their old habits Watts S. (Humphrey 1999).
PSP can only be adopted by small software organizations.
Too much independent is given to software engineers. The Personal Software Process caters for individual software engineers rather than a collective team of engineers. This allows engineers working according to their own process which does not give room for comparing their work because their works are done in isolation.
Team Software Process (TSP)
When a team of engineers are all PSP trained and properly supported, they still have to figure out how to combine their personal processes into an overall team process (Humphrey 1999).
TSP was developed to integrate team capabilities into PSP; it leverages the power of PSP and CMM with the inclusion of team management. PSP is a prerequisite for TSP. TSP is a high-maturity project management process with many of the features of CMM‚ maturity levels 4 and 5 (Steve and Ellen 2004).
The Team Software Process (TSP) extends and refines the CMM and PSP methods to guide engineers in their work on development and maintenance teams. (Humphrey 1999)
TSP shows engineers how to perform as a team and how management should be committed to the team building activities in other to get an environment with successful team. TSP teaches engineers the process of managing their work and the power to champion their process, decision is team based. TSP integration with PSP and CMM takes the project management activities of CMM and the self measurement process of PSP coupled with team building process.
TSP was observed in Microsoft® and it was said to be successful. During TSP training, Microsoft developers reduced unit-test defects dramatically from more than
25 defects/KLOC to about 7 defects/KLOC (Kelly 2004).
Barriers of Team Software Process
TSP relies on CMM maturity levels, a level 1-2 organization cannot use TSP successfully, and this is as a result of PSP/TSP carried out on Boeing.
TSP relies more on management support to function well.
PSP is a prerequisite for TSP, this means before an organization can adopt TSP successfully; all its engineers must be PSP certified.
A chain of three process models: the Integrated Initiatives
Organizations that were not typically software companies—such as financial institutions, car manufacturers, airplane manufacturers, and insurance companies—find that much of their business relies on software. Software is often what differentiates them from their competitors. The problems these organizations address today involve both software and systems engineering (Chrissis M.B et al. 2003).
Different process models answers different business areas, the bulkiness and the disintegration of process documents that addresses different part of an organization process was identified by the Software Engineering Institute (SEI). This lead to the creation of the Capability Maturity Model Integration (CMMI) with its version 1.02 published December 2000.
The SEI introduced the CMMI® as an integrated model for software engineering, system engineering and integrated product and process development. There was a need for these various models working together. The use of multiple models has been problematic (Chrissis M.B et al. 2003). The overhead of using different models to address different process areas was identified and this brought about the CMMI. Most available improvement approaches focus on a specific part of the business and do not take a systemic approach to the problems that most organizations are facing (Chrissis M.B et al. 2003).
CMMI addresses the product lifecycle from the beginning to the end, and at the same time bridging the gap between procedure, people, methods and tools. The following models were integrated into the CMMI framework:
Capability Maturity Model for Software (SW-CMM) v2.0 draft C.
Electronic Industries Alliance Interim Standard (EIA/IS) 731, System Engineering Capability Model (SECM).
Integrated Product Development Capability Maturity Model (IPD-CMM)
Why the Integration Approach
It is becoming impossible to separate software from hardware, because the two industries complements and supplements themselves.
There was a high demand for software intensive system in the market.
Benefits of CMMI
Linking management objects with software and systems engineering models.
Leveraging from additional areas of best practice (e.g., measurement, risk management, and supplier management) (Ferguson, 2003).
Visibility into additional organizational functions critical to their products and services (Ferguson, 2003).
Challenges of CMMI
CMMI is not a focused framework; in that it is composed of different SEI’s process improvement models that address the areas of Project management, engineering, and support and process management, this leave the models documentation bulky.
“What is the basis for using CMMI as a model, has it been proven to improve businesses?”. (Rold and Fleisher).
What are the relationships of the many process models adopted by CMMI?
CMMI inherits all the activities of the CMM; this means the model is still a reference to the staged representation of CMM, but is there any part of CMMI that helps organizations that was in a level in CMM to continue in the same level in CMMI?
One Assessment model now fits all
With the introduction of the ISO/IEC 15004 (SPICE), organization does not need to change their process models to conform to assessment standards, because the ISO\IEC 15004 caters for an integrated assessment and capability determination for the wide ranges of process models. ISO 15004 can assess software organizations with different software models i.e. software organizations with SW-CMM can be assessed against a software organization with Bootstrap.
SPICE is a world class collaborative effort by International Organization for standardization (ISO), International Electrotechnical Commission, Joint technical committee etc for assessing software process improvement initiatives.
Obstacles of process Improvements
The industry focus is more on the level based rating, rather than the continuous process improvements. Impression is “pass the test”, get the rating and be done with
this initiative (Rold and Fleisher, n.d).
What are the evidence that process improvements leads to high Return On Investment (ROI)? According to Rold and Fleisher, conflicts over “what is the basis for using CMMI as a model, has it been proven to improve businesses?”. You cannot prove ROI unless you start the journey and begin measuring performance.
Process improvement adoption is becoming expensive with the rapid introduction of newer version of SPI models; does the customer pay for the product and also the improved processes too?
What are the testimonies that organizations in the same CMMI level rating will produce the same quality software? Does adopting CMMI as a benchmark really means success and high capability in engineering software.
Process Improvement methodologies explosion. Different SPI best practices defined by different bodies, we have the US SPI standards, we also have the European SPI standards, there is no standard unification. All standard should speak the same language.
Possible SPI failures Lack of Commitment to SPI
The Lack of commitment to software process improvement will lead to process immaturity, inconsistent process activities, missing productivity measurement and improper planning because when commitment is missing there will be no Proper documentation, planning and maintenance of process. Effective software process Improvement will not start until management insists that product development work be Planned and properly managed (Humphrey 1998).
“Process improvement for its own sake will soon die. It must address strategic organizational imperatives if it is to be successful.” (Waina n.d).
Management may be too scared to commit resources and time to software process, the shorter route to software process improvements was to rush and adopt methodologies like CMM – CMMI – ISO900-3 that may not directly provide answers to their problems because problems and culture are different across processes and they shouldn’t be built around the SPI models. Software development policies should reflect current organizational commitments rather than "CMM-compliance. (Pat O’Toole, 2002).
Process improvement for its own sake will soon die. It must address strategic organizational imperatives if it is to be successful (Waina n.d). Organizations need to address what should be improved and how it should be improved because of the deficiency in their commitment and absence of measurable goals. It its more talk than practices but when problems arises at the end management wait too late to address issues that could have been avoided if proper process documentation were carried out “The most common approach for process improvement we have seen during the last 10 years is to document all processes.( Sarkry M and Potter N, 2000). The following are problem areas when management’s commitment is too low; poor software quality measurement, poor defects identification, software is not reliable, risk assessment is not observed, and no proper feedback is given. The goal of commitment to process improvements includes better time management, quality product, risk reduction and well documented process. Organizations cannot mature in process when there are no benchmarks to follow because an undocumented process is a barrier. The process may be superb, but it is an ad-hoc process, not a defined process (Tyrrell n.d).
Overhead in process improvements:
There are times when deliverables are expected within a given time constraint, to meet customer deadline. During this period, management may consider process or its improvements as a time killer activities that will only disrupt development time and targets.
CMM is not a religion. You should use it as a guide to help you focus your improvement energies where they will likely pay off, rather than simply racing up the maturity scale as fast as you can. (Wiegers n.d)
Misinterpretation of industry endorsed process models:
CMM – CMMI – ISO9003 – Six Sigma etc are process benchmarks not a total solution for all organizations process improvements; there may be a misconception about the use of these methodologies to improve the organization processes.
The Learning curve is high
They will react against things because they fear the learning curve linked with the introduction of new things (in software engineering and also everywhere else). But if we do not learn new things and try to improve ourselves and the way we work, we would not have nice cars, mobile phones, great software to support our business, etc (Global Knowledge source for software development professionals, 2002)
The SPI part of software engineering is still evolving, as far as software evolves, the development methods, software patterns and practices and process improvements also will. There is no final destination of the software engineering process improvement, because it has to address the current issues observed in the industry today, this makes process improvement dependants on the current practices in software engineering.
We have seen the lessons learnt in the history of CMM and how these have lead to several other process improvement frameworks in Software Engineering Institute. Progress was introduced after the several feedbacks from the users of CMM and these lead to the adoption of PSP/TSP, the SW-CMM and also the CMMI.
After examining the different approaches discussed in the main body of this paper, a question comes to mind, does integrating different processes really solves the problem of an organization, especially smaller organizations that does nothing but software, how will an organization that does only software invest a lot into a framework like CMMI without the need for system engineering? Are these frameworks really addressing current issues of making software engineering a discipline?
We are experiencing the shift in the way processes Improvements were done, but we have several improvement methodologies defined by different bodies, although the SPICE assessment model can be leveraged to assess different organizations with different SPI models, but for software engineering to retain its consistent engineering approaches, there should be an SPI model that can be used to assess all software engineering organizations both home and abroad no matter what the processes used.
The more critical software are developed today, the more process Improvement are introduced. There was an impression that progress in process improvement is higher than the development tool. Organizations are now getting to know that, the process that are used in building software, is as important as the software.
1. Chrissis M.B et el, (2003). Introduction to CMMI. [Online]. Available from: http://www.informit.com/articles/printerfriendly.asp?p=98146&rl=1. [Cited Dec 2006].
2. Computer Resources Support Improvement Program (CRSIP) Revision 3.0 March 31, (1999). A Gentle Introduction to Software Engineering [Online]. Available from: http://www.stsc.hill.af.mil/resources/tech_docs/GISE.DOC.
3. Ferguson J.R, (2003). CMMI Overview [Online]. Available from: http://www.ndia-tvc.org/SESEC/Presentations/SE2%20Conference%20PDFs/DAY_1/D1_Session_1/S1B3_Ferguson.pdf. [Cited Dec 2006].
4. Global Knowledge source for software development professionals (2002). Methods and Tools [Online]. Available from: http://www.methodsandtools.com/PDF/dmt0302.pdf. [Cited Dec 2006].
5. Humphrey W.S. (1998). Three dimensions of process Improvement – Part 1: Process maturity [Online]. Available from: http://www.stsc.hill.af.mil/crosstalk/1998/02/processimp.asp.
6. Humphrey W.S. (1999). Pathways to process Maturity: The Personal Software Process and Team Software Process [Online]. SEI Interactive. Available from: http://www.sei.cmu.edu/news-at-sei/features/1999/jun/Background.jun99.pdf. [Cited Nov 2006]
7. Jainiszewski S and George E, (2004). Integrating PSP, TSP, and Six Sigma [Online]. Available from: http://www.asq.org/pub/sqp/past/vol6_issue4/sqpv6i4janiszewski.pdf. [Cited Nov 2006].
8. Kimberland, K. (2004). Microsoft’s Pilot of TSP Yields Dramatic Results. [Online]. Available from : http://www.sei.cmu.edu/news-at-sei/features/2004/2/pdf/feature-1-2004-2.pdf. [Cited Nov 2006].
9. Leeson P, (2005). Use CMMI correctly and the business benefits. [Online]. Available from http://www.bcs.org/server.php?show=ConWebDoc.3051. [Cited Nov 2006].
10. Pat O’Toole, 2002. Pat O’Tooles’s Dos and Don’ts of Process Improvements: Do Establish Organization Policies, Not CMM Policies [Online]. Available from: http://www.informit.com/articles/article.asp?p=27124&rl=1 [Cited Nov 2006].
11. Rold J and Fleisher N. CMMI as a Process Improvement Tool [Online]. Available from: http://proceedings.ndia.org/587J/rold.pdf. [Cited Dec 2006].
12. Royce W, (2002). CMM vs. CMMI: From Conventional to Modern Software Management [Online]. Available from: http://www-128.ibm.com/developerworks/rational/library/content/RationalEdge/feb02/ConventionalToModernFeb02.pdf. [Cited Dec 2006].
13. Sarkry M and Potter N, (2000). Goal-Problem Approach for Scoping an Improvement Program [Online]. Available from: http://www.stsc.hill.af.mil/crossTalk/2000/05/sakry.html. [Cited Dec 2006].
14. Sirvio S.K. (2004). Development and Evaluation of Software Process Improvement Methods [Online]. Available from: http://virtual.vtt.fi/inf/pdf/publications/2004/P535.pdf.
15. Sommerville I, (2001). Software Engineering Sixth Edition. Ch 25.
16. Thomas B. Delivering on the Promise of Process improvements [Online]. Available from: http://www.sei.cmu.edu/news-at-sei/features/1999/jun/Spotlight.jun99.pdf. [Cited Nov 2006].
17. Tyrrell S,. The many dimensions of the software Process [Online]. Available from: http://www.acm.org/crossroads/xrds6-4/software.html. [Cited Nov 2006].
18. Wania R.B. Five Critical Question in Process Improvement [Online]. Available from: http://www.chips.navy.mil/archives/01_summer/five_critical_questions_in_proce.htm. [Cited Dec 2006].
19. Wiegers K.E. Misconception of the Capability Maturity Model [Online]. Available from: http://www.processimpact.com/articles/miscon.html. [Cited Dec 2006].
20. Wilson H, (2003). Capability Maturity Model Integration [Online]. Available from: http://www.dtic.mil/ndia/2003technical/wilson.ppt. [Cited Dec 2006].