Systematic Software Reuse:
Architecture, Process and Organization are Crucial

Martin L. Griss, Laboratory Scientist
Software Technology Laboratory, HP Laboratories
Tel: (415) 857-8715
Email: griss@hpl.hp.com

(Fusion Newsletter, Oct 1996)

Introduction

Based on many years researching software reuse and consulting with HP and customers' reuse projects, I have found that the following ten guidelines are key to ensuring systematic reuse:

This article considers why reuse is so important to competitiveness, and explains and illustrates these ten guidelines. Follow-up articles will begin to address extensions to Fusion for systematic reuse.

What is reuse, and why do you want it?

Reuse is based on a simple, well-known idea. When we build a new firmware or software application, we should use previously developed software components. Doing so will save the cost of developing, testing, documenting and maintaining multiple copies of essentially the same software.

These benefits can be quite significant. Reuse programs within AT&T, GTE, Ericsson, NEC, Toshiba, and HP itself, demonstrate that time to market can be improved by a factor of 1.5 to 2.5 or more. Software or firmware that used to take 24 months or longer to produce, can take only 6 months or less to develop with reuse. Quality can be increased by 5-10 times, and costs of development and maintenance can be reduced. And there are other benefits. Reuse of standard modules can increase the interoperability and consistency of products. Appropriate components and supporting tools can enable a professional services organization to rapidly produce ``almost custom solutions'' cost effectively.

But reuse is not a silver bullet. This simple idea is quite complex in practice. It is not enough to gather interesting pieces of software into a library and offer them to people to reuse. Components have to be carefully designed and carefully developed so that they are high quality and work well together. They have to be documented well so that reusers can understand them. Components have to be carefully chosen so that this extra investment will be repaid by significant reuse. This works best when reusing components between members of a product line or product family (or product ``domain'').

The software development process needs to be changed to support the development and reuse of components. People need to be trained in the use of the components and library systems, in the new technologies and the process. Some people need to take on new roles, such as component developers or librarians. New technologies and methods, such as objects and frameworks, generators and Wizards, require new metrics, and new development models. Mechanisms to fund component development and reuse need to be established. The organization needs to gain experience with these changes.

What seemed like a simple idea has become complex, and needs management attention and organizational support. An incremental plan must be developed and followed. Nevertheless, systematic software reuse is still widely believed to be best way to dramatically improve the software process.

Lessons from HP reuse projects

I have observed the evolution of several HP reuse programs closely, and learned a lot from them. Started around 1990, a Test and Measurement reuse program developed a small number of large components, which fit together within a test instrument architecture targeted to cover about 30% of the firmware within a typical instrument. The reuse team developed an interesting social process and cross-divisional organization to ensure that projects in several divisions were able to share the cost of developing the architecture, components and tools. They piloted the use of the components to ensure that they worked well. As a consequence, the reuse components have been used in many products (17 to date, 10 more underway). In some, they have been used as a "domain-specific kit"[3,4], using all of the components, architecture and generator tools. In other cases, only a few of the components were used. Work continues today within the reuse center to develop new components.

An article in the April issue of the Fusion Newsletter provides an excellent discussion of experiences developing a reusable OO framework using EVO Fusion [5]. Many organizational, management and technical issues had to be addressed. Some were anticipated, while others were discovered along the way. It is important to identify these risks early, and plan to handle them systematically.

Systematic reuse needs a systematic approach

By systematic reuse, we mean an institutionalized organizational approach to product development in which reusable assets are purposely created or acquired, and then consistently used and maintained to obtain high levels of reuse, thereby optimizing the organization's ability to produce quality software products rapidly and effectively.

Over the last ten years, software reuse researchers and practitioners have learned that to succeed at systematic reuse requires careful attention be paid to both technical and non-technical issues[6,7,8,9]. The non-technical issues are more pervasive and complex than at first realized.

In order to get large amounts of reuse, perhaps division or group wide, we need a systematic approach. Without a balanced focus on people, process and product issues, we will not succeed at managing the scope and magnitude of the changes and investment needed. Simply creating and announcing a reusable class library will not work. Without a ``reuse mindset,'' organizational support, and methodical processes directed at the design and construction of reusable assets, the reuse investment will not be profitable. Research and practice have been directed at the development of assessments, guidelines, processes, models, methods and prescriptions that promise more cost-effective reuse.

This requires a significant effort to change culture, organization, and a multitude of other factors. These changes are quite radical, and more widespread than those incremental and ongoing changes associated with CPI (continuous process improvement). Reuse is a business issue. We have to change the way we view software at a fundamental level. It must be viewed as an organizational asset, to be invested in, improved, and leveraged effectively and consistently

Often, sweeping changes in the software development organization will be needed to institute large-scale, systematic reuse. Organization changes separate asset creators from asset users who build applications. Business changes fund product family design and construction with the goals of improved time to market and cost for several related products, investing in reusable assets. Software process changes manage and use a reusable asset repository. Many cultural and management changes create new, unfamiliar roles. The magnitude of the changes, and the issues encountered, are similar to those encountered when doing Business Process Reengineering. We find that the systematic methods and skills used to design and implement such changes can also be applied to software organizations.

Incremental adoption of reuse

Experience shows that it is best to adopt a large scale reuse program in stages, making organization and process changes in a series of incremental steps. Usually, an organization becomes ready for a transition when the business issues it faces motivate a change. Figure 1 illustrates some of the steps that a typical organization may progress through. This figure is based on a synthesis of experiences at several organizations within HP and elsewhere, and as such, is both a summary of history, and perhaps a prescription of how to proceed.

Figure 1. The incremental adoption of reuse is driven by compelling business needs

What this means is that an organization must move incrementally, must start by focusing on a narrow domain and only a part of the complete product architecture. This is best done by using a series of reuse pilots. As the reusable software (components and framework) is being developed, the organization must gain and retain management support, must customize process and organization, and must ensure that the reusers are involved in the dialogue early. In the process, one can not ignore social and cultural realities. We need to consciously address and overcome several issues related to mistrust, perceived loss of creativity and independence. It won't happen fast enough without specific plans and attention, but it is also important to note that most organizations can't move instantaneously either. Certain of the changes take time, perhaps several years.

What we observe is that the benefits due to reuse, such as improved time to market, or higher-quality systems, or lower overall development costs, increase as the levels of reuse and the sophistication of the reuse program increase. Furthermore, it takes time, investment and experience with reuse in your organization to get to those levels of reuse. So what seems to happen is that the organization moves from stage to stage as it consolidates its gains and feels the pressure for more improvement. For example, the transition from no reuse, to informal reuse (sometimes called leverage or cloning) in which chunks of code are copied, adapted slightly and then incorporated into the new system, occurs when developers:

This strategy works--for a while. Development time is reduced, and testing is often less tedious than with totally new code. But as more products are developed using this approach, maintenance problems increase. Multiple copies of the software, each slightly different, have to be managed. Defects found in one copy have to found and fixed multiple times. This often leads to a ``black box'' code reuse strategy, in which a carefully chosen instance of code is reengineered, tested and documented for reuse. All projects are then encouraged or required to use just this copy without modification.

This works well for a while, and then the issue of dealing with changes to satisfy an increasing number of reusers arises. Should everyone be ``forced'' to use only the standard version? Should multiple versions be maintained? Should adaptation be allowed? Who decides? Should test files, designs and specification files also be reused? All of these issues lead to the creation of a component management process (similar to a preferred parts process), with supporting organization. People need education and help in using these components.

Beyond this point, to get higher levels of reuse, and more coverage of the life-cycle, it is important to architect the components and the systems that will use them. This is the only way to ensure that components fit together. The development and use of a common architecture involves even more organizational commitment and structure.

The reuse process and organization

Among the many issues which must be addressed are the software processes to be followed, and the structure and management of reuse organizations. A well defined software development process is crucial to effective reuse. Since no single software process model is ideal for all situations, different software life-cycles describes specific roles, activities, methods, metrics, responsibilities and deliverables of the developers and managers. A reuse process makes explicit guidelines that will ensure successful reuse. These include explicit separation of component developers (creators) and reusers (utilizers), and explicit reuse-oriented management and support. Effective reuse requires significant organization changes, typically spanning several teams within the company. Such sweeping changes can only be accomplished with informed and explicit management leadership.

Figure 2 shows a basic reuse process model. To explicitly support reuse-enabling guidelines, the model has four distinct process elements:


Figure 2. Ideal reuse process and organization

Specific metrics are needed to manage the creation, support and utilization of reusable software, helping utilizers understand the value of the assets, and helping creators prioritize investments.

In a reuse organization operating according to the above process, there is tension between product development objectives (especially capabilities and schedule) and asset development and evolution (especially long-term reusability and quality). Ideally, all members of an organization have a shared reuse mindset, allowing them to operate from knowledge of the process and desired balance between product and reuse objectives. However, in practice, reuse only succeeds within a structure with separate creator and utilizer teams, and high-level management leadership and support.

I have found that the most effective reuse organization structure has four distinct kinds of teams, corresponding to the processes shown in Figure 2. These can sometimes be combined into a smaller set of organizations:

There are several issues that compete as a reuse organizational structure is set up. Each structure favors some stability and connections at the expense of some another. The reuse management team and process must be vigilant to ensure appropriate balance. While some structures are easier to establish initially, they are less effective if their creators and supporters do not have full-time, visible and official responsibility for reusable assets alone. Ideally, a single manager is able to coordinate interactions between teams. Whether creators and utilizers are in the same organization or not, they need leadership empowered to help meet both product objectives and organization-wide reuse objectives. If the managers of the other teams do not report to the reuse manager, or a manager close to the reuse effort, some other mechanism is needed to resolve priority and conflict issues, and provide funding for asset creation and support.

On one hand, if creators are embedded as members of product teams or report to the product team manager, they too often get redirected to assist with urgent product construction at the expense of asset creation. On the other hand, independent asset creation teams can easily neglect their responsibility to meet the needs of utilizer teams. They may then prioritize their efforts to produce assets, libraries and tools that are poorly matched to needs, even though technically excellent. Perceived as ``overhead'' or ``unresponsive,'' they get frustrated and either leave the organization, or try to become ``independent component vendors'' to sell the assets or make their own products.

Reuse efforts can fail in a variety of ways without appropriate structure and informed management. The process and organization must assist a manager in anticipating and detecting these situations soon enough to correct an imbalance.

Conclusion

I will continue my discussion in future columns. I hope I have whet your appetite sufficiently so that you might enjoy looking at the introductory article in the September 1994 IEEE Software special issue on systematic reuse [6], or some of my Object Magazine columns [1,4]. I hope that this column will both provide information and also stimulate discussion.

Biography

I have been working in software engineering technology and process research since 1972, first at the University of Utah as an associate professor, and then at HP Labs since 1982 as industrial researcher, prior director of the Software Technology labs and most recently as a Lab Scientist. Since 1987 I have focused on software reuse and object-oriented software engineering. I spent two years with HP SWI, helping establish its work on software process and reuse.

I currently write a column on software reuse for the Object Magazine, and frequently give tutorials on software reuse for the HP SESD Software Development Executive forum, HP PSO, and at object and reuse conferences. For several years I have been working closely with Ivar Jacobson, of Rational Software Corporation, on extending his use case driven object-oriented software engineering method (OOSE) to support reuse. We are completing a book on software reuse. I lead an HP Labs project on component-based, domain-specific software kits, using Visual Basic, Visual C++, OLE and NT to prototype a distributed measurement software development environment. We recently developed a two-day Systematic Reuse Workshop that is available to customers through the HP Professional Services Organization (PSO).

References

  1. ML Griss, Systematic Software Reuse - Objects and frameworks are not enough, Object Magazine, February 1995.
  2. Ivar Jacobson, Martin Griss and Patrik Jonsson, Software Reuse: Architecture, Process, and Organization for Business Success, Addison-Wesley 1997 (to be published).
  3. ML Griss and K Wentzel, Hybrid Domain Specific Kits, Journal of Systems and Software, Dec 1995.
  4. ML Griss and RR Kessler, Building Object Oriented Instrument Kits, Object Magazine, May 1996.
  5. R Malan and T Dicolen, Risk Management in an HP Reuse Project, Fusion Newsletter, April 1996
  6. W Frakes and S Isoda, Systematic Reuse, Special Issue IEEE Software, May 1994.
  7. J Hooper & R Chester, Software Reuse Guidelines and Methods, Plenum, 1991.
  8. EA Karlson, Software Reuse: A holistic approach, Wiley 1995.
  9. W Schäfer, R Prieto-díaz and M Matsumoto, Software Reusability, Ellis Horwood, 1994.