Hybrid domain-specific kits for a flexible software factory
Martin L. Griss and Kevin D. Wentzel
[Proceedings of SAC'94, Reuse and Reengineering Track, Phoenix, AZ, Mar, 1994]
As part of HP Laboratories research into a systematic process for domain-specific reuse, we are exploring the notion of domain-specific kits. Kits are comprised of compatible, domain-specific components, frameworks and glue languages, supported by a variety of technologies and tools, such as domain-specific languages, builders, generators and domain-tailored environments. We are particularly interested in hybrid kits, which combine both generative and compositional reuse, and domain-specific language tools to support the generative part. This paper describes our current work and directions.
Keywords: Reuse, kits, builders, generators, domain-specific languages, hybrid reuse.
Systematic software reuse is a key business strategy that software managers can employ to dramatically improve their software development processes, to decrease time-to-market and costs, and to improve product quality. Effective reuse requires much more than just code and library technology. We have learned that careful consideration must be given to the way reusable workproducts must be designed and packaged, and to a systematic integration of people, process and technology issues.
At Hewlett-Packard Laboratories, we have initiated a multi-disciplinary research program to investigate these issues and to develop better methods for domain-specific, reuse-based software engineering. Our approach to these is a vision based on domain-specific kits and flexible software factories.
This paper will summarize some of our work on software reuse, domain-specific-kits and flexible software factories, focusing on hybrid reuse, domain-specific languages and related issues.
Systematic software reuse
The concept of systematic software reuse is simple: the idea of building and using "software preferred parts." By building systems out of carefully designed, pre-tested components, we will save the cost of designing, writing and testing new code. In practice, there are many technical, process, economic and organizational issues to overcome. However, several companies find that reuse does work, and can pay off handsomely, with high levels of reuse. There are significant corporate reuse programs at AT&T, HP, IBM, GTE, NEC, Toshiba, and others. They see major improvements in time-to-market, quality and costs.
Changing the process
The systematic application of software reuse will dramatically change the software process. Three of the most important changes are:
From libraries to factories: Incremental improvement
Fortunately, one does not need to go all the way in one giant step. One can choose an investment level, based on experience, to get an increasing number of benefits. An incremental adoption strategy will include several distinct steps. Figure 1 summarizes steps we call: leverage or cloning; black-box code reuse; broad workproduct reuse; architected reuse; and, systematic software reuse. As we move from step to step, we get increased benefits. Each step requires additional funding and upfront work before benefits are seen. These steps can be loosely seen as a transition from library-based reuse to kit-based reuse and then to a systematic reuse-based software factory.
To get high levels of reuse and consequent high levels of benefits, one needs more than just code or object libraries. One needs to deal with the careful design of reusable architectures and components that fit together, and with the systematic introduction and institutionalization of reuse. These issues will be discussed further in the following sections. Note that while code library use is common and helpful, most libraries handle only low-level, generic parts, leading to low levels of reuse. Current library research is focussed on classification, library tools and library interoperability. Object technology (analysis, design, frameworks) indeed provides additional benefits, but not enough for dramatic improvement. Object-oriented languages provide encapsulation, inheritance, and polymorphism which simplify use and permit design and interface reuse as well as code reuse.
Domain-specific kit and flexible software factory research at HP
HP Laboratories started a multi-disciplinary software reuse research program in 1992 to complement HP's Corporate Reuse Program[1,3]. Key to our research program is an integrated approach to technology, method, process and organization issues.
Our program has two major themes: domain-specific-kits and flexible software factories. The domain-specific kit research focuses on the technologies and methods for the production, use and support of kits, while the flexible software factory work concentrates on the processes, organization design, and software engineering and communication infrastructures for kit-based software development.
A domain-specific kit is a set of reusable work products that fit well together and are designed to make it easy to build a set of related applications. A kit (illustrated in figure 2) consists of reusable components, a carefully designed framework which captures the architecture or shape of a family of related products, and some form of "glue code" which could be either a typical language such as C or C++ or some problem oriented language specific to the application domain. Some components come in "plug-compatible" sets with the same interface (eg, sorting routines), while others are heavily parameterized. Other reusable workproducts include design fragments, tests, templates, macros, documentation, tools and examples.
Different kits may be used to construct different sets of related applications. Sometimes, more than one kit can be used in a single application. Kit interoperability then becomes an issue.
We are researching the methods and technologies of constructing and using these kits. We are developing kit analysis and design methods. Specific reuse-oriented methods need to be employed in the production, use and support of kits. Such methods include analysis, architecture, design, certification, etc. We also are investigating common technologies which could be used in the construction of kits, such as software-bus technologies; generators and problem-oriented languages; component integration technology; and, customization and end-user programming technology.
The flexible software factory is the organization and processes that are optimized to support the systematic use of domain specific kits. The term "flexible software factory" is derived from the ideas and methods used by organization re-engineering as applied to manufacturing, product development and flexible automation. We are looking at the processes, organization and tool environments for these organizations, specifically at how standard process elements and organizational design methods can be used to construct an optimal and efficient software factory.
The two themes are connected by a phase of business modeling, and by the tool and technology support. The kind of kit methods and technology and the kind of organization and processes that are followed should be based on the specific business situation, such the stability of the application domain, and which business issues (such as time to market, conformance to standards, interoperability) drive the factory. We are exploring modeling, economics, metrics and segmentation issues.
Our goal is to develop methods and technologies for HP divisions to dramatically improve application construction tasks using reuse. Most HP reuse situations involve the construction of families of closely related applications. To encourage the coherent design, implementation and packaging of a variety of compatible domain-specific workproducts, we first identify and structure the domain (domain engineering) and then use new methods (kit engineering) to build a domain-specific kit consisting of domain-specific components, supporting infrastructure and tools. The kit is then used to build one or more applications.
The Domain Analysis Working Group at the 5th Annual Workshop on Reuse explored the different styles of domain analysis appropriate to either generative or compositional reuse. Purely generative approaches (e.g. Draco) were deemed too complex for most cases, even though the payoff is high. Purely constructive approaches were judged unlikely to be used to full advantages since they suffer from the reuse inhibitors such as inability to identify appropriate components and lack of architectural support of reuse. Hybrid reuse, combines both generative and compositional approaches by providing domain specific language support for composing applications from collections of components within particular architectures. Current DA methods do not facilitate systematic design for hybrid reuse, or identify tradeoffs as to which route to take for which part of a complete application family.
A typical "hybrid domain-specific kit" will contain most of the items shown in figure 2:
Theory and Practice of Domain-Specific Kits
Our research goal is to produce a theory and practice of domain-specific kits, abstracting ideas from division studies and experiences in developing and using several domain-specific kits. A complete methodology should include:
Construction of domain-specific kits
Kits are developed via a two path process, shown in figure 4. It is critically important that the kit be appropriate for the application domain it will be applied to, but it is just as important that it be appropriate for the application developers who will use it to build software within the domain. Therefore, we have developed a kit building process in which includes both domain engineering and kit engineering.
Domain Engineeringis analysis, design and construction of software requirements in the application domain. The goal is ton understand, design and build domain specific components, a domain specific architecture, and other workproducts, such as a domain specific language or generator, an application framework, and generic application(s). It is a three stage process:
Kit Engineering starts with an analysis of kit user (i.e., the application developer) needs and preferred development style. Here the goal is to develop appropriate tools, generators, builders, development environment, documentation, etc. to optimize developer comfort and productivity with the kit. In the case of a hybrid kit, language and builder technology will be an important part of kit engineering. This is also is a three stage process:
While these steps can be described separately and each will have its own deliverables, they usually are not done independently.
Domain and kit analysis can be done with little interaction; however, the results of these steps will drive the next steps in both paths. For example if kit analysis finds that developers will prefer a domain specific language as their application building style (a hybrid kit,) domain design must be oriented toward developing a language, an application architecture which is appropriate for using a language, and components which will support a language. The kit design will then develop a language-based kit architecture and find (or develop) appropriate tools for a language based kit. The kit and domain implementation overlap significantly, but there will still be separate tracks of work. For example, kit implementation may be involved with developing the kit use infrastructure (configuration management, etc.) while domain implementation will develop domain functionality.
An underlying collection of kit support technology will aid in kit construction. This supporting technology could include more generic kits (such as user interface or generator kits), component interconnection technology (such as a software bus, remote procedure call or distributed object system), and development support technology (such as source code control and configuration management tools).
We believe that if domain engineering and kit engineering are carefully defined with the right resulting work products, it should be possible to retool a domain by developing a different kit, without repeating the domain analysis or all of the domain design and implementation. We should also be able to (re)use the results of kit engineering for a particular domain in a kit for another domain.
Application developers will use a kit by performing an "application needs analysis" for the particular application their users need. Their analysis should be in the context of the kit, rather than starting from zero ("design with reuse"). Application construction will use the kit for overall application architecture and all functionality which is covered by the kit. Application specific functionality will have to be designed and developed for user needs not covered by the kit or for areas (e.g. hardware interface) not planned to be covered by the kit. Openness is an important factor in realistic kits since it is unlikely except in very well defined domains that a kit will include all required functionality. Since developers may combine more than one kit to develop applications, common kit architectures and base technology can be important.
Domain-specific languages for hybrid kits
A tutorial at the 2nd Internaltional Reuse Workshop by Frakes, Batory and Devanbu on application generators and domain specific languages partially addressed these issues, but did not suggest how to design appropriate domain-specific languages for the level of generator technology chosen[10 , 11 , 12]. However, there were no guidelines for hybrid reuse.
Use of language technology does not mean only compilation of language into executable code. It can be used in several ways to build applications from hybrid domain-specific kits:
In addition to traditional languages, visual languages, menu-driven template fillers and "walk-throughs" that generate these languages can be used to produce more user friendly language interfaces.
The following list illustrates some implementation choices:
Using a traditional language such as Basic or C, the domain-specificity is offered as a library of subprograms which extend the language in domain specific directions.
Object oriented languages such as C++ and Smalltalk allow use of inheritance, polymorphism and type abstraction mechanisms to develop domain specific classes which integrate cleanly into the language and its development model.
Using macros or syntax extension capabilities in LISP and TCL, or using CPP, AWK or PERL, Yacc, or STAGE as preprocessor, languages can be extended by addition of concise, domain-oriented expressions. These expressions will be translated to the base language for compilation. However, application developers (kit users) view them as integrated language extensions.
Language development tools can also be used to develop completely new, concise domain specific programming languages, which can greatly simplify the selection, customization, generation and integration of kit components into a complete application.
In HP, Yacc and Lex, and C-based Tree-Meta have been used to produce a variety of instrument system programming languages, which are used to as part of an "instrument developers kit" to rapidly produce complete instruments. These languages are tailored to permit users of an "instrument kit" (instrument developers) to configure and select parts of an instrument, create front-panel display and remote-port drivers, declare bindings to measurement routines, establish interfaces to higher-level analysis routines, and fill in data tables.
Visual application construction tools can be viewed as higher level languages which are used to define how kit components are combined to specify a particular application. As with languages, generation may involve generation of a lower level language, parameter tables for components, connection of components into a framework, or generation of application specific data files.
Within HP, several systems combine multiple techniques at the same time, leading to some issues of consistency and interoperability. One way to ensure that several languages (or customization techniques) be consistent or interoperate, is to implement them using the same language kit, which can be embedded with the development and delivery environments. In addition to Lex and Yacc, small embedded interpreters (Xlisp, TCL, PERL) and simple-meta compilers (Tree-Meta, Pmeta) are quite attractive for this purpose.
Another element of interest is that the domain-specific language can be either used to produce the software and firmware for a complete (instrument) application, or interpreted in a simulation environment (using generated stubs, and library modules) to allow applications specified in the language to be quickly and easily prototyped. This has resulted in much quicker turnaround for new development and maintenance, since testing in the real target environment can be delayed.
Next Steps and Summary
We are researching the methods, technologies, processes and organization designs to support systematic software reuse for HP. We are testing the ideas of domain-specific kits and flexible software factories through a series of prototypes and divisional pilots.
We are prototyping a series of small domain-specific kits to help drive the development of our kit design and implementation methodology and tools. Our first kit is in the domain of task-list managers, using our software bus for component integration. Following a minimal domain analysis and kit design, components were written in several languages (LISP, C++, TCL/TK) for item display, definition, time management, and task-list item storage management. Alternative components and options are selected, and data-structures are defined, using a simple (LISP-based) configuration language, from which a complete application is generated. A conversational rule-based tool uses data derived from our domain analysis to present decisions and consequences to the application builder to help generate the kit configuration file.
In our next cycle, we will refine our domain analysis method, and be more systematic about the design of the kit architecture and style. We will use hypertext tools (Kiosk) to browse and display domain and kit workproducts, issues and decisions and as an interface to the kit components and documents.
We are surveying the use of hybrid kits, looking at language and generator style and supporting technologies. We are collecting information on available language, builder and generator technologies, and will do some analysis and prototyping to understand and evaluate several of these.
There are many important issues that should be addressed systematically. These include:
We have had useful discussions with Joachim Laubsch, Jon Gustafson and Joe Mohan as we prepared this paper. Dennis Freeze carefully reviewed the final drafts..
Martin L. Grissis Technical Director for Software Engineering and a senior Laboratory Scientist in the Software Technology Laboratory at Hewlett-Packard Laboratories, Palo Alto. As manager of the Software Reuse Department, he leads research on software reuse, domain-specific kits and flexible software factories. He works with HP Corporate Engineering to systematically introduce software reuse into HP's software development processes. He was previously Director of HP's Software Technology Laboratory. Before that, he was an Associate Professor of Computer Science at the University of Utah.working on Professor of Computer Science at the University of Utah. He received a Ph.D. in Physics from the University of Illinois in 1971. He has published over 30 papers, authored a reuse handbook and reuse book chapter, and written many technical reports.
Kevin Wentzel is a project manager in the Software Reuse Department of HP Labs' Software Technology Lab. Together with Martin Griss, he has proposed and built a multi-disciplinary team oriented toward experimental research in software engineering and reuse. Kevin has been with Hewlett Packard for 16 years. Prior to joining HP Labs, Kevin worked in various engineering and management roles in several HP divisions developing software products. Kevin has BS and MS degrees in Computer Science from the University of Wisconsin.
Contacting the authors
Hewlett-Packard Laboratories, 1u16
1501 Page Mill Road
Palo Alto, CA 94304-1126
Tel: (415) 857-8715 and (415) 857-4018
Fax: (415) 813-3668
Email: email@example.com and firstname.lastname@example.org
 Martin L. Griss, John Favaro, and Paul Walton. Managerial and Organizational Issues - Starting and Running a Software Reuse Program, chapter 3, pages 51-78. Ellis Horwood, Chichester, GB, 1993.
 Martin L. Griss. Software reuse: from library to factory. IBM Systems Journal, 32(4):1-23, November 1993.
 Martin L. Griss. A multi-disciplinary software reuse research program. In Martin Griss and Larry Latour, editors, Proceedings of the 5th Annual Workshop on Software Reuse, pages Griss-1:8. Department of Computer Science, University of Maine, November 1992.
 Martin L. Griss and Will Tracz. Workshop on software reuse. Software Engineering Notes, 18(2):74-85, April 1993.
 James M. Neighbors. The Draco approach to constructing software from reusable components. IEEE Transactions on Software Engineering, SE-10(5):564-574, September 1984.
 Gerhard Fischer. Cognitive view of reuse and redesign. IEEE Software, 4(4):60-72, July 1987.
 Jeff A. Johnson, Bonnie A. Nardi, Craig L. Zarmer, and James R. Miller. Ace: Building interactive graphical applications. Communications of the ACM, 36(4):40-55, April 1993.
 Ruben Prieto-Diaz and Guillermo Arango. Domain Analysis: Acquisition of Reusable Information for Software Construction. IEEE Computer Society Press Tutorial, New York, 1991.
 Brian W. Beach, Martin L. Griss, and Kevin D. Wentzel. Bus-based kits for reusable software. In Proceedings of ISS'92, UCI, Irvine, March 6, pages 19-28, March 1992.
 Bill Frakes. Application generators. Lecture note slides, June 1993. (Private communication).
 D. S. Batory. The Genesis database system compiler: Large scale software reuse from domain modeling. In Larry Latour, editor, Proceedings of the 4th Annual Workshop on Software Reuse, pages 1-6, Department of Computer Science, 222 Neville Hall, Orono, Maine 04469, November 1991. University of Maine, University of Maine.
 Don Batory and Sean O'Malley. The design and implementation of heirachical software systems with reusable components. ACM Transactions on Software Engineering and Methodology, 1(4):355-398, October 1992.
 J. C. Cleaveland. Building application generators. IEEE Software, 4(9):25-33, July 1988.
 Michael Creech, Dennis Freeze, and Martin L. Griss. Using hypertext in selecting reusable software components. In Proceedings of Hypertext'91, pages 25-38, Palo Alto, CA, December 1991. Software and Systems Laboratory.