Chisholm (1994) used the term "tactioneering" to refer to user-centered software design. The word taction means touch or contact. Tactioneering is "a measure of quality and intensity of interaction between a person and something else; in this case software." Accordingly, the software designer's job is to "maximize the taction between the user and the software within system constraints."

The idea of tactioneering highlights two areas that make user interface development especially challenging from a reliability standpoint: intensity and quality. For the purposes of this paper, intensity refers to the "usability" of the software. Quality refers to the ability of the software to perform the required tasks.

Many developers use prototyping as a tool for improving usability and quality. However, prototyping is just one of several methods/practices for enhancing interface usability/quality, and is generally used for requirements clarification (Davis, 1992). These methods/practices makeup part of "usability engineering." Usability engineering is an attempt to mature the user interface development process, both externally and internally (Curtis, 1993). Maturity in this case is a transition from an art-like process to a more industrial process, and, eventually, to an automated process (Lai, 1993).

This paper looks at usability engineering and how it impacts user interface reliability. Specifically, the paper focuses on the usability and quality issues related to interactive software development. For usability, the paper looks at the costs/benefits of usability testing, goes through a simple discount method for usability testing, and covers a list of general usability principles. For quality, the paper looks at the need for automated interface testing tools, and goes through a feature list for these tools with examples from current market software.

Curtis (1993) talked about maturity of interfaces as being both external and internal. Maturity from an internal perspective is concerned with an organization's engineering and management practices as they are applied to interface development. External maturity is how the user perceives the product and the organization that built it. That perception is "the user's experience with all interfaces - graphics, dialogues, documentation, help services - they interact with using the product." External maturity is, essentially, usability. It is important that the development organization focuses on both types of maturity. "While no interface enhancement will fix defective software, a bad user interface can ruin an otherwise flawless package. Users often can't make the distinction."

As with other reliability elements, usability is subject to tradeoffs. Nielsen (1993) briefly described a cost/benefit model for applying usability engineering. The costs associated with usability include: personnel costs of the staff doing the usability analyses, miscellaneous facilities costs, and personnel costs of the user testers. These costs vary from project to project based on interface complexity and the number of user types.

Benefits vary according to the type of software. In-house benefits include reduced training time, increased user productivity, and increased user satisfaction. Increased user satisfaction is difficult to quantify financially, although it can lead to reduced absenteeism and turnover. Reduced training time is a one-time savings concentrated soon after the system release. Improved productivity provides small benefits on a day to day basis.

Commercial benefits are reduced support costs and increased sales. Support savings come from reduced support services staffing. Increased sales coming from usability is harder to measure, since usability is just one of many factors determining market size/share. However, PC trade journal studies show that users rate ease of use as important as functionality and performance; in one study usability accounted for 35% of the final evaluation.

Usability cost/benefits can be described by the following formula:

UP(i) = N(1 - (1 - )ⁱ)

Where UP(i) is the number of usability problems in i observations,

N is the total number of problems in the interface,

is the probability of finding the average usability problem when software is run for a single user.

A plot of the curve for a medium size project is shown in Figure 1. The ratio shows a benefit of 50 times the cost of a single user, with the optimum benefit/cost ratio of around 70 times the cost with three users (Nielsen, 1993). These findings suggest that even a minimal amount of usability testing is dramatically beneficial, with diminishing returns reached after a relatively small number of users.

Nielsen (1995) highlights some of the minimalist properties of usability engineering in what he calls "discount usability engineering." "Discount methods are more informal and rely less on statistics" based on the premise that they are faster, cheaper, and, unless the product is safety critical, "finding every last usability problem is not serious." Nielsen's technique is a sequence of four studies: card sorting to discover categories, testing for icon intuitiveness, distribution of cards to icons, and mock-up walkthroughs.

Card sorting uncovers users' models of the application's structure. The development team writes each product primitive function on an index card and randomly place the cards on a table top. Users then group the cards into small piles based on their perception of what primitives belong together. Users give names to their piles, then repeat the process by grouping piles into larger piles. The process produces a bottom-up menu hierarch from the users point-of-view.

Icon intuitiveness is testing to see if the users interpretation of icons is consistent with the development group's. Even if icons have accompanying labels, the icons themselves should be clear enough for users to make a selection.

Card distribution is a top-down implementation of card sorting. The development group makes enlarged paper printouts of the planned menus and/or icons and places them in their relative positions on a large table. Users are then given index cards with primitive functions on them and asked to place them under the appropriate interface object. At the end of the distribution, users comment on each interface object's aesthetics and list the objects they liked/disliked.

Walkthroughs involve taking users through a series of screen prints covering various standard transactions. Developers ask users what information they think would be behind each interface object. Users also comment on the general aesthetics of the interface and list objects they liked/disliked.

Achieving consistent high user-software taction requires certain development process maturities. Although there is no consensus methodology, there seems to be a consensus that a development organization needs to establish and integrate some set of interface design principles. These principles will vary based on product type, size, and target audience, but some generic design principles are listed below:

• Conceptual integrity - the product should "conform to a single, unifying, vision users can perceive. The unifying vision ties the product's components together, making the product seem natural and straightforward (Chisholm, 1994)." The interface needs to have a consistent look and feel across all components (Hobart, 1995).

• Grammatical - "build an inverted hierarchy of user interaction based on as few atomic elements as possible." This means a vocabulary with a "limited number of primitives (analogous to sounds or syllables in human speech)." These are combined to form a larger number of "higher order constructs (like words)." These constructs are then combined to form application specific commands. "Having a grammatical language of user interaction makes the software easier to learn (Chisholm, 1994)."

• Well mapped - the look and position of each control "should suggest its purpose and use. Good mapping reduces the user's dependency on memory, training, and documentation (Chisholm, 1994)."

• Custom fit - the ability of the user to modify the environment to best fit their ability levels. Constantine (1994) suggests a "triphasic architecture." The components of a triphasic architecture are:

  • Acquisition - this is what the novice sees when first using the software; it must let this user type get useful work done right from the start.

  • Transition - the interface for "improving intermediates;" those beyond slow/sloppy point and click, but who are no yet power users.

  • Production - this makes it possible for an experienced, fully-trained user to produce sophisticated results with high efficiency.

• Self helpful - current technologies in help formats include:

  • Coaching - active assistance to "solve an immediate performance shortfall and training to teach the functions over time."

  • Human-like assistants

  • Context-sensitive help - "provides help in the proper context of the application and synchronizes help to the tasks being performed."

  • Adaptive user support - "tailoring help to support the user's tasks and customizing the interface's commands to assist users." The level of help offered and how it is presented is controllable, to some extent, by the user.

  • Task automation - turns "commonly used tasks into automated functions (Hefley, 1995)."

• Trustworthy - "the appearance and interaction of the system should impart confidence in navigating and using the product. The program must respond predictably and consistently; it must let users change their minds and recover gracefully from mistakes (Chisholm, 1994)."

• Engaging - the interface may include elements of "humor, graciousness, friendliness, kindness, or generosity by the program towards the user (Chisholm, 1994)." However, engagement is like seasoning; you can overdo it. The designer must constantly assess how much seasoning is optimal.

The unpredictable nature of human-interface interaction calls for running numerous tests to demonstrate a measure of quality. The number of variables, combinations of values, and order of operations involved make traditional test methods unwieldy. "The difficulty of testing grows exponentially with every variable in the system (Hanna, 1993)." Additionally, most of these interfaces are being developed using visual programming tools and/or CASE tools. These tools empower developers to create large, complex interfaces in far less time than traditional development methods, meaning more system to test and less time to do it.

Most organizations still generate test cases manually. However, in a 1993 study, 77% of development organizations expressed a need for automated interface testing tools and 21% had already purchased testing software (Korzenlowski, 1994). Testing's failure to keep pace with more complex development threatens to offset productivity gains from development technologies (Hanna, 1993).

Software testing tools are evolving. The first stage, low-level capture/replay and verification tools, originated and migrated from character-based systems. These have given way to second, third, and fourth generation of tools with more advanced features, network extensions, and object oriented extensions respectively (Hanna, 1993). Like most technologies in their infancy, there is no standard list of features for software testing tools. However, the various tools in the market do help provide a list of desired features:

• Portability - with current trends to open system development, "clients are more concerned with portability than front ends." Not only do the applications have to port to different environments, but the testing systems for these applications also need to port. Most software testing tools support porting to some degree, although the degree varies greatly from product to product. Examples of portable testing tools are AutoTester (Damore, 1993) which runs on DOS, Windows, OS/2, and Unix. QA Partner from Segue Software currently supports 22 environments, including DEC VMS, DOS, Windows, Macintosh, Unix, Motif (Korzenlowski, 1994).

• Database - all testing tools have some type of storage facility for test definitions. Other common database features are test results storage and centralized, shared test/results storage. Examples of advanced testing databases are SQA Manager from Software Quality Automation, Automated Test Facility (ATF) Executive from Softbridge providing "tests and results for examination and comparison (Hanna, 1993)," and TestDirector from Mercury Interactive that provides "a central repository for all testing data (Korzenlowski, 1994)."

• Layered - Hanna (1993) writes about the fourth generation of testing tools as being able to "work with the GUI level rather than against it" to test the GUI itself and the application underneath separately instead of treating them as one entity. A good example of a layered tool is QA Partner which "uses the information the GUI has to help run the test. By extending object oriented technology to the testing phase, users have another level of abstraction on top of the keystrokes and clicks."

• Test design interface - testing tools should have flexible and intuitive interfaces to help users at various levels of ability construct tests. Examples of test design interfaces are ATF Executive as "an OS/2 application for writing and editing tests (Hanna, 1993)," and SQA TeamTest which "uses object-oriented recording technology that allows users to intuitively generate test cases (Korzenlowski, 1994).

• Scripting language - most second-generation testing tools include some type of scripting capability for defining tests. Examples of scripting capabilities are AutoTester which allows users to "generate document-oriented tests, mixing text and script commands (Damore, 1993)," SQA TeamTest that uses Microsoft Visual BASIC as its scripting language, and QA Partner with it's 4Test 4GL (Korzenlowski, 1994).

• Record/playback - most testing tools include some type of playback mechanism, allowing users to simulate/record keyboard and mouse events and replay them to repeat tests. Examples of record/playback tools are TestPro by Sterling Software and Microsoft Test, both of which simulate keyboard and mouse events.

• Error trapping - tools should have the ability to trap application errors and continue with testing. An example of error trapping capability is Microsoft Test which "can intercept unrecoverable errors and either log them and keep on going or stop the test (Hanna, 1993)."

• Performance monitoring - testing tools should also have the ability to measure system performance while running the application. ATF Executive provides "stress, limit, concurrency, and LAN integration testing (Korzenlowski, 1994)." Ideally, these tests could report by application/system section and system resource to help isolate performance bottlenecks. QA Partner is a good example of package that has this capability.

• Non-intrusiveness - an important factor in performance measurement is that the testing tool itself does not bias or "intrude" on the test results. Korzenlowski (1994) defines an intrusive tool as one that "runs on the same CPU and uses the same system resources as the application being tested. Thus it could alter the performance results." Examples of non-intrusive testing tools are SQA Robot and Ferret from Tiburon Systems, which uses hardware to "intercept and convert mouse, keyboard, and video signals from the system under tests into standard protocols. Ferret then uses the protocols to automate the testing process."

In summary, interfaces pose verification challenges for testers beyond those of more conventional software. These challenges converge around the issues of interface usability and interface functionality.

Nielsen highlighted dramatic paybacks from even minimal usability testing. Benefit/cost ratios of between 50 and 70 to 1 make a case that "it is always well worth the cost to do usability engineering."

Nielsen also consistently emphasized the minimalist nature of usability engineering, stating that "the best results are often gained by combining several small tests with three to five users and iterating the design between each evaluation." These methods can be relatively small in terms of procedures as well as the number of tests, as shown by discount usability engineering. "Even if your project schedule seems too compressed to do much usability testing, you can still do something, and some usability testing is so much more than none."

Testing techniques are one part of usability engineering. The other components are design/implementation principles that developers need to either integrate into their processes or use as a basis for re-engineering processes. This type of organizational discipline is not unique to interface development, but it is a necessary part of maturing to the point where the developer can turn out consistent, usable products.

"Good GUI designs don't just happen naturally. They require that the developer learn and apply some basic principles (Hobart, 1995)." The list derived in this paper can be summarized as: conceptual integrity: simple and straightforward; grammatical: learnable; well mapped: understandable; custom fit: malleable; self helpful: remedying and educating; trustworthy: dependable; engaging: fun.

Looking at interface functional testing, the literature review shows a rapidly expanding need for automated testing tools. This is due to a number of factors, including the increased use of CASE tools, visual programming languages, and the trend of migrating from mainframe systems to mini-computer- and PC-based systems. Developers test the interface's functionality during module, integration, and acceptance testing. Maintenance actions also require regression testing.

The market for software testing tools is relatively immature. The tools themselves are evolving; the features list presented earlier represents the state-of-the-practice in testing software. One problem is that both developers and customers are trying to decide what they want in testing software. Because of this, many organizations are waiting on the sideline for some industry standards to emerge before committing to a tool. With over 95% of companies surveyed responding that they had either purchased testing software or were interested in software testing tools, and the increasing demand for those tools, a majority of development organizations will move to using automated testing tools to some degree in the near to mid term, given that some standardization takes place.

Lastly, the role development organizations give to software testing tools is important. Developers need to realize "there is no replacement for a person's sense of rightness in assessing interface software." A tool can not replace the common-sense of a QA engineer. "However, it does relieve much of the QA engineer's burden." Usability engineering is a mix of usability testing and interface functional testing. Developers need to determine the right proportions of each, making sure that minimums for each are maintained, depending on their project type and audience.

Chisholm, J. (March, 1994). The Art of Software Design. Unix Review. pp. 11-18.

Constantine, L.L. (July, 1994). Interfaces for Intermediates. IEEE Software. pp. 96-99.

Curtis, W. (July, 1993). Maturity from the User's Point of View. IEEE Software. pp. 89-90.

Damore, K. (December 27, 1993). AutoTester lets users try out applications. Infoworld. pp. 20.

Davis, A.M. (September, 1992). Operational Prototyping: A New Development Approach. IEEE Software. pp. 70-78.

Hanna, M. (February, 1993). GUIs Pose Challenge to Software Testers. Software Magazine. pp. 35-44.

Hefley, W. (March, 1995). Helping Users Help Themselves. IEEE Software. pp. 93-95.

Hobart, J. (September, 1995). Principles of Good GUI Design. Unix Review. pp. 37-46.

Korzenlowski, P. (October, 1994). Bug-eyed C/S Developers Look to Automate Tests. Software Magazine. pp. 77-81.

Lai, R. (July, 1993). The Move to Mature Processes. IEEE Software. pp. 14-17.

Nielsen, J. (January, 1995). Applying Discount Usability Engineering. IEEE Software. pp. 98-100.

Nielsen, J. (November, 1993). Is Usability Engineering Really Worth It? IEEE Software. pp. 90-92.