INTERNALLY CIRCULATED MEMO (a client would get something more formal) OC 1.0 formal testing, round one, April 2003 [company confidential] -------------------------------------------- Method ------ Four test subjects were observed during a scripted interaction with OC-Host 1.0 that modeled two typical scenarios: creating a user in OC and assigning them software (including one software item that was intentionally on the second page of the catalog), and then creating a software pack containing the software that had just been assigned. Environment ----------- The testing environment consisted of two machines, one running OC-Host and hosting the database, the other running Agent, on which the software was to be installed. The environment was pre-configured: a unix login already existed for the user to be created in OC, and software was already in the existing catalog. Findings -------- Documentation: 1 - During setup I was reminded that a Linux login for a user has to exist on the Agent machine before software can be successfully installed for that user on that machine. This is mentioned nowhere in the documentation (should probably be under both create user and install software, or as part of a general setup section.) 2 - The documentation link in the Host interface links to the URL www.opencountry.com/documentation. The first link on that page is to a file titled "readme.first," but that file is really for install information, not general information. (That file also contains a link at the bottom of the text to the same documentation URL. One subject went around this infinite loop three times). Layout and design: 1 - Three out of four testers did not understand that the purple arrows were "next page" and "previous page" links (this was very unexpected). While this is documented, the arrows should be redesigned to a more standard shape (can easily be done for 1.x). 2 - Three testers tried to use the "Manage Software" interaction to assign software. This was also noted in the heuristic analysis, and should be revisited in 2.0. 3 - One tester clicked repeatedly on dead breadcrumb links. This has already been identified as an issue. 4 - Two testers used the back button on their browser to try to navigate between screens, thereby accessing invalid screen states. (This can be fixed in 1.x, right?) 5 - One tester said the gray buttons used to denote the current state in the "show all, etc." interactions "look disabled" (they are, but perhaps this should be revisited). Interactions: 1 - One tester tried to search for a node via IP address and failed. 2 - One tester tried to search for software with dashes in the name, left out the dashes, and the search failed. (Dashes mean something specific in boolean searches, and we need to address this discrepancy.) 3 - One tester couldn't access lower levels in the software directory, apparently because of the discrepancy in display between directory-containing directories, and asset-containing directories. (Mixed directories are standard, but this may not be an easy fix). 4 - One tester clicked to select the user before going on to the lower-level interactions under User > Nodes (Not a fatal problem, just unexpected). 5 - One tester selected a number of software items, and then went on to the next page in the catalog before installing, losing those selections (heuristic analysis also flagged this as a potential problem). Tree: There were general tree position issues where testers were looking for interactions while in the wrong branch, for example going to Users > Nodes to add another user to a node. Not all of these can be addressed, and many are learning curve issues. Meta: 1 - 1.0 still has many instances of weird english, courtesy of the original two developers. These need to be hunted down. 2 - There are still some instances of missing training wheels (and still no way to turn them off). 3 - We use "username" "login name" for the same thing. (General english review needed). 4 - Creating a user and filling in a new password pops up a "remember this form?" interaction.