'Usability Test Documents'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kde-usability
Subject:    Usability Test Documents
From:       Irwin K <emerald-arcana () rogers ! com>
Date:       2002-06-29 20:57:08
[Download RAW message or body]

Some comments:

I haven't finalized the list of tasks.  I have a good number of tasks to 
choose from, but I haven't decided which ones to put on the actual test.  For 
Lauri's organization, the warehouse workers probably don't need a list of 
tasks, and only need to be provided with an overview and to fill out the 
questionnaire at the end of the day.

The questionnaire must be modified to reflect the actual tasks that the users 
must perform.

The setup must be modified to reflect the actual tasks that the users must 
perform.

The original document is "usability_test_1.txt" by David Hugh-Jones, of which 
the subsequent documents were heavily based on.

My documents are "setup.txt", "user-guidelines.txt", and "setup.txt".

Please provide some comments for the initial drafts of these documents.

-- 
-- Arcana  (Irwin)
["usability_test_1.txt" (text/plain)]

KDE USABILITY TEST

TEST 1: LOGGING IN, STARTING APPLICATIONS, FINDING AND OPENING FILES, LOGGING OUT

==========
Test setup:
==========

Create a user named "test" with a password of "test457dx" on your machine.
Ensure that the user is visible from the KDM login manager, and that the user
can connect to the internet as normal. In the user's home directory, create a
folder called "instructions", and in it, create an HTML file called README.htm with
a few words about the website www.kde.org.

Create another file called "open_source_notes.txt"
with a few words about open source philosophy.
Put it in a directory which the test user can read, but not in an "obvious" place (e.g. not in
the home directory).

Have a watch ready to monitor how long the test subject takes to perform each of the tasks
detailed in the user instructions. Take notes on each of the tasks - did anything cause
the user difficulty? Did the subject do things in a different way from what you expected?
Were there system behaviours that frustrated the subject? Did the subject get "stuck" and have
to ask for help?

==========
Test subject instructions:
==========

1. An account has been created for you on the new computer system. The user's login is
"test" and the password is "test457dx". Please login to the new system.

2. Your administrator has given you a README file to tell you about the new system. The
file is called README.htm and is in a folder called "instructions" in your home directory.
Please find the file and open it.

3. Now you've opened the file, you would like to look at the www.kde.org website. Start
up an internet browser and browse to the www.kde.org website.

4. Earlier, you made some notes about 'open source' software,
but you can't remember where you put the file.
You think it's called something like 'Open source', but you aren't sure what! Try to find
the file on your computer. When you have found it, open it and read it.

5. That's all you're going to do for today. Please log out of the computer.

6. Congratulations - you've finished the tasks! Now we would like to ask you some questions
about how easy you found these tasks.

==========
Test subject questions:
==========

1. How easy was it to log in to the computer?

VERY EASY - QUITE EASY - ABOUT AVERAGE - QUITE HARD - VERY HARD

2. Think about finding and opening the README.htm file. How easy was it to find the file?

VERY EASY - QUITE EASY - ABOUT AVERAGE - QUITE HARD - VERY HARD

2a. When you had found it, how easy was it to open the file?

VERY EASY - QUITE EASY - ABOUT AVERAGE - QUITE HARD - VERY HARD

3. How easy was it to start the internet browser on this computer?

VERY EASY - QUITE EASY - ABOUT AVERAGE - QUITE HARD - VERY HARD

4. How easy was it to find the "open source" document?

VERY EASY - QUITE EASY - ABOUT AVERAGE - QUITE HARD - VERY HARD

5. Finally, how easy was it to log out of the computer?

VERY EASY - QUITE EASY - ABOUT AVERAGE - QUITE HARD - VERY HARD

6. Overall, would you say the new computer system was easy to use?

VERY EASY - QUITE EASY - ABOUT AVERAGE - QUITE HARD - VERY HARD

6a. Can you think of anything that could be done to make the computer system easier to use?

7. Did you enjoy using the new computer system?

VERY MUCH - QUITE A LOT - NO STRONG FEELINGS - NOT VERY MUCH - NOT AT ALL

8. Can you think of anything that could be done to make the computer system more fun to use?

9. Please use this space to make any other comments about this computer system:

10. How often do you normally use a computer? Choose the closest answer.

NEVER USED A COMPUTER - ONCE A MONTH OR LESS -
ONCE A MONTH OR MORE - ONCE A WEEK OR MORE- DAILY

11. How long have you used computers for? Choose the closest answer.

NEVER USED A COMPUTER - UP TO 1 YEAR - UP TO 2 YEARS - UP TO 5 YEARS - MORE THAN 5 YEARS

12. What sort of computers do you have previous experience with?

NONE - WINDOWS - MACINTOSH - UNIX/LINUX

==========
Observation notes:
==========

1. Login

Time taken:
Notes:

2. Finding and opening file:

Time taken:
Notes:

3. Opening web browser, browsing to website

Time taken:
Notes:

4. Searching for file by name:

Time taken:
Notes:

5. Logging out:

Time taken:
Notes:

Other notes:

["setup.txt" (text/plain)]

KDE Usability Test

==========
Hardware and Software Setup
==========

Install KDE on the target machine.  The packages that should be installed are:
	-qt
	-arts
	-kdelibs
	-kdebase
	-kdenetwork
	-kdemultimedia
	-kdegraphics

Create a user named "test" with a password of "kdetest" on your machine.

Ensure that the user is visible from the KDM login manager, and that the user can \
connect to the internet as normal.

Create the following directory structure in the user directory.  The script \
"create_environment.sh" will copy the necessary files for you.

~test
	README.html
	kde.txt
	images/konqi.jpg
	images/kdelogo.jpg
	music/welcome.mp3
	.bash_logout

Install the included logout script, ".bash_logout", in the "test" user's directory to \
ensure that the environment is clean when the user exits the KDE login environment.

==========
Test Administrator Guidelines
==========

Ensure that the user receives a copy of the test subject guidelines, the task \
instructions, the questionnaire, and the consent form.  Ensure that the consent form \
is signed before the user begins the test.

Have a watch ready to monitor how long the test subject takes to perform each of the \
tasks detailed in the user instructions. Take notes on each of the tasks - did \
anything cause the user difficulty? Did the subject do things in a different way from \
what you expected?  Were there system behaviours that frustrated the subject? Did the \
subject get "stuck" and have to ask for help?

If the test administrator cannot personally witness each test, then ask that the \
users read over the documentation carefully before they start.  Ensure that they do \
not feel confused or frustrated as to the purpose of the test.  Remember, they are \
not obligated to perform all of the objectives, and remind them that the test is \
totally voluntary.  It is not a test of their abilities, but instead a test of the \
system.

["user-guidelines.txt" (text/plain)]

KDE USABILITY TEST

User Guidelines
===============

Overview of the Test
-------------------

A usability evaluation actively studies specific activities performed by users on a \
computer.  The study gathers experimental evidence by witnessing users and gathering \
user feedback to evaluate the ease of use, the intuitiveness, the functionality, and \
the speed at which users can execute common actions.

Before We Begin
---------------

The test administrator should have briefly explained to you the purpose of the test \
and what the result are used for.  They will also provide you a copy of the following \
documents:

1. A User Guidelines sheet (this document), which summarizes the purpose of the test \
and gives you some background information. 2. A Consent form, which states your \
agreement to participate in the usability study. 3. A list of tasks to perform in \
front of the computer. 4. A questionnaire consisting of questions to answer for \
evaluation purposes.

Please read the provided consent form carefully.  Sign both copies if you agree to \
the terms presented.  Keep a copy for yourself and give the other copy to the test \
administrator.

A list of "User Tasks" is included with the Usability Test Package.  These tasks will \
be evaluated for their usability.  Please review them briefly.  If there are any \
terms or definitions that you are not familiar with, please ask the test \
administrator for clarification before you begin. 

Executing the Test
------------------

As you execute the test, speak aloud as you execute actions.  For example, if you are \
trying to open a menu, say aloud, "I'm going to the menu.  I see 'File'.  I'm going \
to click here."  This helps the evaluator guage your reactions to the tasks and \
elements on the screen.

Do not feel compelled to fulfill all of the actions asked of you by the evaluator.  \
If you cannot perform an action, then announce that you do not know what to do, and \
the test can proceed to the next step.  Remember, the test is for the software, not \
for you.

The following are questions that you may want to ask yourself as you perform tasks in \
front of the computer:

- How does the user interface feel?  Does it look warm, familiar and friendly?  Do \
                you like the colours and the layout?  Is it something that you would \
                like to look at?
- What is the first reaction you have when asked to perform a task?  Does the \
computer conform to your first reaction, or do you need to look farther to perform a \
                task?
- Is the user interface easy to scan?  When you are looking at it, can you easily \
decipher the different parts of the interface?  Are there too many items on the \
                screen?  Too few?
- Are the elements laid out in an order that makes sense?  Does it seem like items \
                have a logical grouping?
- How long does it take you to perform a task?
- How long does it take for the task to finish executing?  Is this time acceptable to \
you as a user?

Finishing the Test
------------------

When the test is finished, please fill out the questionnaire that is included with \
the Usability Test package.  When you are finished, please return the test to the \
test administrator.

["questionnaire.txt" (text/plain)]

KDE USABILITY TEST

Questionnaire
=============

Please fill in the questionnaire after you have finished executing the tasks.  Return \
the questionnaire to the test administrator.

You do not need to answer all of the questions.

1. How long have you used computers for? Choose the closest answer.

NEVER USED A COMPUTER - UP TO 1 YEAR - UP TO 2 YEARS - UP TO 5 YEARS - MORE THAN 5 \
YEARS

2. How often do you normally use a computer? Choose the closest answer.

NEVER USED A COMPUTER - ONCE A MONTH OR LESS -
ONCE A MONTH OR MORE - ONCE A WEEK OR MORE - DAILY

3. What sort of computers do you have previous experience with?  Circle all that \
apply.

NONE - WINDOWS - MACINTOSH - UNIX/LINUX - OTHER (Specify: __________)

1. How easy was it to log in to the computer?

VERY EASY - QUITE EASY - ABOUT AVERAGE - QUITE HARD - VERY HARD