WS Usability Study Report
Author: Jindrich Dinga, Dusan Pavlica
Version: First draft - 09/22/2004
Study Dates: September 6th and 7th, 2004
Table of Contents:
- Introduction
- Participants
- Tasks Summary
- Task Analysis
- Task 1: Looking through the IDE
- Task 2: Creating a simple Java application
- Task 3: Creating a form with Swing and AWT buttons
- Task 4: Changing Design->Source, creating a method
- Task 5: In Design mode add an EventHandler
- Task 6: Minimizing all windows to the EdgeBar
- Task 7: Moving minimized windows to another EdgeBar
- Task 8: Using keyboard for TASK 6 and 7
- Conclusions and Recommendations
- ECS shortcut is hidden a bit, but when user found it, said that it's useful [C-1]
- Component Palette shouldn't be in Auto hide mode by default because is too hidden [C-2]
- There is no way how to resize slided window in Auto Hide state [C-3]
- There is no keyboard shortcut for switching to Editor's views [C-4]
- There is missing Tooltip above toggle buttons in the Editor Toolbar [C-5]
- Participants had problem with appearance of Dock icon and didn't know what does it mean [C-6]
- Participants expected tooltip above icons in Titlebar of each window [C-7]
- Name of Auto Hide action is not clear enough for users, they expected different name [C-8]
- Users expected D&D by window in Auto Hide mode, when they tried to move to another EdgeBar [C-9]
- There is missing Auto Hide (Minimize) menu item in Window menu [C-10]
- There is no shortcut for moving windows inside IDE [C-11]
- Another revealed bugs
- Other information as a side effect of study
Introduction
The usability study was focused on observing the first hour experience when using Window system features (Sliding window and Multiview) included in NetBeans IDE. Eight participants were asked to complete 8 tasks using NetBeans IDE Beta1 build. This usability was special a bit, because we needed to observe detail behavior of participants not only their hit ratio in tasks. That is why we decided to have two observers in the usability room.
Usability study has revealed some UI problems which could cause really substantial problems to user. E.g. some of most visible:
- There is no way how to resize slided window in Auto Hide state [C-3]
- Users expected D&D by window in Auto Hide mode, when they tried to move to another EdgeBar [C-9]
Another group of problems is not so big, however after their fixing mentioned WS features
in Netbeans would be easy to use during the first hour of usage. Then usability of those
features could be increased too. E.g. some typical problems:
- Participants had problem with appearance of Dock icon and didn't know what does it mean [C-6]
- Participants expected tooltip above icons in Titlebar of each window [C-7]
- Component Palette shouldn't be in Auto hide mode by default because is too hidden there [C-2]
In the study we noted 11 UI problems (till this time) and obstacles directly related to UI
of Netbeans Beta1 build and all are described in detail below.
Participants
We could categorize participants according to programming experience as:
- 2 student (P3, P7)
- 3 junior programmers (P1, P2, P8)
- 3 experienced programmers (P4, P5, P6)
The full participant characteristics will be found here: Participants.
Tasks Summary
Nine tasks were focused on testing following:
- Task 1: Looking through the IDE
- Task 2: Creating a simple Java application
- Task 3: Creating a form with Swing and AWT buttons
- Task 4: Changing Design->Source, creating a method
- Task 5: In Design mode add an EventHandler
- Task 6: Minimizing all windows to the EdgeBar
- Task 7: Moving minimized windows to another EdgeBar
- Task 8: Using keyboard for TASK 6 and 7
The full task's script can be found here: Task Script.
Participants finished nearly all tasks but with problems mentioned below.
Task Analysis
Task 1: Looking through the IDE
Objectives
- Maybe strange layout of windows will provoke the user into rearranging it.
- We wanted to observe if he uses the Sliding window or Close action, eventually D&D.
Notes
- P1 everything moved to the left side used to having it, because of Eclipse, used doubleclick to maximize the window. Tried Auto Hide of the Output window by mouse focusing.
- P2 looked just through the IDE without any special action.
- P3 arranged the layout by using D&D function. He invoked the Component palette, he didn't understand the icons. He also looked for switchers between Source/Design, he thought, there should be TABs.
- P4 arranged the layout by using D&D function. Looking through the toolbar. Looking what features the IDE/Refactoring could offer.
- P5 arranged the layout by D&D function. He was satisfied with the icons in the ToolBar. He wasn't sure about the meaning of buttons in TODO window, D&D function is slow.
- P6 spent a lot of time by browsing menu, Options dialog and Properties dialog. He had problem to dock this Properties dialog, because he thought that it's Properties window.
- P7 looked just through the IDE.
- P8 looked through the IDE. He wanted to change the keyboard profile.
Summary
- All participants had different comments about the IDE based on their experiences with other IDEs.
Task 2: Creating a simple Java application
Objectives
- Just checking if user understands Edit/Compile/Run cycle, which is important for next tasks.
- We wanted to find out how users cancel the Output window in Sliding state (ESC, click, key press).
Notes
- P1 is used to programming with Code Completion
- P2 did it without any problems. For Run cycle, he uses F6.
- P3 started by icon in the ToolBar.
- P4 run the project via the context menu. The users mentioned, Build Successful is redundant in the Output window.
- P5 was successful with creating the Java app. He saved the project by shortcut Ctrl+S, he noticed it was saved because of *.
- P6 tried to find abbreviations and then used Code Completion by shortcut.
- P7 created the project via the Welcome screen. For writing a code, the user used Code completion (Ctrl+Space).
- P8 did it without problems.
Summary
- No problems. One participant said, Build Successful is redundant in the Output Window.
- One participant used a shortcut Ctrl+S for saving.
- One participant used a shortcut F6 for running, one participant used the icon for running.
- Three users used a shortcut Ctrl+space for Code completion.
100.0% of participants(8) found out canceling the Output window by mouse click easily
Problems related to WS
- ECS shortcut is hidden a bit, but when user found it, said that it is useful [C-1]
Task 3: Creating a form with Swing and AWT buttons
Objectives
- We test whether user user is able to find Palette in Auto Hide state situated at the right Edgebar
- We test whether user user is able to find right component in Component Palette and right Property in Property Sheet.
- We need to know if user understand toggle-switchers in those two windows easily.
Notes
- P1 Didn't notice the component palette, layout changed via the context menu, in the Component palette:he selected the component and then placed it in to desired place, He immediately noticed the toggle buttons, renaming of buttons did via the context menu.
- P2 Immediately found out where the Component palette was. In the Component palette, he selected the button, then placed it in to desired place. Renaming was done via the context menu, he had no problem with renaming the buttons in the Property sheet. The layout was changed via the context menu.
- P3 changed layout and add the buttons via the context menu. Then he tried to rename the buttons by doubleclicking on them->the AWT button invoked the Source mode->he was really confused. He discovered the toggle buttons, but he mentioned that they are not so visible. He suggested to add these buttons for a class as well(they could be disabled).
- P4 Prepared all palettes so he discovered the Component palette. He changed the layout via the context menu, however at first he invoked the properties and complained about size of the window.For adding the buttons he tried the D&D at first. The name was changed via the context menu.
- P5 wasn't sure about differences between JFrame and JPanel. He expected the layout in the Property sheet->changed the layout via the context menu.He looked for the Component palette in the ToolBar-> found the Palette manager.At last he found the Component palette in the EdgeBar. To add the buttons to the frame he tried D&D at first.The renaming of the buttons was without problems, however at first he tried doubleclick.
- P6 found out switching of view of Editor in View Menu accidentally. He didn't find Palette and added new Component from Contextual menu above Form editor area. He understood switching by toggle buttons in Properties window. He switched into Source view by clicking on node in Projects window, but immediately after that he noticed toggle buttons in the Editor's toolbar.
- P7 immediately discovered the Component palette. The layout was changed via the context menu, however mentioned he used to finding it in the Property sheet. For changing the name he tried double click->to change from Source to Design, he almost closed the project(couldn't find the switchers-he expected TABs). In the design mode, renaming was done via the context menu.
- P8 looked for the GUI editor in the panel Tools, then in the panel File. After creating a frame, he waned to change the layout in the Property sheet, then he changed it via the context menu. He switched on the Component palette, for inserting the buttons he tried D&D function. For renaming of the buttons he used doubleclick, then context menu.
Summary
- Two participants didn't have any problem.
- LAYOUT: Seven participants changed the layout via the context menu, four participants looked for this feature in the Property sheet.
- COMPONENT PALETTE: Four participants discovered minimized Component palette in the EdgeBar, two participants didn't find it, two users invoked the components via the context menu.
- RENAMING: Five participants used context menu, one used The Property Sheet.
- ADDING a COMPONENT: Three participants wanted to add the buttons using D&D function.
- RENAMING: Four participants used doubleclick for renaming -> source.
- PROPERTY SHEET: One user complained about size of the Property Sheet. Four users looked for the layout in the Property Sheet.
- OTHERS: One user wasn't sure about difference between JFrame/JPanel
Problems related to WS
- Component Palette shouldn't be in Auto hide mode by default because is too hidden [C-2]
- There is no way how to resize slided window in Auto Hide state [C-3]
Other issues
- Participants wanted to change the layout in the Property sheet - Possible Enhancement
- Participants wanted to use D&D function for adding the buttons
- Possible Enhancement
Task 4: Changing Design->Source, creating a method
Objectives
- Testing of switching between Design and Source views.
- User could switch it by four ways (By doubleclick on component, in Editor's Toolbar, Main Menu and Projects window ) and we need to know whether they are discoverable enough.
Notes
- P1 Immediately noticed the toggle buttons, he tried to write a code inside the guardian blocks.
- P2 Immediately find out where the toggle buttons are, he liked these buttons instead of TABs.
- P3 Switched from Design to Source, added the method and ran the project.
- P4 discovered the toggle buttons immediately, however he would expect them somewhere down. With Running the project, the user was successful, he tried to remember the shortcut.
- P5 immediately noticed the toggle buttons, but he expected the TABs. He tried the CodeCompletion by Ctrl+Space and expected the ToolTip (wasn't sure about from which package the method was).
- P6 added new method from contextual menu above node in Projects window and then used Run File action from Contextual menu too.
- P7 wanted to write into the guarded blocks. Otherwise, the user had no problem.
- P8 tried to switch between Design/Source by pressing the shortcut Ctrl+F12 or F12, then he tried doubleclick on the form. He didn't notice the toggle buttons.
Summary
- Four participants found the toggle buttons immediately, two users found them at last and two users didn't find the switchers (one user used Projects window and the second one tried a shortcut Ctrl+F12/F12/doubleclick).
- Two participants tried to write the method into guarded blocks, same number of
participants expected TABs for switching.
75.0% of participants(6) used the switchers
25% of participants(2) used different method
Users were able to switch into Source View without any problems and most of them used toggle buttons in Editor's Toolbar
Problems related to WS
- There is no keyboard shortcut for switching Editor's view [C-4]
Task 5: In Design mode add an EventHandler
Objectives
- Which way user choose to switch into Design view.
- Testing of discoverability of Events panel in Property Sheet.
Notes
- P1 Switched to the design mode, clicked on the Events in the Property Sheet.
- P2 Switched to the designer, then he used the context menu for adding the EventHandler.
- P3 For different switching to the Design mode, he tried a shortcut, then he looked for it at Window menu, then he found it in View menu.
- P4 Didn't have any problem.
- P5 Used the Property sheet, he mentioned the right mouse button should react on the place, where the buttons are placed. The ToolTip should appear for longer time, he couldn't read it. The Property sheet is too small, needed resizing.
- P6 added the EventHandler via the context menu.
- P7 did it without problems.
- P8 added the EventHandler via the context menu, another possibility for adding the EventHandler would be doubleclick on the buttons.
Summary
- Four participants did not have any problems.
- The EventHandler was added from the EVENT tab in the PROPERTY SHEET twice, from the CONTEXT MENU four times and by double click twice.
- One user switched to the Designer in the menu VIEW.
- One participant mentioned the right mouse button should react on the place, where the buttons are placed.
- The ToolTip should appear for longer time, he could not read it.
- The Property sheet is too small, needed resizing.
- Another user expected another possibility for adding the EventHandler as doubleclick on
the buttons.
87.5% switched to the Design mode by toggle buttons in the Editor's ToolBar
12.5% switched to the Design mode by View>Editors>Design menu action
Problems related to WS
- There is missing Tooltip above toggle buttons in the Editor Toolbar [C-5]
Other issues
- Long text in ToolTip for Events should be visible for longer time or should be cut a bit.
Task 6: Minimizing all windows to the EdgeBar
Objectives
- To check possible ways of minimizing windows (by icons, by Contextual menu, Shortcuts, D&D in the future) and if users understand the icons in titlebar of Windows.
- To find out if users understand term minimize and distinguish it from close state.
- To observe how users switch window to the Sliding state
Notes
- P1 did it, because there are just two buttons (one is closed)
- P2 was successful, because there are just two icons, Why not to use down arrow?
- P3 looked for a feature, which would be able to close all windows.
- P4 looked at icons in minimized windows at first, then he fulfilled this task.
- P5 was successful, because there are just two icons, suggested to have icons in the context menu as well
- P6 tried to click on Auto Hide icons immediately with success. Dock icon he treated as special close icon.
- P7 did it without problems.
- P8 did it without problems.
Summary
- No Problems. However, they don't understand icon for docking. Participants suggested to have icons in the context menu as well.
Problems related to WS
- Participants had problem with appearance of Dock icon and didn't know what it means [C-6]
- Participants expected tooltip above Auto Hide icons [C-7]
Other issues
- Add the icons to the context menu above Titlebar of windows - Possible Enhancement
Task 7: Moving minimized windows to another EdgeBar
Objectives
- To observe D&D usage by switching position of window in Auto Hide state
Notes
- P1 tried D&D function, he tried all 4 ways.The last one, from docked state, was successful
- P2 docked the window at first, then he used D&D function.
- P3 tried D&D the sliding window,then invoked the context menu. He didn't find anything what would be useful for him so he docked the window and moved it.
- P4 tried all three ways. D&D of docked mode was successful.
- P5 tried all three ways. D&D of docked mode was successful.
- P6 tried all three ways. D&D of docked mode was successful.
- P7 tried all three ways. D&D of docked mode was successful.
- P8 couldn't move the windows, so he looked to the Window manager. Then he tried right mouse click on the edgebar, did not work. He looked to the panel View. Didn't fulfill the task.
Summary
- One participant did it immediately, six participants used all possible ways for moving,
one participant didn't fulfill the task.
87.5% of participants(7) fulfilled the task (12.5% of participants(1) did it immediately,
75.0% of participants(6) used all ways of D&D at first before they were successful)
12.5% of participants(1) did not fulfil the task
To support D&D functionality for moving minimized windows is necessary because it was first participants' action in this task
Problems related to WS
- Name of Auto Hide action is not clear enough for users, they expected different name [C-8]
- Users expected D&D by window in Auto Hide mode, when they tried to move to another EdgeBar [C-9]
Task 8: Using keyboard for TASK 6 and 7
Objectives
- To test user's manipulation with windows by keyboard
- To show us, where the users would look for the certain features
Notes
- P1 looked for these features in the panel Window.
- P2 looked for this features in the panel Window. He wasn't successful, so he tried to find out, where he can invoke the context menu, he also looked for the shortcuts (in Options menu)
- P3 discovered the shortcut Ctrl+Backspace for minimizing the window. Also Suggested to add this features to the panel Window.
- P4 would look for these features in panel Window.
- P5 looked for these features in panel Window, he also suggested to have panels, which aren't active, disabled.
- P6 looked into the View menu, but didn't find it. Was confused from suddenly appearing Auto Hide window. Tried Ctrl+F4 shortcut and Ctrl+1 shortcut, then tried Enter for docking.
- P7 tried shortcut Ctrl+Tab, ESC, then he was looking for the shortcut in the panel Tools. He also looked for these features in Help.
- P8 tried some shortcuts (e.g.arrows). He also looked to the panel Window, then to the panel View, he expected something like Move to...
Summary
- One participant discovered the shortcut for minimizing. Six participant looked for these
features in the panel Window, three in the panel View, One in the panel Tools and panel
Help. One participant used a shortcut Ctrl+Tab, Esc, Arrows.
Best place for Auto Hide action and assigned shortcut is Window menu:
12.5% of participants(1) discovered the shortcut
12.5% of participants(1) tried shortcuts, such as Ctrl+Tab, Esc, Arrows
75.0% of participants(6) looked for these features in the Window menu
37.5% of participants(3) looked for these features in the View menu
12.5% of participants(6) looked for these features in the Tools menu
12.5% of participants(3) looked for these features in the Help menu
Problems related to WS
- There is missing Auto Hide (Minimize) menu item in Window menu [C-10]
- There is no shortcut for moving windows inside IDE [C-11]
Conclusions and Recommendations
We found following problems during usability study and separated them into by priority:
Highest priority (3)
- There is no way how to resize slided window in Auto Hide state [C-3]
- There is no keyboard shortcut for switching to Editor's views [C-4]
- Users expected D&D by window in Auto Hide mode, when they tried to move to another EdgeBar [C-9]
Higher priority (5)
- Component Palette shouldn't be in Auto hide mode by default because is too hidden [C-2]
- There is missing Tooltip above toggle buttons in the Editor Toolbar [C-5]
- Participants had problem with appearance of Dock icon and didn't know what it means [C-6]
- Participants expected tooltip above icons in Titlebar of each window [C-7]
- There is missing Auto Hide (Minimize) menu item in Window menu [C-10]
Lower priority (3)
- ECS shortcut is hidden a bit, but when user found it, said that it's useful [C-1]
- Name of Auto Hide action is not clear enough for users, they expected different name [C-8]
- There is no shortcut for moving windows inside IDE [C-11]
We will describe them in detail in the following part of document.
[C-1] ECS shortcut is hidden a bit, but when user found it, said that it's useful
Users canceled the Output window (which slided into work area of IDE) by mouse click
and there is no way how to find assigned shortcut (ESC) for this action.
Because of this problem:
- User used another way how to cancel Output area, mostly mouse click
Recommendation
There could exist tooltip above the Output window (or it's Titlebar at least) and it will contain information about ESC shortcut.
[C-2] Component Palette shouldn't be in Auto hide mode by default because there is too hidden
Users didn't notice the component palette in Auto Hide state,
that's why they added components via the context menu
Because of this problem:
- Users used another way how to add components, but they expected visible Palette with components close to the Form editor, not in Auto Hide state
- Users were confused by missing Palette at the start of this task, then found out another way how to add components, but it takes to much time.
Recommendation
Simply, the Component Palette should be visible in docked state by default.
[C-3] There is no way how to resize slided window in Auto Hide state
Users tried to resize slided window in Auto Hide state, but
weren't successful.
Because of this problem:
- User tried it more times by different style of dragging, but without success
Recommendation
To Implement resizing for window in Auto Hide state similar to standard resizing of docked windows.
[C-4] There is no keyboard shortcut for switching to Editor's views
User wanted to switch into another view in the Editor by
keyboard, but there isn't shortcut for it.
Because of this problem:
- Users tried to switch between views more times by different shortcuts, but without success
- Users are slowed a bit when they prefer keyboard more then mouse manipulation
Recommendation
To add shortcut ALT+SHIFT+LEFT or ALT+SHIFT+RIGHT for navigation among actual views in the Editor.
[C-5] There is missing Tooltip above toggle buttons in the Editor Toolbar
This is related to previous issue. User tried to stay with mouse
cursor above toggle buttons in the Editor for obtaining tooltip with shortcut.
Because of this problem:
- Users weren't able to find right shortcut for view switching
Recommendation
To show name of view (Source View, Design View) with assigned shortcut (ALT+SHIFT+LEFT or ALT+SHIFT+RIGHT) in tooltip above toggle buttons.
[C-6] Participants had problem with appearance of Dock icon and didn't know what does it mean
User treated DOCK icon as special close icon and they
hesitated to use it
Because of this problem:
- Some users didn't want to use this icon-action
- Some users tried it without knowing what it causes
Recommendation
To improve look of this icon a bit.
[C-7] Participants expected tooltip above icons in Titlebar of each window
User wanted to know what exactly icons situated in the Titlebar
of windows causes, but couldn't realize it without trying it.
Because of this problem:
- Users didn't use those icon-actions or hesitated to use it
Recommendation
To add ToolTips for each icon in the Titlebar of Windows.
[C-8] Name of Auto Hide action is not clear enough for users, they expected different name
Users treat Auto Hide action as something else then it sounds.
They realized that it's similar to minimizing of windows while they were using it.
Because of this problem:
- They had a problem to connect name of this action and effect of this action
Recommendation
To discuss this name more and try to use some different one which is more close to their experience maybe
[C-9] Users expected D&D by window in Auto Hide mode, when they tried to move to another EdgeBar
Users wanted to D&D window into Edgebar, between Edgebars and
from Edgebar into work area too.
Because of this problem:
- Users had problem to rearrange layout of IDE
- Users were slowed down in using IDE
Recommendation
To implement D&D support for windows in Auto Hide state, which should fit in current D&D support of windows
[C-10] There is missing Auto Hide (Minimize) menu item in Window menu
User wanted to switch into another view in the Editor by
keyboard, but there isn't shortcut for it.
Because of this problem:
- Users aren't able to change state by windows in Auto Hide mode quickly by keyboard
- Users had problem to found out shortcut assigned to Auto Hide action
Recommendation
To add this menu item into appropriate section in Window menu. It means that section which contains Close Window action and Maximize Window action
[C-11] There is no shortcut for moving windows inside IDE
This isn't so big bug because users use D&D functionality for such type of action
mainly.
Because of this problem:
- However we should support it too for keyboard users or users with some disabilities.
Recommendation
To explore and discuss it more and find out right shortcut(s), which will fit in known shortcuts for manipulation with windows
Another revealed bugs
We found another bugs during usability study, but those bugs aren't connected with WS features. However we would like to list them too:
- Participants wanted to change the layout in the Property sheet - Possible Enhancement
- Participants wanted to use D&D function for adding the buttons - Possible Enhancement
- Long text in ToolTip for Events should be visible for longer time or should be cut a bit.
- Add the icons to the context menu above Titlebar of windows - Possible Enhancement
- .
- .
- TBD...
Other information as a side effect of study