WS Usability Study Report

Author: Jindrich Dinga, Dusan Pavlica
Version: First draft - 09/22/2004
Study Dates: September 6th and 7th, 2004

 

Table of Contents:

• Introduction
• Participants
• Tasks Summary
• Task Analysis
  • Task 1: Looking through the IDE
  • Task 2: Creating a simple Java application
  • Task 3: Creating a form with Swing and AWT buttons
  • Task 4: Changing Design->Source, creating a method
  • Task 5: In Design mode add an EventHandler
  • Task 6: Minimizing all windows to the EdgeBar
  • Task 7: Moving minimized windows to another EdgeBar
  • Task 8: Using keyboard for TASK 6 and 7
    
    
• Conclusions and Recommendations
  • ECS shortcut is hidden a bit, but when user found it, said that it's useful [C-1]
  • Component Palette shouldn't be in Auto hide mode by default because is too hidden [C-2]
  • There is no way how to resize slided window in Auto Hide state [C-3]
  • There is no keyboard shortcut for switching to Editor's views [C-4]
  • There is missing Tooltip above toggle buttons in the Editor Toolbar [C-5]
  • Participants had problem with appearance of Dock icon and didn't know what does it mean [C-6]
  • Participants expected tooltip above icons in Titlebar of each window [C-7]
  • Name of Auto Hide action is not clear enough for users, they expected different name [C-8]
  • Users expected D&D by window in Auto Hide mode, when they tried to move to another EdgeBar [C-9]
  • There is missing Auto Hide (Minimize) menu item in Window menu [C-10]
  • There is no shortcut for moving windows inside IDE [C-11]
    
    
• Another revealed bugs
• Other information as a side effect of study
  

 

Introduction

The usability study was focused on observing the first hour experience when using Window system features (Sliding window and Multiview) included in NetBeans IDE. Eight participants were asked to complete 8 tasks using NetBeans IDE Beta1 build. This usability was special a bit, because we needed to observe detail behavior of participants not only their hit ratio in tasks. That is why we decided to have two observers in the usability room.

Usability study has revealed some UI problems which could cause really substantial problems to user. E.g. some of most visible:

• There is no way how to resize slided window in Auto Hide state [C-3]
• Users expected D&D by window in Auto Hide mode, when they tried to move to another EdgeBar [C-9]

Another group of problems is not so big, however after their fixing mentioned WS features in Netbeans would be easy to use during the first hour of usage. Then usability of those features could be increased too. E.g. some typical problems:

• Participants had problem with appearance of Dock icon and didn't know what does it mean [C-6]
• Participants expected tooltip above icons in Titlebar of each window [C-7]
• Component Palette shouldn't be in Auto hide mode by default because is too hidden there [C-2]

In the study we noted 11 UI problems (till this time) and obstacles directly related to UI of Netbeans Beta1 build and all are described in detail below.

 

Participants

We could categorize participants according to programming experience as:

• 2 student (P3, P7)
• 3 junior programmers  (P1, P2, P8)
• 3 experienced programmers (P4, P5, P6)

The full participant characteristics will be found here: Participants.

 

Tasks Summary

Nine tasks were focused on testing following:

• Task 1: Looking through the IDE
• Task 2: Creating a simple Java application
• Task 3: Creating a form with Swing and AWT buttons
• Task 4: Changing Design->Source, creating a method
• Task 5: In Design mode add an EventHandler
• Task 6: Minimizing all windows to the EdgeBar
• Task 7: Moving minimized windows to another EdgeBar
• Task 8: Using keyboard for TASK 6 and 7

The full task's script can be found here: Task Script.
Participants finished nearly all tasks but with problems mentioned below.

 

Task Analysis

 

Task 1: Looking through the IDE

Objectives

• Maybe strange layout of windows will provoke the user into rearranging it.
• We wanted to observe if he uses the Sliding window or Close action, eventually D&D.

Notes

• P1 everything moved to the left side – used to having it, because of Eclipse, used doubleclick to maximize the window. Tried Auto Hide of the Output window by mouse focusing.
• P2 looked just through the IDE without any special action.
• P3 arranged the layout by using D&D function. He invoked the Component palette, he didn't understand the icons. He also looked for switchers between Source/Design, he thought, there should be TABs.
• P4 arranged the layout by using D&D function. Looking through the toolbar. Looking what features the IDE/Refactoring could offer.
• P5 arranged the layout by D&D function. He was satisfied with the icons in the ToolBar. He wasn't sure about the meaning of buttons in TODO window, D&D function is slow.
• P6 spent a lot of time by browsing menu, Options dialog and Properties dialog. He had problem to dock this Properties dialog, because he thought that it's Properties window.
• P7 looked just through the IDE.
• P8 looked through the IDE. He wanted to change the keyboard profile.

Summary

• All participants had different comments about the IDE based on their experiences with other IDEs.

 

Task 2: Creating a simple Java application

Objectives

• Just checking if user understands Edit/Compile/Run cycle, which is important for next tasks.
• We wanted to find out how users cancel the Output window in Sliding state (ESC, click, key press).

Notes

• P1 is used to programming with Code Completion
• P2 did it without any problems. For Run cycle, he uses F6.
• P3 started by icon in the ToolBar.
• P4 run the project via the context menu. The users mentioned, “Build Successful” is redundant in the Output window.
• P5 was successful with creating the Java app. He saved the project by shortcut Ctrl+S, he noticed it was saved because of *.
• P6 tried to find abbreviations and then used Code Completion by shortcut.
• P7 created the project via the Welcome screen. For writing a code, the user used Code completion (Ctrl+Space).
• P8 did it without problems.

Summary

• No problems. One participant said, “Build Successful” is redundant in the Output Window.
• One participant used a shortcut Ctrl+S for saving.
• One participant used a shortcut F6 for running, one participant used the icon for running.
• Three users used a shortcut Ctrl+space for Code completion.
  
  100.0% of participants(8) found out canceling the Output window by mouse click easily

Problems related to WS

• ECS shortcut is hidden a bit, but when user found it, said that it is useful [C-1]

 

Task 3: Creating a form with Swing and AWT buttons

Objectives

• We test whether user user is able to find Palette in Auto Hide state situated at the right Edgebar
• We test whether user user is able to find right component in Component Palette and right Property in Property Sheet.
• We need to know if user understand toggle-switchers in those two windows easily.

Notes

• P1 Didn't notice the component palette, layout changed via the context menu, in the Component palette:he selected the component and then placed it in to desired place, He immediately noticed the toggle buttons, renaming of buttons did via the context menu.
• P2 Immediately found out where the Component palette was. In the Component palette, he selected the button, then placed it in to desired place. Renaming was done via the context menu, he had no problem with renaming the buttons in the Property sheet. The layout was changed via the context menu.
• P3 changed layout and add the buttons via the context menu. Then he tried to rename the buttons by doubleclicking on them->the AWT button invoked the Source mode->he was really confused. He discovered the toggle buttons, but he mentioned that they are not so visible. He suggested to add these buttons for a class as well(they could be disabled).
• P4 Prepared all palettes so he discovered the Component palette. He changed the layout via the context menu, however at first he invoked the properties and complained about size of the window.For adding the buttons he tried the D&D at first. The name was changed via the context menu.
• P5 wasn't sure about differences between JFrame and JPanel. He expected the layout in the Property sheet->changed the layout via the context menu.He looked for the Component palette in the ToolBar-> found the Palette manager.At last he found the Component palette in the EdgeBar. To add the buttons to the frame he tried D&D at first.The renaming of the buttons was without problems, however at first he tried doubleclick.
• P6 found out switching of view of Editor in View Menu accidentally. He didn't find Palette and added new Component from Contextual menu above Form editor area. He understood switching by toggle buttons in Properties window. He switched into Source view by clicking on node in Projects window, but immediately after that he noticed toggle buttons in the Editor's toolbar.
• P7 immediately discovered the Component palette. The layout was changed via the context menu, however mentioned he used to finding it in the Property sheet. For changing the name he tried double click->to change from Source to Design, he almost closed the project(couldn't find the switchers-he expected TABs). In the design mode, renaming was done via the context menu.
• P8 looked for the GUI editor in the panel Tools, then in the panel File. After creating a frame, he waned to change the layout in the Property sheet, then he changed it via the context menu. He switched on the Component palette, for inserting the buttons he tried D&D function. For renaming of the buttons he used doubleclick, then context menu.

Summary

• Two participants didn't have any problem.
• LAYOUT: Seven participants changed the layout via the context menu, four participants looked for this feature in the Property sheet.
• COMPONENT PALETTE: Four participants discovered minimized Component palette in the EdgeBar, two participants didn't find it, two users invoked the components via the context menu.
• RENAMING: Five participants used context menu, one used The Property Sheet.
• ADDING a COMPONENT: Three participants wanted to add the buttons using D&D function.
• RENAMING: Four participants used doubleclick for renaming -> source.
• PROPERTY SHEET: One user complained about size of the Property Sheet. Four users looked for the layout in the Property Sheet.
• OTHERS: One user wasn't sure about difference between JFrame/JPanel

Problems related to WS

• Component Palette shouldn't be in Auto hide mode by default because is too hidden [C-2]
• There is no way how to resize slided window in Auto Hide state [C-3]

Other issues

• Participants wanted to change the layout in the Property sheet - Possible Enhancement
• Participants wanted to use D&D function for adding the buttons - Possible Enhancement
  

 

Task 4: Changing Design->Source, creating a method

Objectives

• Testing of switching between Design and Source views.
• User could switch it by four ways (By doubleclick on component, in Editor's Toolbar, Main Menu and Projects window ) and we need to know whether they are discoverable enough.

Notes

• P1 Immediately noticed the toggle buttons, he tried to write a code inside the guardian blocks.
• P2 Immediately find out where the toggle buttons are, he liked these buttons instead of TABs.
• P3 Switched from Design to Source, added the method and ran the project.
• P4 discovered the toggle buttons immediately, however he would expect them somewhere down. With Running the project, the user was successful, he tried to remember the shortcut.
• P5 immediately noticed the toggle buttons, but he expected the TABs. He tried the CodeCompletion by Ctrl+Space and expected the ToolTip (wasn't sure about from which package the method was).
• P6 added new method from contextual menu above node in Projects window and then used Run File action from Contextual menu too.
• P7 wanted to write into the guarded blocks. Otherwise, the user had no problem.
• P8 tried to switch between Design/Source by pressing the shortcut Ctrl+F12 or F12, then he tried doubleclick on the form. He didn't notice the toggle buttons.

Summary

• Four participants found the toggle buttons immediately, two users found them at last and two users didn't find the switchers (one user used Projects window and the second one tried a shortcut Ctrl+F12/F12/doubleclick).
• Two participants tried to write the method into guarded blocks, same number of participants expected TABs for switching.
  
  75.0% of participants(6) used the switchers
  25% of participants(2) used different method
  
  Users were able to switch into Source View without any problems and most of them used toggle buttons in Editor's Toolbar
  

Problems related to WS

• There is no keyboard shortcut for switching Editor's view [C-4]

 

Task 5: In Design mode add an EventHandler

Objectives

• Which way user choose to switch into Design view.
• Testing of discoverability of Events panel in Property Sheet.

Notes

• P1 Switched to the design mode, clicked on the Events in the Property Sheet.
• P2 Switched to the designer, then he used the context menu for adding the EventHandler.
• P3 For different switching to the Design mode, he tried a shortcut, then he looked for it at Window menu, then he found it in View menu.
• P4 Didn't have any problem.
• P5 Used the Property sheet, he mentioned the right mouse button should react on the place, where the buttons are placed. The ToolTip should appear for longer time, he couldn't read it. The Property sheet is too small, needed resizing.
• P6 added the EventHandler via the context menu.
• P7 did it without problems.
• P8 added the EventHandler via the context menu, another possibility for adding the EventHandler would be doubleclick on the buttons.

Summary

• Four participants did not have any problems.
• The EventHandler was added from the EVENT tab in the PROPERTY SHEET twice, from the CONTEXT MENU four times and by double click twice.
• One user switched to the Designer in the menu VIEW.
• One participant mentioned the right mouse button should react on the place, where the buttons are placed.
• The ToolTip should appear for longer time, he could not read it.
• The Property sheet is too small, needed resizing.
• Another user expected another possibility for adding the EventHandler as doubleclick on the buttons.
  
  87.5% switched to the Design mode by toggle buttons in the Editor's ToolBar
  12.5% switched to the Design mode by View>Editors>Design menu action
  
  

Problems related to WS

• There is missing Tooltip above toggle buttons in the Editor Toolbar [C-5]

Other issues

• Long text in ToolTip for Events should be visible for longer time or should be cut a bit.

 

Task 6: Minimizing all windows to the EdgeBar

Objectives

• To check possible ways of minimizing windows (by icons, by Contextual menu, Shortcuts, D&D in the future) and if users understand the icons in titlebar of Windows.
• To find out if users understand term minimize and distinguish it from close state.
• To observe how users switch window to the Sliding state

Notes

• P1 did it, because there are just two buttons (one is closed)
• P2 was successful, because there are just two icons, Why not to use down arrow?
• P3 looked for a feature, which would be able to close all windows.
• P4 looked at icons in minimized windows at first, then he fulfilled this task.
• P5 was successful, because there are just two icons, suggested to have icons in the context menu as well
• P6 tried to click on Auto Hide icons immediately with success. Dock icon he treated as special close icon.
• P7 did it without problems.
• P8 did it without problems.

Summary

• No Problems. However, they don't understand icon for docking. Participants suggested to have icons in the context menu as well.

Problems related to WS

• Participants had problem with appearance of Dock icon and didn't know what it means [C-6]
• Participants expected tooltip above Auto Hide icons [C-7]

Other issues

• Add the icons to the context menu above Titlebar of windows - Possible Enhancement
  

 

Task 7: Moving minimized windows to another EdgeBar

Objectives

• To observe D&D usage by switching position of window in Auto Hide state

Notes

• P1 tried D&D function, he tried all 4 ways.The last one, from docked state, was successful
• P2 docked the window at first, then he used D&D function.
• P3 tried D&D the sliding window,then invoked the context menu. He didn't find anything what would be useful for him so he docked the window and moved it.
• P4 tried all three ways. D&D of docked mode was successful.
• P5 tried all three ways. D&D of docked mode was successful.
• P6 tried all three ways. D&D of docked mode was successful.
• P7 tried all three ways. D&D of docked mode was successful.
• P8 couldn't move the windows, so he looked to the Window manager. Then he tried right mouse click on the edgebar, did not work. He looked to the panel View. Didn't fulfill the task.

Summary

• One participant did it immediately, six participants used all possible ways for moving, one participant didn't fulfill the task.
  
  87.5% of participants(7) fulfilled the task (12.5% of participants(1) did it immediately,
  75.0% of participants(6) used all ways of D&D at first before they were successful)
  12.5% of participants(1) did not fulfil the task
  
  To support D&D functionality for moving minimized windows is necessary because it was first participants' action in this task
  

Problems related to WS

• Name of Auto Hide action is not clear enough for users, they expected different name [C-8]
• Users expected D&D by window in Auto Hide mode, when they tried to move to another EdgeBar [C-9]

 

Task 8: Using keyboard for TASK 6 and 7

Objectives

• To test user's manipulation with windows by keyboard
• To show us, where the users would look for the certain features

Notes

• P1 looked for these features in the panel Window.
• P2 looked for this features in the panel Window. He wasn't successful, so he tried to find out, where he can invoke the context menu, he also looked for the shortcuts (in Options menu)
• P3 discovered the shortcut Ctrl+Backspace for minimizing the window. Also Suggested to add this features to the panel Window.
• P4 would look for these features in panel Window.
• P5 looked for these features in panel Window, he also suggested to have panels, which aren't active, disabled.
• P6 looked into the View menu, but didn't find it. Was confused from suddenly appearing Auto Hide window. Tried Ctrl+F4 shortcut and Ctrl+1 shortcut, then tried Enter for docking.
• P7 tried shortcut Ctrl+Tab, ESC, then he was looking for the shortcut in the panel Tools. He also looked for these features in Help.
• P8 tried some shortcuts (e.g.arrows). He also looked to the panel Window, then to the panel View, he expected something like “Move to...”

Summary

• One participant discovered the shortcut for minimizing. Six participant looked for these features in the panel Window, three in the panel View, One in the panel Tools and panel Help. One participant used a shortcut Ctrl+Tab, Esc, Arrows.
  
  Best place for Auto Hide action and assigned shortcut is Window menu:
  12.5% of participants(1) discovered the shortcut
  12.5% of participants(1) tried shortcuts, such as Ctrl+Tab, Esc, Arrows
  75.0% of participants(6) looked for these features in the Window menu
  37.5% of participants(3) looked for these features in the View menu
  12.5% of participants(6) looked for these features in the Tools menu
  12.5% of participants(3) looked for these features in the Help menu
  

Problems related to WS

• There is missing Auto Hide (Minimize) menu item in Window menu [C-10]
• There is no shortcut for moving windows inside IDE [C-11]

 

Conclusions and Recommendations

We found following problems during usability study and separated them into by priority:

Highest priority (3)

• There is no way how to resize slided window in Auto Hide state [C-3]
• There is no keyboard shortcut for switching to Editor's views [C-4]
• Users expected D&D by window in Auto Hide mode, when they tried to move to another EdgeBar [C-9]

Higher priority (5)

• Component Palette shouldn't be in Auto hide mode by default because is too hidden [C-2]
• There is missing Tooltip above toggle buttons in the Editor Toolbar [C-5]
• Participants had problem with appearance of Dock icon and didn't know what it means [C-6]
• Participants expected tooltip above icons in Titlebar of each window [C-7]
• There is missing Auto Hide (Minimize) menu item in Window menu [C-10]

Lower priority (3)

• ECS shortcut is hidden a bit, but when user found it, said that it's useful [C-1]
• Name of Auto Hide action is not clear enough for users, they expected different name [C-8]
• There is no shortcut for moving windows inside IDE [C-11]

We will describe them in detail in the following part of document.

[C-1] ECS shortcut is hidden a bit, but when user found it, said that it's useful

Users canceled the Output window (which slided into work area of IDE) by mouse click and there is no way how to find assigned shortcut (ESC) for this action.
Because of this problem:

• User used another way how to cancel Output area, mostly mouse click

Recommendation

There could exist tooltip above the Output window (or it's Titlebar at least) and it will contain information about ESC shortcut.

 

 

[C-2] Component Palette shouldn't be in Auto hide mode by default because there is too hidden

Users didn't notice the component palette in Auto Hide state,   that's why they added components via the context menu
Because of this problem:

• Users used another way how to add components, but they expected visible Palette with components close to the Form editor, not in Auto Hide state
• Users were confused by missing Palette at the start of this task, then found out another way how to add components, but it takes to much time.

Recommendation

Simply, the Component Palette should be visible in docked state by default.

 

 

[C-3] There is no way how to resize slided window in Auto Hide state

Users tried to resize slided window in Auto Hide state, but weren't successful.
Because of this problem:

• User tried it more times by different style of dragging, but without success

Recommendation

To Implement resizing for window in Auto Hide state similar to standard resizing of docked windows.

 

 

[C-4] There is no keyboard shortcut for switching to Editor's views

User wanted to switch into another view in the Editor by keyboard, but there isn't shortcut for it.
Because of this problem:

• Users tried to switch between views more times by different shortcuts, but without success
• Users are slowed a bit when they prefer keyboard more then mouse manipulation

Recommendation

To add shortcut ALT+SHIFT+LEFT or ALT+SHIFT+RIGHT for navigation among actual views in the Editor.

 

 

[C-5] There is missing Tooltip above toggle buttons in the Editor Toolbar

This is related to previous issue. User tried to stay with mouse cursor above toggle buttons in the Editor for obtaining tooltip with shortcut.
Because of this problem:

• Users weren't able to find right shortcut for view switching

Recommendation

To show name of view (Source View, Design View) with assigned shortcut (ALT+SHIFT+LEFT or ALT+SHIFT+RIGHT) in tooltip above toggle buttons.

 

 

[C-6] Participants had problem with appearance of Dock icon and didn't know what does it mean

User treated DOCK icon as special close icon and they hesitated to use it
Because of this problem:

• Some users didn't want to use this icon-action
• Some users tried it without knowing what it causes

Recommendation

To improve look of this icon a bit.

 

 

[C-7] Participants expected tooltip above icons in Titlebar of each window

User wanted to know what exactly icons situated in the Titlebar of windows causes, but couldn't realize it without trying it.
Because of this problem:

• Users didn't use those icon-actions or hesitated to use it

Recommendation

To add ToolTips for each icon in the Titlebar of Windows.

 

 

[C-8] Name of Auto Hide action is not clear enough for users, they expected different name

Users treat Auto Hide action as something else then it sounds. They realized that it's similar to minimizing of windows while they were using it.
Because of this problem:

• They had a problem to connect name of this action and effect of this action

Recommendation

To discuss this name more and try to use some different one which is more close to their experience maybe

 

 

[C-9] Users expected D&D by window in Auto Hide mode, when they tried to move to another EdgeBar

Users wanted to D&D window into Edgebar, between Edgebars and from Edgebar into work area too.
Because of this problem:

• Users had problem to rearrange layout of IDE
• Users were slowed down in using IDE

Recommendation

To implement D&D support for windows in Auto Hide state, which should fit in current D&D support of windows

 

 

[C-10] There is missing Auto Hide (Minimize) menu item in Window menu

User wanted to switch into another view in the Editor by keyboard, but there isn't shortcut for it.
Because of this problem:

• Users aren't able to change state by windows in Auto Hide mode quickly by keyboard
• Users had problem to found out shortcut assigned to Auto Hide action

Recommendation

To add this menu item into appropriate section in Window menu. It means that section which contains Close Window action and Maximize Window action

 

 

[C-11] There is no shortcut for moving windows inside IDE

This isn't so big bug because users use D&D functionality for such type of action mainly.
Because of this problem:

• However we should support it too for keyboard users or users with some disabilities.

Recommendation

To explore and discuss it more and find out right shortcut(s), which will fit in known shortcuts for manipulation with windows

 

 

 

Another revealed bugs

We found another bugs during usability study, but those bugs aren't connected with WS features. However we would like to list them too:

• Participants wanted to change the layout in the Property sheet - Possible Enhancement
• Participants wanted to use D&D function for adding the buttons - Possible Enhancement
• Long text in ToolTip for Events should be visible for longer time or should be cut a bit.
• Add the icons to the context menu above Titlebar of windows - Possible Enhancement
• .
• .
• TBD...

 

Other information as a side effect of study