OOBE Usability Test Results: Overview

Sheet 1: Pre-tasks Questionnaire



Proffesional Beginner Proffesional Beginner Proffesional Beginner Beginner Beginner Proffesional 4P/5B

Questions Participant 1 Participant 2 Participant 3 Participant 4 Participant 5 Participant 6 Participant 7 Participant 8 Participant 9 SUMMARY
1 Are you a male/female? M M M M M M M M M 9
2 Are you a student/an employee? S E S E E E S S E 4S/5E
3 What is your education?










Elemetary school










High School










College X
X

X X
X 5

University
X
X X

X
4

Other









4 What kind of position/responsibility do you have in the company?










Emloyee
X

X


X 3

Manager


X




1

Director




X


1

Other






X
1
5 How many people does the company have?










0-25
X


X
X X 4

25-50










50-100


X X



2

More than 100









6 Do you have a personal computer? YES YES YES YES YES YES YES YES YES 9
7 What is typical hardware setting when developping applications? 2GHz P4 1.8Ghz Athlon 2500+ PowerBook 1GHz 1,8Ghz Apple G4 Athlon 1800+ Apple G4


Resolution 1125x812 1024x768 1024x768 1280x854 1280x1024 1280x1024 1280x1024 1152x768 1024x768

OS WIn XP,Linux Win XP Win XP Mac OS X Linux Mac OS X Win 2000 Mac OS X Win 2000

RAM 768MB 512MB 512MB 768MB 512MB 1GB 256MB 768MB 512MB
8 Do you have an Internet access? YES YES YES YES YES YES YES YES YES 9
9 What kind of connection do you have?










Dial-up










ADSL







X 1

Cable


X X X X X
5

Other X X X





3
10 What kind of browser do you use?










Mozilla X X
X X



4

MS Explorer
X X

X X

4

Netscape










Other
X
X
X
X X 5
11 How often do you use a computer in a week?










Once a week










Twice – 3times a week










4-5 times a week





X

1

6-7 times a week X X X X X X
X X 8
12 What programming languages do you know?










Java X X X X X X X X X 9

Visual Basic X

X
X

X 4

PHP
X


X X

3

Delphi
X






1

C/C++
X X
X

X X 5

4 Dimension


X
X


2

Pascal

X




X 2
13 Do you have any experience with Java? YES YES YES YES YES YES YES YES YES 9
14 How long have you been using Java? 6 years 1 year 2 years 3 months 4 years 1 month 1 year 1 year 3 years
15 How often do you use Java in a week?










None or once a week
X
X
X
X X 5

Twice – 3times a week

X


X

2

4-5 times a week










6-7 times a week X


X



2
16 What IDE(s) do you know?










Eclipse X X X
X



4

Kawa X







1

NetBeans X X X X X X X X X 9

JBuilder X X X X

X
X 6

Blue J

X


X

2

Idea



X



1

JCreator





X

1

XCode






X
1

Visual Studio






X X 2

Make Filer






X
1

Code Warrior






X
1

JDeveloper







X 1

Deplhi







X 1

4 Dimension


X
X


2
17 What IDE(s) do you use for your work? Eclipse Ace HTML BlueJ JBuilder 4 Dimension NetBeans 4 Dimension JCreator CodeWarrior Delphi
18 What advantages/disadvantages does the IDE have?









19 What kind of tasks do you use your IDE for? DesktopApps WebApps SchoolProjects SchoolProjects UserInterface Databases WebApps GUI Databases Apps SchoolProjects UserInterface Plugins

20 Did you work in a different IDE in the past? Kawa, NetBeans JBuilder, Delphi, C++ JBuilder
Idea FileMaker BlueJ CodeWarrior Delphi
21 What was the reason for migrating? Kawa – finished


GUI, TomCat Capacity Editor CodeWarrior – finished



NetBeans – functionality is missing








22 What do you think the “right IDE” would have? 1.hierarchical structured files, projects 1.shortcuts 1.support of logging 1.should be able to do all stuff with a project 1.GUI 1.simplicity 1.hierarchical structured files, projects 1.hierarchical structured files, projects 1.GUI


2. links for all libraries, which can be used or are usad 2.syntax 2.generating of JavaDoc
2.JavaDoc 2.on-line documentation 2.Enter->next line 2.API



3.refactoring(right mouse button) 3.CodeCompletion 3.import/export of projects
3.debugger 3.compatibility
3.CodeCompletion



4.help
4.shortcuts
4.profiler 4.userfriendliness
4.debugger

23 Who is the decisionmaker, what IDE will be used for developping, in the company?
Developer
Himself Developer Himself
Developer

24 What kind of IDE(s) do your colleagues use for developping?
HomeSite, PHPedit
4 Dimension NetBeans, Idea
JCreator, BlueJ


25 Have you ever heard about NetBeans IDE before? YES YES YES YES YES YES YES YES YES
26 Where did you get the information about NetBeans IDE?










Press X







1

Internet


X


X X 3

Friend




X X

2

Colleague










Other

X
X



2
27 What was the information? (scale 1(good)-5(bad))


1 1 4 2


28 Have you ever worked in NetBeans IDE before? YES NO NO YES YES NO YES NO NO
 

Sheet 2: Tasks Questionnaire



Proffesional Beginner Proffesional Beginner Proffesional Beginner Beginner Beginner Proffesional



Task Questions Participant 1 Participant 2 Participant 3 Participant 4 Participant 5 Participant 6 Participant 7 Participant 8 Participant 9



1.
Visit www.netbeans.org and find out, if this IDE is really suitable for you.









What kind of information are you interested in? Internet was out of order













GUI Support of JDBC with mySQL System requirements Screenshots Documentation of applications Basic information Specification of IDE How to change JDKs





Basic information
Specification of IDE How to make the programming easier Documentation of language Info about the highest version CVS GUI editor





System requirements




Help Import











Multiplatform Code Completion


















Did you find them?
YES YES YES NO YES YES





Reason





Noticed features, did not know it is for free





Were you satisfied with these information? (scale 1(yes)-5(no))
1 2 1.5
1 2





What kind of information did not you find? Price Price Specification for czech environment
Documentation in PDF, czech version
Info about CVS, remote debugger





Give us a list of three attributes, which are the most important for you, to start using the IDE?













1 GUI Price GUI System requirements Price Documentation in PDF Price Price Documentation




2 Refactoring GUI Support of SQL Easy to manage the projects Maximum information(JavaDoc, etc.) Multiplatform Recommendation Multiplatform Easy to manage the projects




3 Easy to manage the projects
Clarity Stability GUI, TomCat Recommendation Public information
Control and navigation




Did you find these attributes?

YES YES YES NO
YES





What do you think about NetBeans website? (scale 1(clear)-5(not clear))
2 2 2 2 3 2 2 2




Give us a list of three points what do you like/dislike on NetBeans wesite.













+1
Design Graphics Good structure Tutorial about J2EE
Design Not so many screenshots Positioning of links




+2

Structure


Structure
Fonts




+3







White background and white text




-1
Did not find it is for free Too many texts
Confusing first page of website Czech version is missing
Many mouse clicks(better to have mouse over menu) Graphics




-2

Just english version
Screenshots should be earlier(e.g. In product info)


Shadowing




-3












2. Download and install the latest version of NetBeans 4.0 IDE.










Was it easy/complicated to reach the desired file or not? (scale 1(easy)-5(hard)) 1 1 1 1 1.7 1 2 1 1




How confident are you that you downloaded the correct file? (scale 1(satisfied)-5(not satisfied)) 1 3 1 1 1 2 1 1 1




Was the installation process clear?(scale 1(clear)-5(not clear))
1 1 1 1 1 1 1 1 1




What would have made this task easier? Nothing Nothing Nothing Nothing Less steps Nothing Less steps There are two licence agreements Nothing



3.
Start IDE, browse the menu and functions and tell us, what do you expect behind.









What is your first impression about NetBeans IDE? Good Same as JBuilder Unusual Good, likes welcome screen, it is good for beginners
Common feelings Likes welcome screen Scared of big blue window (Welcome screen) Uninteresting graphics




What is the first thing that springs into your eye? Welcome screen Welcome screen Welcome screen Welcome screen Empty environment, fuzzy pictuer Welcome screen (blue colour) Welcome screen Welcome screen Welcome screen, Files, Projects, Runtime tabs




What windows did you expect to open when you start the IDE? Same Welcome screen, tips Blank window Blank window Sample projects Navigation manu Same Welcome screen, blank windows dod not expect to open files, projects, runtime tabs Blank window




What is your overall first impression of this IDE about layout? (scale 1(good)-5(bad)) 1 1 2 2 1 2 1 2 1




What is your overall first impression of this IDE about appearance? (scale 1(good)-5(bad)) 1 1 2 2 3 2 1 2 2




Another:
1 (Icons can be smaller) Icons are big

1(Look&Feel XP)







4. Select the topic realated to this IDE, which you might be interested in, and explore it with Help.








At first what kind of information in program do you look for? ANT Refactoring, VCS GUI Quick start guide Information about technologies (J2EE, J2ME)
He searched for versioning word He prefers PDF file Importing of old projects, refactoring




Why did/did not use the tutorial? Did not use, just in case he will not know how to continue Did not use, just in case he will not know how to continue Did not use, just in case he will not know how to continue Did not use, just in case he will not know how to continue Did not noticed, otherwise he would use it
Did not use, just in case he will not know how to continue Did not use, just in case he will not know how to continue Did not use, because everything should be intuitive




Do you have any suggestions for us?



Links for J2EE, J2ME (did not find it in Help)

In Help it forwarded him to the website...strange




5. Write and run Hello World program. You can choose if the program display Hello World message a) into console b) into a dilaog or c) into a web browser.





Have you found out any problems with completing this task? NO NO NO NO NO YES (used Help on the website) YES (set up wrong project, did not notice the console) NO NO




What would have made this task easier? Nothing Nothing Nothing Nothing Nothing YES (sample project with documantation about creating) Nothing Nothing Nothing




Do you have any suggestions for us?












6. Now you have about 45 minutes for trying of NetBeans 4.0 IDE out. After finishing you should make a decision if you use it or not and why. (We gave a new prototype of GUI editor to participants)




You have already tried NetBeans IDE. Please could you provide some recommendation, what would be improved? Improve consistency of refactoring,Help starts in fullscreen
GUI (possibility to write into source) Bigger icons in GUI Positioning of objects in GUI GUI Floating palette







If you compare NetBeans IDE to your favourite IDE, what are the main differencies that you see? Eclipse is simplier, better clarity In favourite IDE the outline editor is missing Better GUI, JBuilder has better appearance The favourite IDE has better GUI and can simplier create preview NetBeans is better...there are just important things NetBeans is bigger, editation of palette Dialogs, running of projects, generating of source(GUI) GUI is similiar, CodeCompletion is better





If you just downloaded and started NetBeans IDE at your home or work, in order to chcek it out, what would you do? Would have you same steps as you have done in this task? More programming Differences between JBuilder and NetBeans in GUI JDBC, settings Write simple programs WebApp, GUI, refactoring Same steps GUI, creating of classes, adding of methods via menu Write simple program, investigate what cen be set up





Your decision about using NetBeans IDE? Do not know (3) YES (2.5) YES (2) YES (1) YES (1) YES (1) YES (2) YES (1) YES (1)




Your decision about recommending NetBweans IDE to your friends? YES (for trying) YES (for trying) YES YES YES YES YES YES YES



7. Uninstall NetBeans 4.0 IDE and delete all files you created.










Have you found out any problems with completing this task? NO NO NO NO NO NO NO NO NO




What would have made this task easier? Nothing Install should be in Program files Install should be in Program files Nothing Install should be in Program files Install should be in Program files Nothing Install could do uninstall as well Nothing




Do you have any suggestions for us?












 

Sheet 3: Sources and Graphs 2



NO YES












Is this session enough to decide whether NetBeans IDE is good or bad? 44.44% 55.56%












Your decision about recommending NetBweans IDE to your friends? 0.00% 100.00%














































 

Sheet 4: Post Test Questionaire




Proffesional Beginner Proffesional Beginner Proffesional Beginner Beginner Beginner Proffesional 4P/5B

Questions Participant 1 Participant 2 Participant 3 Participant 4 Participant 5 Participant 6 Participant 7 Participant 8 Participant 9 SUMMARY
1. What do you like/dislike on this study?










Like

Good experience

Nice people




Dislike


Resolution of screen
Could not try it on Mac OS



2. What do you like/dislike on NetNeans IDE?










Like
Outline Editor GUI
GUI Similar environment as Apple has Whole IDE
Environment



Help

Speed of NetBeans 4.0
GUI
Outline Editor










UML

Dislike
Refactoring Graphical appearance GUI – needs to be improved Could not find some things in Help (uninstall) “fat” icons English version GUI – needs to be improved




GUI


Could not find Tutorial
Window menu – confusing










Help-connect to the Inet

3. Is this session enough to decide whether NetBeans IDE is good or bad? NO NO YES YES NO YES NO YES YES

Reason Needs more programming
It is free, layout
Decision is based on long satisfaction Good first impression after trying it out Needs more programming


4. How much time would the familiarizing with NetBeans approxmately take you? 1 week 1 day (proffesional – if you have an experience with other IDEs) 1 day (beginner) and 1 month (professional) 1 month (beginner) and 3-6 months (proffesional) 1 month (proffesional) 1 week (beginner) more than 6 month (proffesional) 1 month 1 week 1 month
5. How comprehensible were the menu items for you? (scale 1(clear)-5(not clear)) 1 1 2 3 1 1 1 2.5 1 1.5

Reason

Many items in Window menu






6. How comprehensible were the icons for you? (scale 1(clear)-5(not clear)) 1 1 3 2 2 1 2 2 1 1.67

Reason





Some of them were not clear


7. Do you have any suggestions for us?









Sheet 5: Sources and Graphs 3


Proffesional(1) Proffesional(3) Proffesional(5) Proffesional(9) Beginner(2) Beginner(4) Beginner(6) Beginner(7) Beginner(8) AVG


How comprehensible were the menu items for you? (scale 1(clear)-5(not clear)) 1 2 1 1 1 3 1 1 2.5 1.5























































































































































Proffesional(1) Proffesional(3) Proffesional(5) Proffesional(9) Beginner(2) Beginner(4) Beginner(6) Beginner(7) Beginner(8) AVG


How comprehensible were the icons for you? (scale 1(clear)-5(not clear)) 1 3 2 1 1 2 1 2 2 1.67




































































































































































































Proffesional(1) Proffesional(3) Proffesional(5) Proffesional(9) AVG proffesionals
Beginner(2) Beginner(4) Beginner(6) Beginner(8) AVG Beginners
AVG
Your decision about using NetBeans IDE?(scale 1(yes)-5(no)
3 2 1 1 1.75
2.5 1 1 1 1.5
1.61























































































































































Sheet 6: Times

Hello World











Participant Time(min)
Proffesional(1) 2.53
Proffesional(3) 4.42
Proffesional(5) 2.25
Proffesional(9) 6.43
AVG proffesionals 3.91


Beginner(2) 3.29
Beginner(4) 5.26
Beginner(6) 6.14
Beginner(7) 4.56
Beginner(8) 4.11
AVG beginners 4.67


AVG 4.33










































Project Features

About this Project

Usersguide was started in November 2009, is owned by kganfield, and has 59 members.
By use of this website, you agree to the NetBeans Policies and Terms of Use (revision 20160708.bf2ac18). © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo
 
 
Close
loading
Please Confirm
Close