Ratings Key
Rating
Spread
The
Instrument
Read the article:
The
State Of Children's Software
Evaluation--
Yesterday, Today And In
The 21st
Century
|
How We Rate Interactive Media:
About the Ratings, and CTR's Software Evaluation
Instrument
Here's the evaluation instrument that
we use to do our full-length reviews (below).
If you're looking for an easy to print form to use with test families (especially good for video games, here's the Serious Games Testers Evaluation form, used by the Mediatech foundation. It is a PDF. Please keep in mind that these forms are copyrighted. If you use them, please attribute the source, and don't modify them without a proper reference. If you do use them and have a suggestion, please feel free to contact the author, Warren Buckleitner (warren at childrenssoftware.com).
The six
categories help you keep in mind what it takes to make a
quality children's software title. This instrument favors
software that is easy to use, child
controlled, has solid educational content, is engaging and fun, is designed
with features you'd expect to see, and is worth
the money.
To be safe, we recommend staying with a rating above 4
stars. CTR editors typically recommend programs that receive a 4.3
star rating or better. You can easily search for these
titles in the Children's
Software Finder (just enter 4.2 or greater for the
rating search field.
Some Interesting Facts About the
Ratings
The ratings are assigned by educators
(CTR reviewers) who have been "trained" on the use of the
instrument below. The word trained means that they have achieved inter-rater reliability when rating
the same title independently. In cases when two reviewers come up with different ratings,
a third reviewer is consulted, along with additional child testing until
all raters "can sleep with" the rating. The system is not perfect; but
it attempts to be the "least worst" rating system. In assigning
ratings, the reviewers consider feedback
from test schools and families.
Key: 1=dud, 2=poor, 3=fair, 4=good,
5=excellent
CTR's
Checklist for Quality
To get a program's 1-5 star rating, you need to
do some simple math; a process that is automated for reviewers using the instrument, but is defined in detail here, for replication purposes. Add up the points in each category
(always = 1 point, some extent = .5 points,
never = 0 points, and N.A. = Not Averaged) and
then divide by the number of items in the category. This
number can then be converted to a 0 to 5 point scale. It is
important to match the instrument with the type of software.
In other words, you can't rate a program low in "Educational
Value" if it is designed primarily as a game. Even though
That's where the "N.A." category comes in. Finally, it is
very important to consider the date that the review was
written (in the Children's Software Finder, all you can do
is check the copyright date, which is a searchable field).
Remember that a highly rated program in 1993 might be equal
to a poorly rated program in the context of current day
software and hardware.For a more in-depth discussion of the
art and craft of software evaluation, consult the article by
CTR editor Warren Buckleitner:
The
State Of Children's Software
Evaluation--
Yesterday, Today And In The
21st
Century.
TEACHERS & LIBRARIANS:
PERMISSION TO COPY THIS PAGE IS GRANTED
Children's Software Evaluation
Instrument
© 1998 Children's Technology Review
I. Ease of Use (Can a child can use
it with minimal help?)
A
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
S. E. N. N.A.
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
|
S.
E. N N.A.
Skills needed to operate
the program are in range of the child
Children can use the program independently after
the first use
Accessing key menus is straightforward
Reading ability is not prerequisite to using the
program
Graphics make sense to the intended user
Printing routines are simple
It is easy to get in or out of any activity at any
point
Getting to the first menu is quick and easy
Controls are responsive to the touch
Written materials are helpful
Instructions can be reviewed on the screen, if
necessary
Children know if they make a mistake
Icons are large and easy to select with a moving
cursor
Installation procedure is straightforward and easy
to do
|
II. Childproof (Is it designed with
"child-reality" in mind?)
1
2
3
4
5
6
7
8
9
10
|
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
|
Survives the "pound on the
keyboard" test
Offers quick, clear, obvious response to a child?s
action
The child has control over the rate of display
The child has control over exiting at any time
The child has control over the order of the
display
Title screen sequence is brief or can be
bypassed
When a child holds a key down, only one input is
sent to the computer
Files not intended for children are safe
Children know when they've made a mistake
This program would operate smoothly in a home or
classroom setting
|
III. Educational (What can a child
learn from this program?)
1
2
3
4
5
6
7
8
9
10
11
12
13
|
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
|
Offers a good presentation of
one or more content areas
Graphics do not detract from the program's
educational intentions
Feedback employs meaningful graphic and sound
capabilities
Speech is used
The presentation is novel with each use
Good challenge range (this program will grow with
the child)
Feedback reinforces content (embedded
reinforcements are used)
Program elements match direct experiences
Content is free from gender bias
Content is free from ethnic bias
A child's ideas can be incorporated into the
program
The program comes with strategies to extend the
learning
There is a sufficient amount of content
|
IV. Entertaining (Is this program fun
to use?)
1
2
3
4
5
6
7
8
9
|
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
|
The program is enjoyable to
use
Graphics are meaningful and enjoyed by children
This program is appealing to a wide audience
Children return to this program time after time
Random generation techniques are employed in the
design
Speech and sounds are meaningful to children
Challenge is fluid, or a child can select own
level.
The program is responsive to a child's actions
The theme of the program is meaningful to
children
|
V. Design Features (How "smart" is
this program?)
1
2
3
4
5
6
7
8
9
|
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
__ __ __ __
|
The program has speech
capacity
Has printing capacity
Keeps records of child's work
"Branches" automatically: challenge level is
fluid
A child's ideas can be incorporated into the
program.
Sound can be toggled or adjusted
Feedback is customized in some way to the
individual child
Program keeps a history of the child's use over a
period of time
Teacher/parent options are easy to find and
use
|
VI. Value (How much does it cost vs.
what it does? Is it worth it?)
Considering the factors rated above, and
the average retail price of software, rate this
program's relative value considering the current software
market. Consider also any extra hardware attachments
required to get full potential of the programming, e.g., a
sound card, CD-ROM, etc.
Poor
|
|
|
|
|
Good
|
__
|
__
|
__
|
__
|
__
|
__
|
__
|
__
|
__
|
__
|
1
|
2
|
3
|
4
|
5
|
6
|
7
|
8
|
9
|
10
|
Don't forget that this form is generic! To use it properly, you have to look at a lot of similarly designed products, and that the "NA" field is particularly powerful in the overall score.
|