main contents | introduction | Pedagogy & design | Evaluation & use | screen shots | web links | references

Italia 2000 Video Software

Evaluation and Use
Dublin-Cambridge Research

Message 1: Plans and Timetable
Message 2: Software Design Summary and Research Questions


Message 1

Date: Thu, 5 Mar 1998 11:38:32 +0100
From: Cormac O Cuilleanain 
To: gb218@cus.cam.ac.uk
Cc: orourkeb@tcd.ie, bsimpson@tcd.ie, cocullnn@tcd.ie
Subject: Italia 2000 research project

Dear Gavin,

Discussions this morning are moving towards the following formula:

Our focus will be on styles of learning and interaction.

We pick 3 pairs of students, and 3 single students, to work with a segment
of CD-ROM for a normal 50-minute self-access session. Probably in their
final term of first year (having had 2 very intensive terms of language
learning).

Cormac chooses one segment of CD-ROM, by agreement with the students' video
class teacher.

Barbara and Breffni write a theoretical background on the analysis of
learner interaction.

Gavin sends a brief memorandum on what he feels are the main advantages of
the format which you devised for the CD-ROM, and the expected style of use
or learning outcomes from any given session.

We incorporate points from that memorandum in setting the agenda for our
post-session questionnaire-based interview.

We show them the clip, introduce them to the on-screen controls, then video
them as they use the CD-ROM, with two cameras producing a split-screen tape
showing both the computer screen and the students. We ask the singleton
users to pause every ten minutes and reflect on what they are doing, noting
their learning activities on open form sheets. At the end of the session
they complete do the questionnaire-based interview incorporating points
from Gavin's memorandum. Total time, including half-hour debriefing
interview, = 90 minutes.

The objective is to focus on the degree of engagement with the material,
and the way in which the type of material being worked affects the style of
interaction and/or engagement. The interaction between pairs may guide us
towards the points of difficulty which we can then elicit from  the single
users.

We then (when I say we, I mean those of us who know about these things!)
analyse the data, draft a paper and send it to you together with the tape
showing the session, plus the responses from questionnaires, etc.

The final paper will include a more elaborate version of your memorandum on
materials design and expected learning styles and outcomes, rewritten in
the light of what you have seen from the data and analysis, and possibly
including some thoughts on future developments in materials design.


We jointly submit the final text for publication, possibly as an Occasional
Paper of the Centre for Language and Communication Studies.

Timescale: Cormac recruits students in Week 1 of next term (30 March-5
April). We record them in weeks 2 and 3 of term. Analyse data after the end
of term.

That's it. (Replicating the data collection in Cambridge would give us more
data, but in fact that would overload our powers of transcription and
analysis. A later study, including data collection, might build on the
findings of this one.)

What do you think? Please respond to all of us.

Regards,

Cormac

Cormac O Cuilleanain
Department of Italian, Trinity College Dublin
cocullnn@tcd.ie
work telephone Dublin 6081527
home telephone Dublin 2831393


Message 2

Date: Thu, 2 Apr 1998 15:50:29 +0100 (BST)
From: Gavin Burnage 
To: Cormac O Cuilleanain 
Cc: orourkeb@tcd.ie, bsimpson@tcd.ie
Subject: Re: Italia 2000 research project

ITALIA 2000 CD SOFTWARE DESIGN

The chief design feature is the linking of video footage into language
learning exercises. To complete an otherwise conventional CALL exercise,
students can view appropriate video segments provided for each question.
They can look at these segments as many times as they need to, with or
without subtitles. Thus, through completing the exercises and viewing the
clip segments, students gain an understanding of the clip, to the point
where they can start using language learned from the clip and written
exercises in speech: the final "dialogue" exercise provided for most of
the clips allows them to attempt this on the basis of what they've learnt.  

In addition, they also get a feel for "real" language, since the video
clips contain authentic Italian speech. To this end, source-language
subtitles are provided. Research suggests that they are a significant help
in language learning (see notably Vanderplank's work on close captions and
teletext subtitles). 

The software is intended to be backup to conventional video viewing and
classroom activity. It is probably also of some use as a standalone
product, without reference to the video tape. 


Interface Design

It's meant to be easy to use (but that's what they all say):

Note I don't use the word "interactive". Since it's now applied to every
computer activity under the sun, it's not clear exactly what it means!

-- Use of VCR symbols should come quite easily to people familiar with
VCRs and videos.

-- Navigation tools are clearly and consistently shown. You can move
forward or back along the trail of exercises for each clip using the
arrows in the bottom right corner. You can return to the unit menu of
exercises by clicking on the Menu button, and thereafter click on it
again to get back to the Menu Principale.  

The current location - that is, unit number and name, and exercise number
and name - are shown centrally at the foot of the screen. 

Navigation round the package should become familiar and straightforward to
students quite quickly, and thereafter become second nature, allowing
concentration on the clip and the language learning. 

-- A Help facility is also available, explaining what to do for each
exercise, and what the various buttons on display do. It is possible to
click on highlighted phrases in the help text, and see illustrations on
the main screen; however, not many people seem to realise this
straightaway. And in general, people tend not to use the Help facility
unless they're really stuck (my observation; I haven't any references to
back it up, though I expect there will be some), preferring intuitive
trial-and-error attempts at using the software unassisted. It will be
interesting to see how much or how little people use the Help available:
the less they resort to using it, it could be argued, the more intuitive
and self-evident the design of the software must be.

-- The feel of the software is open and relaxed: it is assumed that those
using it are motivated enough to want to learn and explore. 
No point scoring and deducting system is used. Answers and feedback are
understated rather than trumpeted.

Anyone who wants to reveal the answers without studying the clip can do so
easily, but they will, of course, learn nothing. Best results - in terms
of language learning - will come to those who choose to explore each
exercise fully with the facilities built in to the software.  

 
-- The choice of font is softer and less formal than those more usually
seen in computer programs. This lends a more relaxed and attractive feel
to the package whilst maintaining legibility at all times. I would expect
students to report, when asked, that they liked the font, and were not
distracted by it, though there will also be those who dislike the font. I
would hope that few mention the font unprompted, since if  they are
comfortable with it, they will work undistracted and unconcerned by it.
However, they may well say  unprompted things like "it looks nice". 

-- The colour scheme is tasteful (?). It is limited to one major
background colour, a blue, and two main foreground colours, yellow and
white, with grey used to de-emphasize. Generally, yellow text or buttons
mean that something happens when it's clicked. Yellow is also used when
textual answers are revealed, purely as a means of highlighting and
contrasting the answers from the rest of the display. 

Particular Points to look out for

1. Navigation

- how quickly do students come to grips with the navigational control of
the software? Do most find it reasonably easy to handle, or is it badly
designed and a barrier to learning?


2. Video review

- The video window on the right hand side of the unit menu pages
are intended to be for reviewing the clip in full: do they use this
facility, or dive straight into the exercises listed? (I guess this may
depend on prior classroom activity or the lack of it).  Do they ever go
back to the unit Menu while working on exercises (or after completing
them) and use the full video review?
 

3. Using video in the exercises

- In the exercises, do students pick up on the video play button provided
for every exercise? Do they use it for each question? Do they re-use it to
listen to the extract over and over? Do they ever control the clip by
stopping it once it's started (the arrow turns into a square "stop" button
while the segment is playing)?


4. Using subtitles with the video

- Do they use the subtitles with the video segments? Do they report the
subtitles as being of use? Do they turn the subtitles on and off
repeatedly, or do they tend to use the subtitles all the time once they're
turned on? Do the subtitles assist them in understanding and learning? Do
they report liking or disliking them?


5. Suitability of exercises

Do they do the whole exercise, or get frustrated or bored and reveal the
answers or move on? Does this vary for certain exercise types? Which
exercises do they like, and which exercises are effective, 
regardless of what they say about them?

Vocabulary - do they retain a passive knowledge of the words from the
clips taught in the exercises? How much do they learn for active re-use?

Comprehension - do they gain a thorough understanding of the clip and its
message? Are some of the true/false, dettagli and other comprehension
exercises too simple for intermediate/advanced learners? Is the
"ricomponi" exercise too difficult? In all cases, what changes and
improvements might be made? 

Dialogue - do they use the dialogue exercises as envisaged - listening to
the question, using the prompts to attempt a recorded answer, and
comparing it to the original? Do they use the exercise flexibly, checking
questions and answers with and without subtitles, and re-recording answers
till they are satisfied? Do the answers they record sound good, or is this
sort of work best done only with teachers or native speakers to assist,
support and correct?

I'd expect single users to to make most use of this exercise, since it was
designed to imitate for a lone user the classroom practice or pairing off
and playing roles in an interview or discussion; two and three users may
have to take turns, and they may agree agree one strategy for all their
answers rather than constructing new ones. 

- Which exercise types do they say they like? Do they actually learn from
these exercises, or do they also learn from exercises they say they don't
like or find difficult? 


6. Answers and feedback

Does the mechanism for gap-filling work successfully (the word changes
colour on being in typed in; nothing happens till that point), or is it a
source of frustration? Should there be more feedback and help as answering
is in progress? Is the drop-and-drag mechanism
for the Ricomponi exercise easy to use?  


7. Help

Do they use the Help facility at all, or do they rely on their own skill
and exploration? If so, do they use it routinely, or only when confused?
Do they use the "programmed" bits of the help?
Do they expect more linguistic help, rather than just the video play
buttons? Which language do they choose to read the Help texts in, Italian
or English?


8. Language

The rubric of the software is in Italian, largely to ensure widespread use
of the software by learners of Italian regardless of their native
language. Only the Help texts are provided in English as well as Italian.
Do (English-speaking) users resent or appreciate this?

9. General Design

Is the design good or bad? What makes it so? Is the font used attractive
and legible, or not? Does it make students more or less inclined to work
with the software? Does it make learning easier?
Good or bad, is the design something which is forgotten about quite
quickly, or does it remin a hindrance and a distraction?

10. Outcomes

Do students who've used this software gain demonstrable passive and active
knowledge from it?

Do they gain more or less than students who've used other methods -
classroom teaching, books and paper, non-video CALL Exercises?

Do they gain most when it's used in conjunction with classroom teaching
and other activities?

Do they gain more working individually or in twos or threes?


Known bugs CD-2

In some exercises, video segments won't replay once all the answers
have been correctly typed. 

The colour of the subtitles is light blue in some dialogue exercises,
white in other exercises, and yellow in the unit review Menu. 

There are sometimes problems using CD-1 and CD-2 at the same time as, or
just after, each other. 

==