main contents | introduction | Pedagogy & design | Evaluation & use | screen shots | web links | references

Italia 2000 Video Software
Evaluation and Use
Dublin-Cambridge Research

Preliminary Evaluation

Informants

The evaluation was conducted by Cormac Ó Cuilleanáin in Trinity College Dublin (TCD). Ten advanced level students of Italian from TCD degree courses took part in this evaluation. Their reponses are given in full.

The order of the reponses given in each case is as follows. The information below is their initials, gender, and the length of time they have been studying Italian.

  1. EF (f); 3 yrs
  2. JC (f); 3 years
  3. SC (f); 3 years
  4. NK (f); 4 years
  5. ND (f); 4 yrs
  6. SG (m); 3rd year
  7. RH (f); 3 years
  8. IM (f); since around 3 years
  9. EW (f); 3 years
  10. CM (f); 4 yrs

Summary and Conclusions

The results were positive. The design of the software - navigation, VCR control buttons, use of subtitles - appears to be fully vindicated, as they find it easy to use and understand.

Two concerns emerged. The first was the Ricomponi exercise, which as expected proved too difficult and frustrating for some. The second, more unexpected, was the use of light grey text: several commented that it was difficult to read. While the equipment used might have had some bearing on this, it seems likely that the software for publication by Giunti should now use white text throughout.

Questions and Answers

Each question on the two-page form is given, with each of the ten answers, followed by a short comment and interpretation.

General

Which units did you use?

Da grande farò l'attore (CD-1)

How long did you spend on them?

  1. half hour
  2. half hour
  3. 20 minutes
  4. 20 minutes
  5. 25 minutes
  6. 20 minutes
  7. -
  8. 10 minutes
  9. 20 minutes
  10. 15 minutes

Range: 15-30 minutes
Average time: 21 minutes.

How easy was the software to use?

  1. relatively easy
  2. quite easy
  3. very easy
  4. easy
  5. very easy
  6. very easy
  7. easy to follow steps
  8. quite easy
  9. very
  10. easy

The software is easy to use.

How well would this sort of thing fit in with your learning needs? Did you learn anything definite from your use of Italia 2000?

  1. Useful for aural comprehension and vocabulary
  2. Perhaps if it was earlier on in my Italian learning
  3. -
  4. Speaking could be slower, subtitles are good.
  5. Very well, Yes
  6. Would fit in well
  7. It's a good way to learn phrases
  8. It helped me with the pronunciation, vocabulary, rapidity of comprehension
  9. I think it would benefit 1st year more
  10. No, general comprehension and using the language as a living language

Responses are positive. A few feel too advanced to benefit. Particularly encouraging are the reponses which mention comprehension/living language/benefit of subtitling, which is much of what the software is about.

The designer thought that the use of VCR symbols should help people familair with videos. Did you use the controls easily?

  1. yes
  2. yes
  3. yes
  4. -
  5. yes
  6. yes
  7. [tick]
  8. yes
  9. yes
  10. easy. yes.
VCR control buttons presented no problems to this group.

Navigation tools are meant to be self-evident. How easy did you find it to move forward or back along the trail of exercises? Did you use the menu button?

  1. Relatively easy, but there was nearly TOO much to choose from. Yes - did use Menu button
  2. Easy. Yes we did use the menu button.
  3. Easy to follow trail of exercises, used menu buttons.
  4. -
  5. Very easy, yes
  6. Very easy and yes
  7. [tick]
  8. Yes
  9. Very easy to move backward and forward. Yes.
  10. Easy. Yes.

Design of navigation buttons is fine; they undestand it fully.

Did you use the Help button? If so, did you find the advice useful?

  1. No
  2. No
  3. No I didn't use it.
  4. -
  5. -
  6. No
  7. No
  8. No
  9. No - didn't use it.
  10. No

The fact that no one used the help button, coupled with other positive answers on the design, strongly suggests that there was no need to use the help facility due to the intuitive nature of the design. Unfortunately, this means no feedback on the Help Facility is available.

How did you find the feel of the software? Challenging, testing, relaxing, or what?

  1. Challenging enough
  2. The last part was fairly testing, the rest was interesting
  3. Relaxing to use but testing at times
  4. Very modern
  5. All of the above
  6. -
  7. Good idea.
  8. [unclear; possibly "relaxing"]
  9. Test concentration more than anything else
  10. Testing and more fun than a grammar book

Largely positive. Suggestions that the level was not adavanced enough for some in the group.

The designer assumed that the interest of exploring the material will be motivating enough, so no point scoring and deucting system is used. Answers are provided in an undramatic way. Would you have preferred a more points-based approach?

  1. Yes - to compare scores each time
  2. Perhaps it would have provided more motivation.
  3. No, just more indication of whether I'm right or wrong.
  4. Yes
  5. Yes.
  6. Yes.
  7. No
  8. -
  9. No
  10. Not for this type of exercise

Inconclusive.

How did you react to the colour scheme and type style?

  1. Grey text a bit hard to read and text could be a bit bigger.
  2. The grey could sometimes be a bit hard to read.
  3. The grey colour scheme used in type style difficult to read.
  4. Words too grey - difficult to read
  5. Prefer it to bright colours, it was less annoying
  6. Type style fine, grey on blue is far too vague and indistinct
  7. -
  8. I found it attractive and clear to understand.
  9. -
  10. Good

Text in grey is clearly a problem, though comments hitherto have never suggested the problem is a serious as this. How good are the monitors being used? In any event, the Giunti CD for publication should probably remedy this, rendering text in white throughout, avoiding the problem altogether.

Details of Use

Did you replay the video clips all through?

  1. -
  2. yes
  3. yes
  4. yes
  5. no
  6. yes
  7. [tick]
  8. no
  9. no
  10. yes

Most, though by no means all, used the video review section at the start of each unit. It's hard to say how significant this is without knowing whether or not they were seeing the clip for the first time ever.

Did you replay short extracts from the video clips?

  1. yes
  2. yes
  3. yes
  4. yes
  5. yes
  6. yes
  7. [tick]
  8. yes
  9. yes
  10. yes

Anything other than 100% Yes to this question would have been disturbing, since the short extract play buttons are the chief design feature of the software.

Did you switch the subtitles on and off, or just leave them in one position?

  1. on and off
  2. on and off
  3. switched on and off
  4. on and off
  5. switched them on and off
  6. on and off
  7. Left them off
  8. Just left them in on position
  9. No
  10. Left them off

Most used the subtitling flexibly, which is encouraging. Others felt able to manage without them, which is fine.

Which exercises did you prefer: vocabulary, comprehension, text reassembley, dialogue? Were any of these too difficult or technically impossible?

  1. Comprehension and dialogue
  2. -
  3. Vocabulary, comprehension, no great difficulty
  4. Translation -
  5. Preferred vocabulary - re-assembly was too difficult and not very useful.
  6. Comprehension. Text re-assembly was no test, just a guessing game.
  7. Vocabulary. No.
  8. Comprehension / test re-assembly /
  9. Text re-assembly
  10. Vocab, dialogue, text reassembley could have had other answers

Some interesting reponses, some ambiguous. The Ricomponi clearly caused some concern: it could be too difficult in general, or particularly difficult in this instance (everyone did the same unit). Frustration and guessing isn't the intention at any time, though, so some modifications look necessary.

When you typed a word correctly, it changed colour. Was that enough, or would you have liked more advice, or more reaction?

  1. Perhaps alternatives could be given
  2. [unclear]
  3. I found the type colour difficult to see and so would prefer more of a reaction.
  4. Right word but wrong spelling rather than refuse the word blankly
  5. It was enough.
  6. A simple bleep would have helped.
  7. That was fine.
  8. Sometimes there was a small noise, it can be better.
  9. No that's enough
  10. Fine

Opinion is mixed; several like the understated right/wrong feedback, others wonder sensibly about other types of feedback. Would the answers be any different after several other sessions, rather than straight after the first?

Could you follow the reactions in Italian easily enough? Or should more English have been used?

  1. Yes.
  2. Italian instructions were easy enough.
  3. Instructions easy to use.
  4. Italian enough.
  5. Italian was easy enough to understand
  6. Italian was fine
  7. It should be in Italian or we'll never learn it.
  8. Yes except in the beginning, the exercise on text reassembly
  9. The italian instructions were easy enough. I don't think any English should be used if it's possible.
  10. Followed easily.

Largely supportive of the monolingual approach, comment 7 making the point pertinently.

Other Impressions

What would you have liked more of?

  1. Grammar
  2. -
  3. -
  4. -
  5. Synonyms, filling in spaces
  6. The size of the page and images would benefit from an increse in size
  7. More unusual phrases
  8. -
  9. -
  10. -

Comment 6 is a fair one, but allowing for all kinds of equipment to be used means that a larger page size could not be displayed on some equipment (the current size does not fit on some machine with only Large Fonts).

What would you have liked less of?

  1. Text reassembly
  2. -
  3. -
  4. -
  5. Reassembly
  6. Less blue background!
  7. -
  8. -
  9. -
  10. -

Text re-assembly (Ricomponi) is a small but significant problem.

What would you have liked done differently?

  1. Text could be bigger and clearer
  2. -
  3. -
  4. -
  5. -
  6. -
  7. -
  8. -
  9. -
  10. -

Comment 1 also made elsewhere; hopefully the absence of other comments indicates general satisfaction?

If you were able to get any language material of your choice on CD-ROM, what would you choose?

  1. Grammar constructions
  2. -
  3. Translation exercises from both Italian to English or English to Italian
  4. Translating package
  5. Translation and grammar exercises
  6. Translations of literary and print media text
  7. Don't know
  8. -
  9. -
  10. For my personal needs - grammar exercises

Is there any other comment you'd like to make?

  1. -
  2. -
  3. -
  4. Modern and advanced. With more time it would be of great benefit.
  5. -
  6. -
  7. I think Italian video is a good way of improving vocab etc it's also good for comprehension.
  8. I would be interested in doing this type of exercise more often.
  9. The video seemed to stall on occasion, brekaing the speech and therefore making it slightly more difficult to grasp on first viewing.
  10. The video froze occasionally which was fine from the point of view of the picture. But freezing the sound lost my concentration.

Maybe there was some local technical/network problem (the software was running on a network basis)? Other comments were positive.

Back to Use and Evaluation

Top of this page

main contents | introduction | Pedagogy & design | Evaluation & use | screen shots | web links | references