The Institute of TAFE Tasmania
by Marcus Ragus
Summary
The Institute of TAFE Tasmania completed a series of practical trials in the use of the QTI m-Player™ for workplace assessment. Individual learning practitioners/assessors were then given the opportunity to evaluate and comment on the usefulness and practicality of the player for field assessment that is tied to a particular context. Learning practitioners from four delivery programs participated in the field trials. The results of the evaluations are outlined below, together with individual overviews from five trainers-assessors who took part
Introduction
The Institute of TAFE Tasmania delivers products and services through 18 delivery programs operating from more than 20 training sites across the State. TAFE Tasmania encourages a strong flexible delivery culture in its teams and business operations, with training delivered in a variety of contexts including: the home; the workplace; at the State’s online access centres through computer-managed learning; and face-to-face at TAFE campuses.
TAFE Tasmania is the largest registered training organisation in the State and has recorded the most enrolments in a training package and the most training hours delivered in a year under a training package across the country. The organisation has been nationally recognised for its achievements, including twice receiving the Australian National Training Authority’s National Training Provider of the Year award - in 2000 and 2002.
In 2001, TAFE Tasmania delivered more than 4.5 million hours of government-funded training. This rose to 4.64 million hours in 2002 and 4.9 million hours in 2003. Quality reviews and analysis have shown the rise can be directly attributed to increased efficiency in delivery practices.
The institute has developed close partnerships with local enterprise and community clients and recognises there is an increased need for innovative flexible delivery strategies, including the use of new and emerging technologies to deliver learning in the workplace.
The principal bodies within the institute involved in this project were Learning Media Services and the natural resources program area.
The assessment challenge
Assessment tools
A range of assessment tools were used for the trials. These were selected by the participating practitioners and based on actual scheduled assessments and their delivery requirements (see Sections 4.7.3 and 4.7.4 for two examples of the 14 tools used). In all cases, pre-existing assessment documents were used and were converted using a first-generation QTI Creator™ into an accessible interactive format for the mobile devices. The original assessment documents were in formats including Microsoft® Word (.doc), Microsoft® Excel® (.xls) and Adobe® (.pdf) file types. Each format offered its own degree of conversion issues, including problems with the display of Lickert scales in some assessments. This was later rectified.
Each practitioner was asked to provide for conversion a maximum of two assessment documents that they felt they could use in practical application during the trial period. These tools have been categorised informally into two groups:
- practitioner-facilitated, such as observation checklists, witness testimonials and student contact records
- learner-facilitated multiple-choice and short-answer assessments.
Examples of assessment document.
Equipment
The trials used two models of Hewlett Packard® iPAQ Pocket PC®: the recently discontinued HP® iPAQ 2210™ and the new HP® iPAQ 2410™. Some initial difficulties were encountered when the QTI m-Player™ was loaded onto the older HP® iPAQ model, primarily due to the slight difference between the versions in operating system. While the QTI m-Player™ was originally designed for more recent operating systems, an update to the QTI m-Player™ soon resolved this issue.
In all cases, the QTI assessment documentation presented well on both models and there were no apparent differences between the models in performance of the QTI m-Player™ or its ability to play documentation.
The trials
Trials were conducted in four Institute of TAFE delivery areas: carpentry and joinery; community services; information technology; and ecotourism (Drysdale Institute of TAFE). Fifteen staff participated in the pre-trial workshops and presentations, which included a project-introduction session, technical and equipment presentations and introduction to handheld PC workshops. The handheld PC workshops were incorporated into the participants’ professional development programs, ran for three to four hours and covered material ranging from a basic introduction to hand-held devices through to navigation and file management. In total three workshops were delivered and feedback from participants indicated that they were very effective.
Apart from members of one of the teams (community services) who provided their own handheld PCs, all participants in the trial were lent a handheld PC for at least two-weeks. The loan duration varied depending on individual need, particular trial period and handheld PC availability.
Evaluation
Participants were able to provide feedback by way of the project evaluation forms (see Sections 4.7.I and 4.7.2), informal discussions throughout the trial process and formal recorded interviews after the trials.
Although feedback from the participants indicated that they were very satisfied with the introductory sessions and workshops, they all wanted further professional development in the area of mobile assessment. Apart from one individual, all participants could see that this technology would improve their work processes and increase efficiency in administration. All could see significant benefits in using a mobile device for assessment and record keeping in the field and that this mobile assessment approach was worth pursuing. The majority currently felt overloaded with existing hard-copy paperwork when on workplace visits and saw that the functionality and portability of hand-held devices would be a significant advantage to them in this regard.
Many felt that the trial period was too short and that they would have liked more time to assess the full benefits of the QTI m-Player™. This was particularly so in the case of a couple of the teams which felt that, if given another opportunity, they would have trialled other types of assessment documentation different in structure and format to that which they had chosen.
As previously stated, the QTI assessments were converted by the project team from original assessment documents currently used in hard copy by the practitioners in the field. Most of the trial participants felt that, having gained an understanding of the QTI m-Player™ system, they would now prefer to develop and design new assessments specifically for the mobile format. They felt this would provide them with higher quality and more functional tools. It would also be an opportunity to revise, update and change their existing assessment administration and process.
Some participants had concerns over the likelihood of uptake of this type of system by their organisations, stating that:
- There was a lack of understanding of the full potential of hand-held computing devices, with many in their organisations thinking of them as no more than a ‘glorified diary’.
- There was an overall obsession with hard-copy paper trails among organisational system administrators and auditors.
- There were issues resulting from auditors’ failure to recognise a diverse range of electronic evidence.
- There was a lack of commitment to professional development in these specific areas of emerging technology and what was available served only as an introduction.
- While lap-top computers were currently being rolled out to staff by organisations, they felt hand-held devices provided them with greater flexibility than a lap-top when on visits, as they were ‘available’ at ‘a moment’s notice’ - which was very desirable when in the field.
The trial interviews
Detailed below is a selection of five recorded interviews from trial participants.
Community Services
Trainer, coach, assessor
Recordings:
Barbara Holgate - Introduction
Barbara Holgate - What are the current difficulties that affect assessment in the field?
Barbara Holgate - In what ways can mobile assessment help you?
Barbara Holgate - Is the mobile assessment approach you have trialled worth pursuing?
Barbara Holgate - An additional comment from Barbara
View a transcript of the interview with Barbara Holgate
Community Services
Trainer, coach, assessor
Recordings:
Phillip Harrison - Introduction
Phillip Harrison - In what ways can mobile assessment help you?
Phillip Harrison - Is the mobile assessment approach you have trialled worth pursuing?
View a transcript of the interview with Phillip Harrison
Carpentry and Joinery
Trainer, coach, assessor
Recordings:
Glenn Robinson - What are the current difficulties that affect assessment in the field?
Glenn Robinson - In what ways can mobile assessment help you?
Glenn Robinson - Is the mobile assessment approach you have trialled worth pursuing?
Glenn Robinson - Is the mobile assessment approach you have trialled worth pursuing? No 2
Glenn Robinson - To best help you what should we do next to progress mobile learning and assessment?
View a transcript of the interview with Glenn Robinson
Information Technology
Trainer, coach, assessor
Recordings:
Anne Pennington - What are the current difficulties that affect assessment in the field?
Anne Pennington - In what ways can mobile assessment help you?
Anne Pennington - Is the mobile assessment approach you have trialled worth pursuing?
View a transcript of the interview with Anne Pennington
Eco Tourism
Trainer, coach, assessor
Recordings:
Marcus Morse - What are the current difficulties that affect assessment in the field?
Marcus Morse - In what ways can mobile assessment help you?
Marcus Morse - Is the mobile assessment approach you have trialled worth pursuing?
Marcus Morse - To best help you what should we do next to progress mobile learning and assessment?
View a transcript of the interview with Marcus Morse
Future directions
This project has demonstrated the value of mobile technology as a tool to increase efficiency in conducting assessment in the workplace. The success of the QTI m-Player™ in a range of practical scenarios during the field trials verifies that it has the potential to deliver as originally theorised. This now opens up the opportunity for further development.
Throughout the trial there was a strong desire (almost impatience) in all participants to go beyond the current parameters of the project. It was at times difficult to keep them thinking only of the objectives of the trial, which were primarily to see whether:
- mobile assessment is a useful practice for teachers/trainers/assessors and students
- mobile assessment can be improved through QTI compliance.
Participants were very forthcoming with suggestions as to what they would like to see in future QTI products. The main suggestions are outlined below for consideration regarding possible inclusion in any future development:
- Signature blocks for electronic signature validation within documents. This was an element considered early in this project but not incorporated into the development plan due to cost and time constraints.
- Greater internal navigation capability within QTI documents to enable movement from one page/area to any other within one click. This is important where assessments are completed in stages over a period and where repeated use of part of an assessment document is needed. Presently the QTI m-Player™ allows movement only one page at a time, either immediately forwards or backwards.
- The capability to incorporate images and voice files into QTI assessment documentation.
- The capability to view (in print layout) and print completed assessment documents from both a mobile device and a syncronised desktop PC.
- A fully-developed overall product, including a user-friendly QTI Creator™ for the conversion and development of assessment documents and a back end. This may or may not include a student record database, learning management system or similar.
- Trialling of any updated products to be based on a complete package (see above point) and over a longer period, with the process also taking into consideration the professional development of participants and their need to be sufficiently up-skilled to manage the complete system..
TAFE Tasmania ‘Future Thoughts’ on the QTI m-Player™
Prepared by Peter Higgs, Director of Learning Services, Institute of TAFE Tasmania
From an organisational perspective, the QTI m-Player™ offers TAFE Tasmania the potential to engage to an even greater degree with its existing enterprise partners. More importantly, the QTI m-Player™ provides the potential for TAFE Tasmania to establish an even greater number of enterprise and industry partnerships on both a national and state basis.
The QTI m-Player™ offers the potential for the organisation and its partners, existing and new, to deliver secure assessment items on mobile devices in the workplace. Assessment results could then be uploaded to our student management information system to ensure accuracy and security of student data. This has never been achievable when using mobile devices before the outcome of this 2005 New Practices project.
Depending on the human resource management information system (HRMIS) used by partner enterprises and industries, it will also be possible to upload the same student/client assessment results to their systems - again ensuring accuracy and security of both student and organisations’ data. The outcomes of the project afford TAFE Tasmania an opportunity to establish itself as a national leader in secure mobile assessor services for enterprise and industry. This will require strategic commitment from TAFE Tasmania to achieve the full current and future potential of the QTI m-Player.
The 2005 QTI m-Player™ New Practices project has been a great success, meeting its predicted outcomes for both the New Practices initiative and TAFE Tasmania. However, this is only a first stage in achieving a QTI mobile solution. There will need to be continuing funding support to embed and further develop QTI solutions for TAFE Tasmania and the greater VET sector in Australia.
Attachments
Attachment 1 - Use Satisfaction Survey QTI m-Player™ and mobile assessment
Attachment 2 - Field test interview question: QTI m-Player™ Project
Attachment 3 - Sample assessment tool – workplace contact record
Attachment 4 - Sample assessment tool – ICT element