The teaching of quality: Convergent participation for the professional development of learning object designers

Cette etude porte sur la perception de l'efﬁcacite d'une methode d'evaluation collaborative d'objets d'apprentissage pour enseigner des criteres de qualite de ces objets dans le contexte d'un cours a distance a l'Universite Athabasca en Alberta, Canada. Vingt-quatre etudiants ont participe, en equipes de quatre, a deux conferences telephoniques, au cours desquelles ils ont applique un modele de participation convergente. A la suite d'une premiere conference portant sur les neuf criteres de l'instrument utilise pour l'evaluation des objets d'apprentissage (LORI), chaque participant a effectue une evaluation individuelle d'une selection d'objets. Lors d'une deuxieme conference, ils ont partage leurs evaluations et commentaires. Six a neuf mois plus tard, les participants ont repondu a un questionnaire visant a evaluer notamment leur comprehension des objets d'apprentissage et leur habilete a les evaluer. Les perceptions des etudiants a cet egard s'averent positives. Les repondants ont egalement note un impact appreciable de cette experience sur leur pratique professionnelle. Leurs commentaires indiquent que cette experience leur a permis de mieux apprecier la complexite du processus de conception des objets d'apprentissage et les a outilles d'une methode adequate pour en evaluer leur qualite.


Abstract
This study examined the perceived effectiveness of a collaborative evaluation process for teaching qualitative aspects of learning object design in a distance education course at Athabasca University in Alberta, Canada. Working in groups of four, 24 distance students in a graduate level instructional design course participated in two 2-hour audio conferences, using a convergent participation model for the evaluation of learning objects. After the first conference, which featured study and application of the nine criteria of the Learning Object Review Instrument (LORI), the participants independently rated a set of learning objects. In the second audio conference they reconvened to compare and discuss their ratings. Six to nine months later the participants reported favorable outcomes from the experience in their understanding of learning objects, and confidence in determining learning object quality. In judging the impact on their subsequent design efforts, the most frequent comments indicated that the convergent participation experience provided both an appreciation of the complexity of learning object design and a method for judging their quality.

Introduction
Over the past few years learning objects have gained increasing prominence as a paradigm for the development of instructional materials. Defying an absolute definition, the term "learning object" has been interpreted by various authors (Wiley, 2002;Koper, 2001;Richards, 2002) according to their particular instructional or organizational context. In general, learning objects are considered to be the digital files that contain the text, images or interactive media for use in instruction or Web-based learning. The primary potential of learning objects is their ability to be developed in one context and then re-used in others, thus reducing the cost of Web-based instruction while increasing the amount and variety of available instructional materials. Much of the discourse on learning objects concerns technical methods for their storage, retrieval and re-use in new instructional contexts. The work presented here, however, is concerned with quality standards in learning object design and development.
Approaches to judging or filtering learning object quality include academic peer review models (Howard-Rose & Harrigan, 2004;MERLOT, 2004;Nesbit, Belfer & Vargo, 2002), models in which content experts recommend resources (as 57 in the Resource Description Network at www.rdn.ac.uk) or recommender systems that statistically match a user' s preferences with others' ratings (Recker, Walker, & Lawless, 2003). The authors believe that evaluation instruments and tools have important implications for design practices, and that they can be intentionally deployed to instill quality values in novice learning object designers. Although evaluation instruments and tools may not be designed for teaching, they can sustain authentic learning activities that emulate critical elements of professional practice. The corollary of this valuing activity is that students who have learned through such activities are aware of their changed values and are often able to articulate them. To investigate these hypotheses, this paper presents and analyses student instructional designers' reports of their experiences with a Web-based review instrument and collaboration evaluation process.

Evaluation Tools for Professional Development of e-Learning Designers
This study focuses on the use of two tools that had been developed in previous learning object repository projects: (1) an instrument for evaluating object quality, and (2) a community Web site for gathering, aggregating, presenting and managing the evaluations or reviews.

Learning Object Review Instrument
The Learning Object Review Instrument (LORI) was developed to capture, in a concise format suitable for collaborative evaluation, several dimensions of quality recognized in the theory, research and practice of instructional design . Version 1.5 of LORI (Nesbit, Belfer, & Leacock, 2003) consists of the nine items identified in Table 1. It assembles standards from a wide range of sources relevant to learning object quality. For example, the accessibility item integrates the W3C guidelines for Web accessibility (World Wide Web Consortium, 1999) with the IMS guidelines for developing accessible learning applications (IMS, 2002). Of course, in most areas of learning design such as motivation and presentation design, there are no internationally recognized standards but rather a body of theories, research results and practices that are quite diverse and sometimes contradictory. Instead of attempting to tightly define quality standards in these areas, each item in LORI presents a set of attributes that broadly characterize principles extracted from available literature and asks evaluators to rate an object on a 5-point scale (from low to high). Evaluators also enter comments explaining or qualifying their ratings. Unlike traditional assessment instruments that obtain reliable measures by establishing several items for each quality being assessed, LORI is designed to obtain reliability by having multiple raters negotiate quality assessment. In this respect LORI operates more like a framework for negotiation of quality, rather than a traditional evaluation instrument.

Convergent Participation via the eLera Web Site
Convergent participation is a collaborative process in which raters negotiate an evaluation on several dimensions of quality (Nesbit, Belfer, & Vargo, 2002). After an initial individual evaluation phase, raters meet to compare and discuss their assessments. A moderator selects and prioritizes the items (quality dimensions) so that those showing the greatest disagreement are discussed first. The evaluators may alter their individual ratings and comments as the meeting proceeds. However, the evaluators are under no compulsion to resolve their differences and attain consensus. At

Content Quality
Veracity, accuracy, balanced presentation of ideas, and appropriate level of detail Learning Goal Alignment Alignment among learning goals, activities, assessments, and learner characteristics

Feedback and Adaptation
Adaptive content or feedback driven by differential learner input or learner modeling

Motivation
Ability to motivate and interest an identified population of learners

Presentation Design
Design of visual and auditory information for enhanced learning and efficient mental processing

Interaction Usability
Ease of navigation, predictability of the user interface, and quality of the interface help features

Accessibility
Design of controls and presentation formats to accommodate disabled and mobile learners

Reusability
Ability to use in varying learning contexts and with learners from differing backgrounds

Standards Compliance
Adherence to international standards and specifications the end of session the moderator combines all rating and comments into an integrated review that represents the evaluators' final positions on each quality dimension.
An important question concerning the efficacy of the convergent participation model is whether evaluators converge in their assessments. If they do not converge, then the collaborative phase does not increase assessment reliability. A lack of convergence may indicate that the evaluators were not changed by the experience, and that the convergent participation model is ineffective as a method for building common community standards. Using LORI and standard inter-rater reliability statistics, Vargo, Nesbit, Belfer and Archambault (2003) found that a collaborative evaluation of five learning objects resulted in convergence on almost every dimension.
LORI and tools supporting the convergent participation model have been implemented in the eLera Web site, www.elera.net (Li, Nesbit, & Richards, in press;Nesbit, Leacock, Xin, & Richards, 2004). Localized in French, English and Chinese, eLera enables moderators to invite registered members to evaluate a set of learning objects, prioritize items for discussion and publish integrated reviews. Over the past two years, several workshops and field tests of eLera have taken place with secondary teachers, and post-secondary instructors and instructional designers (Leacock, ).

MDE 604: The Instructional Development Context
MDE 604 Instructional Development and Program Evaluation is a core course in Athabasca University' s Master of Distance Education Program (in Alberta, Canada). Courses in the program are taught in a group-paced, Web-supported mode. Learners progress through predefined learning activities -readings, assignments and online asynchronous peer conferences -under the guidance of a professor who sets the overall tone of the course, responds to learner queries, and marks assignments. In 2003, learning objects were added to the panoply of instructional design approaches taught in MDE 604. Collaborative evaluation of learning objects with eLera became an optional activity in 2004.
In teaching MDE 604 for the past four years, Richards noted that although an increasing number of students chose to produce Web-based instructional projects, they had difficulty in understanding the concept of reusable learning objects and applying it to their instructional development projects. Learning objects were still new, the concept was ill-defined, and there were few good examples to follow. Without establishing a better understanding of the design constraints for learning objects, students would have continued to produce Web-based projects as if they were designing for print.
The investigation reported here was designed to assess the role of collaborative evaluation of learning objects in the professional development of learning designers. It was hypothesized that (a) collaborative evaluation would offer constructive engagement with learning object design concepts, and (b) the activity of rating learning objects would be retroactively perceived by students as building knowledge and values relevant to professional practice. In this study, students' perceptions of the value of convergent participation for professional development were examined in two sequential course offerings.

Method
The eLera activity was offered by synchronous telephone conference. The participants, who volunteered as part of an optional assignment, were drawn f r o m t w o s e q u e n t i a l o f fe r i n g s o f MDE 604. The first group ("winter") consisted of 12 participants and the second group ("spring") consisted of

Map of Human Heart
Shows an animation of heart function. http://www.pbs.org/wgbh/nova/heart/heartmap.html 12 participants. The participants were all enrolled in the Master of Distance Education Program. They are male and female adult learners, many of whom are employed in education or training and for whom successful completion of the program may lead to career advancement. The average age of students enrolled in the program is about 42 years. All participants in the study were residents of Ontario, Quebec, Alberta or British Columbia.
The telephone charges for the entire project totaled $210 (CAD). Participants were required to obtain access to telephones and on-line computers. In two cases, where telephone modems were used, it was not possible to have both the computer and telephone on-line at the same time.
A set of five learning objects was selected, representing a variety of learning object types and different levels of quality. Table 2 lists the learning objects used.
To facilitate synchronous small group discussions, participants selected sessions that best suited their schedules. Because participants geographically straddled three time zones, the time of 5:00 P.M. -7:00 P.M. Pacific Time was selected because it was neither too early for British Columbians nor too late for participants in Quebec. Each group of four participants met twice -usually a Tuesday evening followed two days later on the Thursday. This provided a two-hour training session, time for the participants to individually review the objects, and a twohour meeting for the peer group to discuss and alter their ratings. Participants' reports were individually completed and e-mailed to the researchers within a week after the second teleconference.
As part of their optional assignment, participants submitted a two page written reflection on their eLera experience. They were asked to comment on their "likes, dislikes, and suggestions for change." The winter group participated in the eLera exercise after finishing their instructional design projects. Because many in the winter group commented that the activity would be more valuable if it was sequenced before the instructional design project, the spring group was scheduled to participate in eLera before producing their instructional projects. To gauge long term impact and the application of knowledge acquired through the activity, a follow-up questionnaire (Table 3) was administered six months after the completion of the winter semester.

Results
Results of the study are presented as summaries of participant reflections and follow-up questionnaire responses.

Summary of Participant Reflections
Information identifying the participants was removed from the reflection data. Comments in the reflection data were segmented, thematically categorized, and coded as pro (positive) or con (negative) by a research assistant using the ATLAS. ti program (available from http://www.atlasti.com/index.shtml). Comments clustered into ten themes: 1. communication medium (telephone conference with interactive Web site) 2. usability of the eLera site 3. understanding of LORI and convergent participation 4. understanding of the learning objects paradigm 5. fit of the eLera activity with the MDE 604 course 6. timing and sequence of the activity in the course 7. understanding of issues and criteria for quality 8. value of convergent participation 9. confidence in ability to do a LORI evaluation 10. implications for professional practice As was noted above, initial results from the winter group overwhelmingly suggested that the activity would be more effective if sequenced in advance of the instructional design project. Due to this change, the following results focus on comments from the spring group. Note that because the responses were parsed into independently categorized comments, zero or more comments could come from any one participant, and some participants could contribute both positive and negative comments. In the following list, the total number of respondents (n), the number having pro, con, both statements or no response (nr) is shown for each comment cluster. 1. Communication medium (n= 10, pro 7, both 1, con 2, nr= 2) Most of the positive reports commented that the telephone conferences provided a new sense of engagement and community that had been missing in the current printbased delivery model supported with Web conference courses. The negative comments came from one participant who had difficulties scheduling the call, and from one who had technical difficulties with the computer conferencing system. "…an excellent format. The activity could be done with MSN messenger… but phone provides a unique personal contact with fellow participants. It was nice to hear peoples' voices." "It gave an immediacy and presence to the course." 2. Usability of the eLera site (n= 4, pro 1, both 2, con 1, nr= 8) The positive comments related to ease of use, lay out and the ability to have it open in one Web window while viewing the object in another. Macintosh computer users reported frustration with pop-up boxes that would get hidden under other screens. One commented on the difficulty of the Dewey classification system to match real-world information categories (a finding also reported by Leacock, Richards and Nesbit, 2004).
3. Understanding of LORI and convergent participation (n= 4, pro 2, both 0, con 2, nr= 8) Respondents commented that they initially thought the LORI model complex, however they understood the analogy to a jury. Some of the criteria, especially standards compliance, accessibility, and reusability were new to the learners. They expressed concern that they had initially little knowledge of what to look for in these areas, and the rubrics did not tell how to weight deficiencies. Participants commented that the rating activity gave them an opportunity to see a wide range of learning object faults and realize that minor annoyances could be rated less severely than serious content issues or technical issues that caused computer crashes. Other participants found the assessment rubric easy to follow. One participant had confused the more specific SCORM compliance issues with the more generic LORI ratings "I had assumed that a non-SCORM compliant learning object was a waste of time, however none of the objects viewed could be considered SCORMcompliant yet they exist and provide a benefit to the user." Another commented on the breadth of the learning object concept: "The Web sites evaluated were vastly different, not only in type and breadth of content, but also in technological presentation and in purpose." Others commented on the potential complexity of learning objects and appreciated the amount of awareness achieved in the activity. 5. Fit of the eLera activity with the MDE 604 course (n= 6, pro 6, both 0, con 0, nr= 6) The comments were unanimous in the value of the activity for the instructional design course, and many suggestions for improvement were offered. "The analytical activity and discussion has revealed much regarding the creation of quality learning objects [...] It lays a good foundation on which to proceed with other course work." "There is no question, this is a perfect exercise for students in this course, and the timing is great as well." 6. Timing and sequence of the activity in the course (n=9 , pro 5, both 0, con 4, nr= 3) Complaints about timing referred to the short time allowed between the first and second telephone conferences. With other activities and poor connectivity, the ratings could not be easily completed in two days. Suggestions ranged from three days to a whole week between the two conferences. The positive comments reflected on how useful it was to have the activity before the instructional design assignment. "…this activity happens at the right time in the course." One participant suggested the activity be repeated, rating the completed instructional design assignments at the end of the course. 7. Understanding of issues and criteria for quality (n= 2, pro 1, both 0, con 1, nr= 6) Only two participants commented on this theme. One negative comment reflected the degree of confusion over the weight to assign deficiencies. The other comment elaborated how these same confusions revealed the subjectivity inherent in human judgment.
"The experience revealed the subjective and personal nature of evaluation even with a strong guiding tool and process." "The Convergent Participation Model does help to overcome these dependencies, and provides a stronger evaluation then [sic] a single personal evaluation." 8. Value of convergent participation (n= 6, pro 6, both 0, con 0, nr= 6) All comments were positive, and noted not only the value of the discussion for learning objects, but also for the potential extension of this technique to other evaluation settings in the workplace. One particular strength noted was the ability to have meaningful discussion without the need to force consensus on the participants. "I have used inter-rater reliability previously in hospital reengineering, redesign, and fiscal recovery operations and know how effective it is in evaluating criteria. The opportunity to debate points of view is enlightening, and forces critical reflection of ' sacred cows' and ' out of the box' thinking." 9. Confidence in ability to do a LORI evaluation (n=6, pro 1, both 1, con 4, nr= 6) The negative comments mostly stemmed from a desire to have had more practice with the system before rating objects by themselves. Some concern was given for the amount of time a LORI assessment might take for larger objects, and for anonymity of the reviews. Just as a learning object might mature through formative evaluation and revisions, reviewers noted they too would change their criteria with experience and exposure to a variety of objects. One participant was concerned that object creators might be discouraged if reviewers were critical of objects reviewed out of their original context. "I was worried about what the author of the learning object would think when they read my review. I was very critical of some of the objects because they didn't work for me." 10. Implications for professional practice (n= 6, pro 2, both 2, con 2, nr= 6) One participant was very taken with the notion of engineering re-usability into learning objects. Other positive comments noted that it would make a great professional development activity for their educator colleagues. The contrary positions worried about the amount of time reviews take, and the need to allocate human resources. Another noted the paucity of objects in their particular field of study.

Results from Follow-up Questionnaire -Impacts on Practice
To get a longer-term perspective on the impact of eLera, participants were sent a brief set of questions by email (six months after the end of the spring session, and nine months after the end of the winter session). A total of 14 participants replied (8 spring, 6 winter). Their responses are summarized in Table 4. While not all participants had designed learning objects since the end of the course, the 12 that had (8 spring, 4 winter) said the eLera experience had very much or somewhat influenced their design practice. Most participants said that the eLera activity had also increased their expectations of quality -for themselves, for developers working under contract to them, and in learning objects that might be adopted from other sources. As one participant expressed it, "LORI provided a critical framework to evaluate their own work".
Many of the participants also shared their knowledge of learning object quality in discussions with their colleagues and supervisors. Participants in the spring group, who took the LORI training before their design projects seemed more likely to share their knowledge of learning object quality with others.
Participants were very supportive of the activity and were unanimous in rating the telephone sessions as very good or good. In terms of the self-assessment items, all but one participant rated as very good or good 5b) their own level of understanding of LORI, 5c) their understanding of quality issues, and 5d) their confidence in their ability to review learning objects.
The negative comments reflected that the eLera site has yet to grow a collection of learning objects in content areas of interest to a particular individual. Participants volunteered comments that the LORI training "should be mandatory", "should be an entire course", "should be the first thing taught in the course", and that it was "very useful for the rest of the course." One respondent' s comment reflects the overall value of this activity for learners in the MDE 604 course.
"I just remember thinking that getting to be involved in a hands-on/interactive activity it was a great addition to the course. I have kept all of the resources connected to the activity and have referred to some of the Web sites more than once."

Discussion
This study was initiated to answer fundamental questions about collaborative evaluation of learning objects and its potential role in the professional development of learning designers. It investigated the attitudes of graduate students enrolled in a master of distance education program towards an activity in which they reviewed learning objects using a learning object review instrument (LORI) on a Web site (eLera). Collaboration was structured according to the convergent participation model. Referring back to the questions that motivated this research: (a) Participants found the eLera activity engaging and helpful in understanding the learning object paradigm. They believed that it deepened their knowledge of learning object evaluation criteria and concepts of learning object quality. (b) Those participants who went on to design learning objects after the eLera activity felt they were able to produce better learning object projects.
Although unable to determine that the participants actually produced better learning objects, the instructor observed that they were more cognizant of quality issues than previous students taking the course. The results of the study suggest that participants were able to use the quality concepts they learned in the eLera activity in their professional practice.
We found that some of the LORI criteria require additional explanation. The need to more clearly differentiate between presentation design (appearance), inter-action usability (navigation), and feedback and adaptation was noted by some participants. Accessibility was a difficult quality dimension, for which many participants indicated "not assessed". We believe that current research efforts to elucidate accessibility specifications (e.g., Barstow and Rothberg, 2002) will make it easier to rate objects on this criterion and further educate learning object developers on its importance.
The authors have previous experience moderating collaborative learning object evaluation sessions using different communication media, including synchronous text chat, and face-to-face discussion in computer labs. These sessions are best characterized as using blended media because in every case the participants interacted with the eLera Web site as they edited LORI reviews and viewed the reviews of others. Although research data was not collected in all these instances, the authors' perception is that regardless of the synchronous communication medium used, the activity elicited favorable responses and quality ratings tended to converge. The method appears to function effectively across a variety of synchronous media when blended with the Web-based tools provided on the eLera Web site.
Although computer-based tools providing voice-over Internet are improving, software distribution, installation and configuration issues often involve end-user fiddling or require costly technical support. With long distance telephone rates now as low as $0.04 per participant minute and a very high quality of signal, it was an ideal choice for this study. While voice-over Internet technology is expected to expand and further reduce communication expenses, currently, more students have access to telephone than broadband Internet connections. It may soon be feasible to bring raters together with a blend of conferencing technologies.
Perhaps of greater concern for distance educators is the scheduling of asynchronous distance learners into synchronous discussion sessions. Small group size, advanced scheduling, and clustering participants within time zones are recommended to ensure flexibility. In subsequent MDE 604 sessions, the initial training session was replaced with a self-paced training document, and only the final audio-conference was retained. Despite the potential inconvenience of a synchronous event, for many learners this was justified by the efficiency of voice communication for this task. For many, a feeling of connection with classmates was a bonus. eLera continues to be under active development. While the training materials are being revised for the MDE 604 context, additional effort is being placed into enhancing the classification system so school teachers might more accurately meta-tag objects used in their teaching. Although subject-matter expert and instructional design reviews provide valuable information, it is also important to encourage the submission of learning object reviews by instructors who have used learning objects in actual instructional settings. LORI, the convergent participation process, and the Web-based tools on the eLera Web site were designed as means to develop community standards for learning object quality. They explicitly represent beliefs, show when a member changes a belief about quality, and tend to obtain convergence in beliefs (Vargo et al., 2003). The present study demonstrates that these tools and procedures are valued by graduate students as means for building their knowledge of quality standards for learning object design. All of the participants were appreciative of their new knowledge, and many claimed they were able to transfer their critical awareness to other tasks in their professional lives. Quality is every designer' s concern. Acquir-ing a framework for its analysis and being open to discuss quality perceptions with peers are two instructional outcomes that advance instructional designers on their personal journeys of excellence.