|
|
Case Study 3: Analyzing training program successScenario: CompanyB, which makes aluminum parts for the aerospace industry, has a well-regarded online training program in place, consisting primarily of a series of courses with assessment exams. The training staff now wishes to analyize the effectiveness of that training program, identify problem areas, and rework the materials based on those findings. Furthermore, it wishes to do so on an ongoing basis, as the needs and dynamics of the company change. Solution: The goal of an effective training program is not merely to create and display content. A training initiative is worthless if it does not accomplish the actual transfer of knowledge. Fundamental to that process is the measurement of what knowledge is, and is not, being successfully imparted.
Using the Cognitivity e-learning platform, learner performance throughout each course is recorded in detail. Each click of the user throughout the course is recorded, allowing the following basic measurements:
In CompanyB's case, there is particular interest in looking for problem areas in the courses that might be targeted by the next redesign of the materials. The training staff starts with a report showing the average score for each training course, and targets all courses with an average score less than 90%, starting with the lowest average score. Then, for that course they view a report showing the scores on all the quizzes and exams. Two quizzes, covering the properties and uses of two aluminum alloys, have usually low average scores. In one case, the missed questions are spread evenly throughout the quiz questions; in the other, two questions are missed more than 80% of the time. In the first case, a reevaluation of the chapter on 5052 grade Aluminum showed it to be a bit vague and rambling, and while the answers to the quiz are found in the text, they are obscure and somewhat off the main point of the chapter. It is apparent that even the students who pass the course often do so without knowing anything about 5052 Aluminum, and they have gained little from this unit. The first reaction was to rewrite the unit and the questions completely to bring it up to the standard of the rest of the course. However, interviews of line managers (to find an appropriate subject matter expert) revealed that 5052 Aluminum, previously used in foils and honeycomb structures, was no longer used in any of CompanyB's products. The lack of 5052 expertise did not seem to be a problem among the staff trained in the last year. Rather than rewrite the unit, it was decided to add the key points of the chapter to another unit giving an overview of other alloys not used at CompanyB. In the second case, two questions in the 6061-T6 chapter were missed by most of the students. In one question, students chose all of the answers about equally. A careful review showed that the material for that question was not actually covered anywhere in the chapter. The question was dropped and a new question developed. On the second question, most of the students selected the same incorrect answer. In this case, it turned out that factory practice was slightly different from the course author's preferred practice, and students remembered what they did day to day over what they had read. After some consultation with the factory floor, the text of the course was rewritten slightly, and the question was removed and replaced with one covering another topic in the chapter.More on this topic can be found in the section on Cognitivity's Reporting function. -- Copyright © 2002-2005 Cognitivity -- |
Once a company has addressed their most fundamental training question — what training is needed, and how it can best be accomplished — it must then monitor the effectiveness of that training on a continuing basis. In this case, because CompanyB has a training program consisting primarily of online courseware, such analysis is straightforward.