Course Outcome Assessment: Is Using the Average Good Enough?

Abstract

Assessment of environmental engineering course outcomes is critical for measuring student achievement, evaluating course design, and ultimately assessing programs for ABET accreditation. Often such assessment relies upon the interpretation of results from direct, embedded indicators (graded events such as homework assignments or exams). The simplest approach uses the arithmetic mean of a particular graded event (or set of graded events) to assess the outcome: for example, an average of greater than 80% indicates successful achievement of the outcome. However, such an approach assumes a reasonably normal distribution of student grades, and thus may not be adequately descriptive in some cases. For example, one case where the use of the mean fails is a bimodal distribution of results: if half the students score 100% and half score 60% on a graded event, the resulting mean, 80% (which is generally considered successful) fails to capture the fact that half the students failed. Stoker, Blair, and Sobiesk (2014) proposed a “binning” approach, which accounts for the proportions of students who achieve different levels of success on each assignment; for example, if greater than 5% fail, we assess the outcome to be not successfully achieved, regardless of the overall average. Such an approach, however, is more complicated and time-intensive for faculty to employ. This study investigates the impact of course size (as measured by number of students) on assessment results. Specifically, we present a comparison of course outcome assessment results using both the simpler “mean” and more complicated “binning” approaches for three large (more than 160 students) and two small (less than 35 students) environmental engineering and science courses. The results indicate that the small courses had a greater drop in outcome achievement than the large ones when using the “binning” vs the “mean” approach; however, the "binning" approach could still be useful in these courses with minor modification. In addition, we present some adjustments to the method proposed by Stoker et al. (2014) to make overall outcome assessment more efficient. This study will inform programs as to the conditions under which the more time-intensive “binning” approach is worthwhile, and will enable them to implement the approach more effectively if they so choose.

Ref: Stoker, Geoff; Blair, Jean; and Edward Sobiesk. “Meaningful Assessment.” Proceedings of the 15th Annual Conference on Information Technology Education. Atlanta, GA: 15-18 October, 2014.