Validation of Mentee-Teachers’ Assessment Tool within the Framework of Generalisability Theory at the Faculty of Education, University for Development Studies

Abstract

Practitioners in assessment and other researchers have over the years expressed dissatisfaction with the lack of consistency in scores obtained from use of multiple measurement instruments.  Such scores derived from these largely inconsistent and unreliable procedures are relied upon by decision makers in taking very important decisions in education, health and other related fields. The purpose of this study therefore, was to apply the G theory procedures to validate the mentee assessment tool being used at the Faculty of Education, University for Development Studies. The G study involved estimating the generalisability (reliability-like) coefficients of mentees’ assessment scores, and determining the level of acceptability (validity) of these coefficients. A nested design was used because different sets of raters assessed different student-mentees on different occasions in the field. The relationship among these variables (facets); students, raters and occasions, appropriately mirrored a nested relationship. Data obtained by raters on 300 students, in the 2018/2019 off-campus teaching practice, were entered into EDUG software for relevant analysis of the results. The study found that both rater and student facets accounted for the largest measurement errors in mentees’ observed scores, reporting an estimated G coefficient of 0.62(62%), and representing a positive moderate relationship. Based on these findings, the study concluded that, the quality of mentee observed scores could be improved for either relative or absolute decisions by varying the number of levels of both raters and occasions. To achieve acceptable G coefficient values of 0.83 and above, it is recommended that, decision makers employ a model that uses four raters per occasion for three occasions of assessment.