Page 1 of 17
Advances in Social Sciences Research Journal – Vol. 11, No. 5
Publication Date: May 25, 2024
DOI:10.14738/assrj.115.17016.
Shah, A. P. (2024). Reliability, Sensitivity, and Specificity Measures of the Comprehensive Assessment of Accentedness and
Intelligibility (CAAI) Test Battery. Advances in Social Sciences Research Journal, 11(5). 318-334.
Services for Science and Education – United Kingdom
Reliability, Sensitivity, and Specificity Measures of the
Comprehensive Assessment of Accentedness and Intelligibility
(CAAI) Test Battery
Amee P. Shah
School of Health Sciences, Stockton University, Galloway, New Jersey
ABSTRACT
Purpose: This paper aims to introduce and report on measures of validation of a
diagnostic assessment, the Comprehensive Assessment of Accentedness and
Intelligibility (CAAI) Test Battery that identifies communication issues related to a
person’s spoken accent. Due to scant availability of standardized assessments of
accents, this test battery was developed to enable evidence-based diagnoses and
determine severity levels of dialect- and accent-related communication difficulties
(Shah, 2024). The present paper reports on psychometric testing conducted on the
test battery with the resulting measures of reliability, sensitivity, and specificity of
the CAAI. Methods: Native and nonnative speakers of English (N = 61) were
administered the CAAI Test Battery, and their communication was evaluated and
scored. Descriptive statistics were used to examine test sensitivity, specificity,
interrater reliability, inter-item (internal consistency) reliability, and test-retest
reliability. Results: The CAAI Test Battery meets high standards for test sensitivity
in identifying accent-related communication concerns. Test specificity was high for
not falsely identifying native speakers as having accent-related concerns. The
instrument had strong interrater reliability across all 20 sections of the test
(Pearson correlation coefficients ranged from 0.68 to 1.00, all highly significant at
p<0.01). The test showed strong inter-item reliability/internal consistency with a
large number of sections yielding moderate to high Cronbach’s alpha coefficient
range of 0.70 to 1.00. The test showed strong test-retest reliability with correlation
coefficients ranging from 0.73 to 1.00, significant at p<0.05. Conclusions: With high
sensitivity, specificity, and reliability, the CAAI Test Battery is found to be a stable
and meaningful means to assess dialect- and accent-related communication
concerns. The CAAI Test Battery helps fill a crucial gap in the area of
accents/dialects that lacks other validated assessment measures. With a sensitive
and reliable assessment, clinicians and teachers can identify an accurate baseline
pattern of errors to address for accent management or pronunciation teaching and
achieve effective outcomes in a short amount of time.
INTRODUCTION: STATE OF THE PRACTICES
Considering the growing diversity in the United States, the practices of accent management and
teaching of pronunciation to English as Second Language Learners (ESL) have been increasing
in prevalence and popularity over the last fifty years with a significant growth in the past two
decades. Professional position papers in disciples of speech-language pathology and Teaching
English as Foreign Language (TEFL) advocate for addressing accents and pronunciation as part
of the professional curriculum. For example, the American Speech-Language-Hearing
Association (ASHA) formally recognized the area of practice pertaining to dialect- and accent-
Page 2 of 17
319
Shah, A. P. (2024). Reliability, Sensitivity, and Specificity Measures of the Comprehensive Assessment of Accentedness and Intelligibility (CAAI) Test
Battery. Advances in Social Sciences Research Journal, 11(5). 318-334.
URL: http://dx.doi.org/10.14738/assrj.115.17016
related services in two positions papers and the scope of practice (ASHA, 1983, 1985a, 2007).
However, in contrast to the increase in clinical and teaching offerings to ESL clients, not much
progress has occurred in providing evidence using standardized instruments and methods to
move towards an evidence-based practice of accent management (See detailed report in Shah,
2024). A recent systematic study (Gu and Shah, 2019) showed that programs attempting to
provide accent-related training do not use standardized assessments nor measure outcomes in
a systematic manner. This trend has not changed in findings from national surveys from as long
as two decades ago (e.g., Schmidt and Sullivan, 2003; Shah, 2005). As a result, this area of
practice for speech-language pathologists as well as TESL/TEFL teachers remains subjective,
potentially impacting ESL clients’ progress, morale, as well as finances since they pay out-of- pocket for these services. As a crucial step towards toward regulating and standardizing this
area of practice, a recent assessment framework was reported in Shah, 20241, namely, the
Comprehensive Assessment of Accentedness and Intelligibility (CAAI) Assessment Framework.
The CAAI Assessment Framework provided theoretically grounded and clinically-proven areas
to assess and methods to assess using theory-driven, practical assessment material and
examples. Going further, to continue to build an evidence base foundation, Shah (2004) also
describes a testing instrument that builds upon this CAAI Assessment Framework, namely, the
CAAI Assessment Battery. This Test Battery has been used with clients in a large number of
clinics and organizations and has received peer-reviews and endorsement from ASHA,
developing and offering it as a nationally-approved webinar training for SLP clinicians (Shah,
2010). Going further on the path of standardization, the present paper reports on attempts to
validate the CAAI Assessment Battery. Specifically, owing to the lack of reliability or validity
measures on existing tests and tools for accents, for example the Phonological Assessment of
Foreign Accent (Compton, 2002) and the Proficiency in oral English communication (POEC;
Sikorski, 2002), the present paper assesses for internal reliability, internal consistency, test- retest reliability, specificity, and sensitivity measures with the CAAI Assessment Battery. The
following sections will first provide a short summary of the CAAI Assessment Battery and then
report the details of the psychometric assessments and the corresponding results.
CAAI: Introduction and Orientation
The Comprehensive Asssessment of Accentendess and Intelligibility Assessment Battery (CAAI:
Shah, 2007a), was developed to meet the needs and fill the gaps in the assessment of foreign- accented speech and dialectal variations. This test is designed to assess and diagnose dialect- and accent-related communication difficulties in an objective, quantitative manner. The
objective of the test is to help make assessments and follow-up training/teaching of clients as
evidence-based and data-driven as possible. The test is comprehensive, in that it targets a
multitude of typical errors that range from syllable-level to discourse-level communication
abilities. The areas tested are those that specifically pose communication difficulties for dialect
and accented clients. With its comprehensive attempt to capture the errors, the test ensures
that all communication issues are identified in the beginning, so subsequent teaching/training
is effective.
The CAAI assessment battery includes a full set of stimuli material to conduct the assessment,
as well as a test manual, an examiner manual, a scoring form, a response form, as well as a case
history form. As mentioned earlier, the CAAI Assessment Battery uses a systematic framework
1 The original version of the CAAI Assessment Framework first appeared in Shah, 2007b and Shah, 2009a)
Page 3 of 17
320
Advances in Social Sciences Research Journal (ASSRJ) Vol. 11, Issue 5, May-2024
Services for Science and Education – United Kingdom
for targeting and assessing each area—the CAAI Assessment Framework. While the framework
can be used with other examples outside of those used from the CAAI Assessment Battery, it is
most efficacious and reliable if used together, as the present paper will demonstrate.
Table 1 (taken from the test manual of the CAAI, Shah, 2007a) shows a listing of the 22 sections
of the CAAI Assessment Battery and the broad categories of assessment areas they represent.
Table 1: Broad categories of the areas of assessment, and specific section numbers
underlying each broad category (taken from the CAAI Assessment Battery; Shah,
2007a).
Section
No.
Section Title (I.E., The Measured Area) Broad Categories
1 Intelligibility relative to accentedness
2 Intelligibility score & Rate of speech on Baseline intelligibility and rate of speech
narrative passage
3 Sentence-level intonation
Suprasegmental
aspects
of varying length
(from syllables
to sentences)
Speech production
abilities
4 Word-level intonation
5 Lexical stress in single words of varying
syllable-length
6 Derivative stress in multisyllabic words
7 Contrastive lexical stress
8 Emphasis
9 Sentence phrasing
10 Phrasing contrast in sentence pairs
11 Consonant word list (C)
Articulation of
sounds (C, V,
clusters)
12 Consonant clusters word list
13a Vowels word list: General words (V)
13b Vowels word list: Specific words
14 Phonological processes Phonological
patterns
15a Auditory Discrimination (paired contrasts) Speech perception
15b Auditory Discrimination (labeling of single
words)
16 Prepositions Prepositions
Language areas
17 Colloquial/idiomatic use of prepositions
18 Contrasting idiomatic phrases: Idioms
expression & comprehension
19 Comprehension of idiomatic expressions
20 Advanced vocabulary Vocabulary
21 Conversational grammar & parts of speech
(syntax, morphology, and semantics)
Grammatical
and semantic
skills
22 Pragmatic problems Social-pragmatic
skills
(including
nonverbal
communication)