Identifying CRAAP on the Internet: A Source Evaluation Intervention
DOI:
https://doi.org/10.14738/assrj.97.12670Keywords:
epistemic cognition, source evaluation, intervention, digital literacyAbstract
Individuals of all ages struggle to determine the reliability of information on the internet. To address this common issue, many educational institutions have endorsed the CRAAP test (i.e., currency, relevance, authority, accuracy, and purpose) as an effective approach to support identification of unreliable information. The present study extended the CRAAP test by incorporating a modeling component on how to evaluate and integrate multiple sources of varying quality on the internet and evaluated the efficacy of this source evaluation training intervention. Eighty-two participants across Canada were recruited to evaluate six authentic webpages and then construct an argument on a specific topic. Half the sample received training to examine the currency, relevance, authority, accuracy, and purpose (i.e., CRAAP) of the webpages before completing the online activity. Results revealed that the intervention group provided more accurate rank-ordering of the webpages, but no differences were found between groups on source integration via an argumentative essay. These findings suggest that the CRAAP test is effective in improving individuals’ evaluations of online sources but is not effective in promoting better quality source integration.
References
Barnes, M. D., Penrod, C., Neiger, B. L., Merrill, R. M., Thackeray, R., Eggett, D. L., & Thomas, E. (2003). Measuring the relevance of evaluation criteria among health information seekers on the Internet. Journal of Health Psychology, 8(1), 71–82. https://doi.org/10.1177/1359105303008001436
Barzilai, S., & Chinn, C. A. (2018). On the goals of epistemic education: Promoting apt epistemic performance. Journal of the Learning Sciences, 27(3), 353–389. https://doi.org/10.1080/10508406.2017.1392968
Barzilai, S., & Eshet-Alkalai, Y. (2015). The role of epistemic perspectives in comprehension of multiple author viewpoints. Learning and Instruction, 36, 86–103. https://doi.org/10.1016/j.learninstruc.2014.12.003
Barzilai, S., Mor-Hagani, S., Zohar, A. R., Shlomi-Elooz, T., & Ben-Yishai, R. (2020). Making sources visible: Promoting multiple document literacy with digital epistemic scaffolds. Computers & Education, 157, 103980. https://doi.org/10.1016/j.compedu.2020.103980
Barzilai, S., Thomm, E., & Shlomi-Elooz, T. (2020). Dealing with disagreement: The roles of topic familiarity and disagreement explanation in evaluation of conflicting expert claims and sources. Learning and Instruction, 69, 101367. https://doi.org/10.1016/j.learninstruc.2020.101367
Barzilai, S., & Zohar, A. (2014). Reconsidering personal epistemology as metacognition: A multifaceted approach to the analysis of epistemic thinking. Educational Psychologist, 49(1), 13–35. https://doi.org/10.1080/00461520.2013.863265
Barzilai, S., Zohar, A. R., & Mor-Hagani, S. (2018). Promoting integration of multiple texts: A review of instructional approaches and practices. Educational Psychology Review, 1–27.
Batchelor, O. (2017). Getting out the truth: The role of libraries in the fight against fake news. Reference Services Review, 143-148.
Bates, B. R., Romina, S., Ahmed, R., & Hopson, D. (2006). The effect of source credibility on consumers’ perceptions of the quality of health information on the Internet. Medical Informatics and the Internet in Medicine, 31(1), 45–52. https://doi.org/10.1080/14639230600552601
Blakeslee, S. (2004). The CRAAP test. Loex Quarterly, 31(3), 4.
Braasch, J. L. G., Bråten, I., Strømsø, H. I., Anmarkrud, Ø., & Ferguson, L. E. (2013). Promoting secondary school students’ evaluation of source features of multiple documents. Contemporary Educational Psychology, 38(3), 180–195. https://doi.org/10.1016/j.cedpsych.2013.03.003
Bråten, I., Strømsø, H. I., & Andreassen, R. (2016). Sourcing in professional education: Do text factors make any difference? Reading and Writing, 29(8), 1599–1628.
Bråten, I., Strømsø, H. I., & Britt, M. A. (2009). Trust matters: Examining the role of source evaluation in students’ construction of meaning within and across multiple texts. Reading Research Quarterly, 44(1), 6–28.
Breakstone, J., McGrew, S., Smith, M., Ortega, T., & Wineburg, S. (2018). Why we need a new approach to teaching digital literacy. Phi Delta Kappan, 99(6), 27–32.
Britt, M. A., & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20(4), 485–522.
Chevrier, M., Muis., K. R., Pekrun, R., Sinatra, G., & Denton, C. A. (2019). Thinking critically about socio-scientific issues: The roles of epistemic cognition and epistemic emotions [Unpublished manuscript]. Department of Educational and Counselling Psychology, McGill University.
Chiagouris, L., Long, M. M., & Plank, R. E. (2008). The consumption of online news: The relationship of attitudes toward the site and credibility. Journal of Internet Commerce, 7(4), 528–549. https://doi.org/10.1080/15332860802507396
Chinn, C. A., Buckland, L. A., & Samarapungavan, A. (2011). Expanding the dimensions of epistemic cognition: Arguments from philosophy and psychology. Educational Psychologist, 46(3), 141–167. https://doi.org/10.1080/00461520.2011.587722
Chinn, C. A., Rinehart, R. W., & Buckland, L. A. (2014). Epistemic cognition and evaluating information: Applying the AIR model of epistemic cognition. In D. N. Rapp & J. L.G. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 425–453). MIT Press.
Chinn, C., & Sandoval, W. A. (2018). Epistemic cognition and epistemic development. In F. Fischer, C. E. Hmelo-Silver, S. R. Goldman, & P. Resiman (Eds.), International handbook of the learning sciences (pp. 46–55). Routledge.
Canadian Mental Health Association (CMHA). (2018a). Stigma and discrimination. Retrieved from https://ontario.cmha.ca/documents/stigma-and-discrimination/
CMHA. (2018b). Myths about mental illness. Retrieved from: https://cmha.ca/documents/myths-about-mental-illness
Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2013). Applied multiple regression/correlation analysis for the behavioral sciences. Routledge.
Cresswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research (2nd ed.). Sage Publications.
Cresswell, J. W., & Plano Clark, V. L. (2017). Designing and conducting mixed methods research (3rd ed.). Sage Publications.
Denton, C. A., Muis, K.R., Armstrong, S., & Dubé, A. (in preparation). En-garde: Source evaluations in the digital age. Contemporary Educational Psychology.
Duby, H. R. (2018). What a load of CRAAP: Evaluating information in an era of "fake news". Tennessee Libraries, 68(4).
Eshet-Alkalai, Y., & Chajut, E. (2009). Changes over time in digital literacy. CyberPsychology & Behavior, 12(6), 713–715.
Eshet-Alkalai, Y., & Chajut, E. (2010). You can teach old dogs new tricks: The factors that affect changes over time in digital literacy. Journal of Information Technology Education: Research, 9(1), 173–181.
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160. https://doi.org/10.3758/BRM.41.4.1149
Fielding, J. A. (2019). Rethinking CRAAP: Getting students thinking like fact-checkers in evaluating web sources. College & Research Libraries News, 80(11), 620.
Gerjets, P., Kammerer, Y., & Werner, B. (2011). Measuring spontaneous and instructed evaluation processes during Web search: Integrating concurrent thinking-aloud protocols and eye-tracking data. Learning and Instruction, 21(2), 220–231. https://doi.org/10.1016/j.learninstruc.2010.02.005
Green, J. (2019, Jan 22). Check yourself with lateral reading: Crash course navigating digital information #3 [Youtube video]. https://www.youtube.com/watch?v=GoQG6Tin-1E&t=157s.
Greene, J. A., Cartiff, B. M., & Duke, R. F. (2018). A meta-analytic review of the relationship between epistemic cognition and academic achievement. Journal of Educational Psychology. https://doi.org/10.1037/edu0000263
Greene, J. A., Copeland, D. Z., Deekens, V. M., & Seung, B. Y. (2018). Beyond knowledge: Examining digital literacy’s role in the acquisition of understanding in science. Computers & Education, 117, 141–159.
Greene, J. A., Yu, S. B., & Copeland, D. Z. (2014). Measuring critical components of digital literacy and their relationships with learning. Computers & Education, 76, 55–69. https://doi.org/10.1016/j.compedu.2014.03.008
Halverson, K. L., Siegel, M. A., & Freyermuth, S. K. (2010). Non-science majors’ critical evaluation of websites in a biotechnology course. Journal of Science Education and Technology, 19(6), 612–620. https://doi.org/10.1007/s10956-010-9227-6
Hämäläinen, E. K., Kiili, C., Marttunen, M., Räikkönen, E., González-Ibáñez, R., & Leppänen, P. H. T. (2020). Promoting sixth graders’ credibility evaluation of Web pages: An intervention study. Computers in Human Behavior, 110, 106372. https://doi.org/10.1016/j.chb.2020.106372
Jung, E. H., Walsh-Childers, K., & Kim, H.-S. (2016). Factors influencing the perceived credibility of diet-nutrition information web sites. Computers in Human Behavior, 58, 37–47. https://doi.org/10.1016/j.chb.2015.11.044
Jung, W. S., Chung, M.-Y., & Rhee, E. S. (2018). The effects of attractiveness and source expertise on online health sites. Health Communication, 33(8), 962–971. https://doi.org/10.1080/10410236.2017.1323364
Kąkol, M., Nielek, R., & Wierzbicki, A. (2017). Understanding and predicting Web content credibility using the Content Credibility Corpus. Information Processing & Management, 53(5), 1043–1061. https://doi.org/10.1016/j.ipm.2017.04.003
Kiili, C., Laurinen, L., & Marttunen, M. (2008). Students evaluating Internet sources: From versatile evaluators to uncritical readers. Journal of Educational Computing Research, 39(1), 75–95. https://doi.org/10.2190/EC.39.1.e
Kobau, R., DiIorio, C., Chapman, D., Delvecchio, P., & SAMHSA/CDC Mental Illness Stigma Panel Members. (2010). Attitudes about mental illness and its treatment: Validation of a generic scale for public health surveillance of mental illness associated stigma. Community Mental Health Journal, 46(2), 164–176. https://doi.org/10.1007/s10597-009-9191-x
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159. https://doi.org/10.2307/2529310
Machackova, H., & Smahel, D. (2018). The perceived importance of credibility cues for the assessment of the trustworthiness of online information by visitors of health-related websites: The role of individual factors. Telematics and Informatics, 35(5), 1534–1541. https://doi.org/10.1016/j.tele.2018.03.021
Mason, L., Ariasi, N., & Boldrin, A. (2011). Epistemic beliefs in action: Spontaneous reflections about knowledge and knowing during online information searching and their influence on learning. Learning and Instruction, 21(1), 137–151. https://doi.org/10.1016/j.learninstruc.2010.01.001
Mason, L., Junyent, A. A., & Tornatora, M. C. (2014). Epistemic evaluation and comprehension of web-source information on controversial science-related topics: Effects of a short-term instructional intervention. Computers & Education, 76, 143–157. https://doi.org/10.1016/j.compedu.2014.03.016
Mason, L., Scrimin, S., Tornatora, M. C., Suitner, C., & Moè, A. (2018). Internet source evaluation: The role of implicit associations and psychophysiological self-regulation. Computers & Education, 119, 59–75. https://doi.org/10.1016/j.compedu.2017.12.009
McGrew, S. (2020). Learning to evaluate: An intervention in civic online reasoning. Computers & Education, 145, 103711. https://doi.org/10.1016/j.compedu.2019.103711
McGrew, S., Smith, M., Breakstone, J., Ortega, T., & Wineburg, S. (2019). Improving university students’ web savvy: An intervention study. British Journal of Educational Psychology, bjep.12279. https://doi.org/10.1111/bjep.12279
Meppelink, C. S., Smit, E. G., Fransen, M. L., & Diviani, N. (2019). “I was right about vaccination”: Confirmation bias and health literacy in online health information seeking. Journal of Health Communication, 24(2), 129–140. https://doi.org/10.1080/10810730.2019.1583701
Meriam Library of CSU Chico. (2010). Evaluating information - Applying the CRAAP test [PDF document]. Meriam Library of CSU Chico. https://library.csuchico.edu/sites/default/files/craap-test.pdf
Meyers, L. S., Gamst, G., & Guarino, A. J. (2016). Applied multivariate research: Design and interpretation. Sage Publications.
Morse, J. M. (2003). Principles of mixed methods and multimethod research design. In A. Tashakkori, & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 189–208). Sage Publications.
Pretorius, L. (2018). Experiential and self-discovery learning in digital literacy: Developing the discernment to evaluate source reliability. College & Undergraduate Libraries, 25(4), 388–405. https://doi.org/10.1080/10691316.2018.1530626
Queen's University Library. (2020). CRAAP criteria. Bachelor of health sciences toolbox. https://guides.library.queensu.ca/BHSc/CRAAP
Richter, T., & Maier, J. (2017). Comprehension of multiple documents with conflicting information: A two-step model of validation. Educational Psychologist, 52(3), 148–166. https://doi.org/10.1080/00461520.2017.1322968
Richter, T., & Schmid, S. (2010). Epistemological beliefs and epistemic strategies in self-regulated learning. Metacognition and Learning, 5(1), 47–65.
Rouet, J.-F., & Britt, M. A. (2011). Relevance processes in multiple document comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 19–52). IAP Information Age Publishing.
Salmerón, L., Gil, L., & Bråten, I. (2018). Effects of reading real versus print-out versions of multiple documents on students’ sourcing and integrated understanding. Contemporary Educational Psychology, 52, 25–35. https://doi.org/10.1016/j.cedpsych.2017.12.002
Sandoval, W. A. (2017). Situating practices of epistemic cognition. In T. G. Amin & O. Levrini (Eds.), Converging perspectives on conceptual change (1st ed., pp. 253–260). Routledge. https://doi.org/10.4324/9781315467139-30
Smallwood, C. (2015). The complete guide to using Google in libraries: Instruction, administration, and staff productivity (Vol. 1). Rowman & Littlefield.
Stanford History Education Group. (2016). Evaluating information: The cornerstone of online civic reasoning [PDF document]. https://stacks.stanford.edu/file/druid:fv751yt5934/SHEG%20Evaluating%20Information%20Online.pdf
Stanford History Education Group. (2020). COR. Civic Online Reasoning. https://cor.stanford.edu
Subramaniam, M., Taylor, N. G., St. Jean, B., Follman, R., Kodama, C., & Casciotti, D. (2015). As simple as that?: Tween credibility assessment in a complex online world. Journal of Documentation, 71(3), 550–571. https://doi.org/10.1108/JD-03-2014-0049
Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics: International edition. Pearson.
Taylor, W. D., Johnson, G., Ault, M. K., Griffith, J. A., Rozzell, B., Connelly, S., Jensen, M. L., Dunbar, N. E., & Ness, A. M. (2015). Ideological group persuasion: A within-person study of how violence, interactivity, and credibility features influence online persuasion. Computers in Human Behavior, 51, 448–460. https://doi.org/10.1016/j.chb.2015.04.071
The News Literacy Project. (2020). Checkology. Checkology. https://get.checkology.org
Ulyshen, T. Z., Koehler, M. J., & Gao, F. (2015). Understanding the connection between epistemic beliefs and Internet searching. Journal of Educational Computing Research, 53(3), 345–383.
Van Strien, J. L., Brand-Gruwel, S., & Boshuizen, H. P. (2014). Dealing with conflicting information from multiple nonlinear texts: Effects of prior attitudes. Computers in Human Behavior, 32, 101–111.
Walraven, A., Brand-Gruwel, S., & Boshuizen, H. P. A. (2013). Fostering students’ evaluation behaviour while searching the internet. Instructional Science, 41(1), 125–146. https://doi.org/10.1007/s11251-012-9221-x
White, A. M., DeCuir-Gunby, J. T., & Kim, S. (2019). A mixed methods exploration of the relationships between the racial identity, science identity, science self-efficacy, and science achievement of African American students at HBCUs. Contemporary Educational Psychology, 57, 54–71. https://doi.org/10.1016/j.cedpsych.2018.11.006
Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A. (2009). Source evaluation, comprehension, and learning in Internet science inquiry tasks. American Educational Research Journal, 46(4), 1060–1106. https://doi.org/10.3102/0002831209333183
Wineburg, S., & McGrew, S. (2017). Lateral reading: Reading less and learning more when evaluating digital information [PDF document]. https://pdfs.semanticscholar.org/5b26/9628f4dc29b514dfcb0b5e429e49fc0dae6d.pdf?_ga=2.134731505.1090439893.1607365627-347836626.1607210879
Wineburg, S. S. (1991). On the reading of historical texts: Notes on the breach between school and academy. American Educational Research Journal, 28(3), 495–519.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Krista Renee Muis, Courtney Denton, Adam Dubé
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors wishing to include figures, tables, or text passages that have already been published elsewhere are required to obtain permission from the copyright owner(s) for both the print and online format and to include evidence that such permission has been granted when submitting their papers. Any material received without such evidence will be assumed to originate from the authors.
