Assessing Academic and Disciplinary Literacies: Rubric validation to measure argumentation, comparison and source-based writing skills
Abstract
Introduction. Over the last two decades, research has suggested that academic and disciplinary literacies (ADLs) are key to integrating content and language, as language is used to express content knowledge, often through Cognitive Discourse Functions, discourse patterns that respond to cognitive actions in formal education contexts. Nevertheless, systematic assessment tools are required to assess student production of ADLs.
Purpose. This validation study focuses on developing an analytic rubric to measure three ADL skills, consisting of three dimensions. Two of these dimensions are CDFs: students’ skills to argue and compare. A third rubric measures an additional academic skill, students’ ability to write from sources. The rubrics are designed to capture cross-disciplinary and multilingual productions of these skills, therefore being applicable for various disciplines (history, science, mathematics, among others…) and languages.
Methods. A five-step validation process based on expert judgement was used, involving 13 international experts. They quantitatively and qualitatively evaluated the proposed rubrics based on pertinence, conceptual clarity, coherence, and relevance across two iterative rounds. Quantitative descriptive statistics and agreement indices were used, in addition to thematic analysis of qualitative feedback.
Findings. The quality of the rubric showed clear progression between validation rounds. In the first version, several issues were raised by the experts–most criteria showed weak agreement and low means. After revisions, there was a substantial improvement in the second version, with 83 % of the criteria reaching strong or acceptable agreement. Qualitative feedback highlighted the need for precise and multidimensional operationalisations in each ADL dimension. To illustrate the application of the rubrics, multilingual student samples are provided.
Conclusions. The rubrics offer the first steps towards systematising ADL assessment by combining qualitative and quantitative feedback from 13 experts, underscoring the importance of expert input in advancing assessment practices. The study has theoretical and practical contributions: it highlights the multidimensional nature of ADLs and provides an adaptable rubric that can be used across disciplines, languages and educational contexts. This study focused on the validation process; therefore, empirical use of the rubric is still required. Future research should apply the rubrics to large-scale corpora and complement the expert-based validation with psychometric approaches.
Downloads
References
Allagui, B. (2019). Investigating the quality of argument structure in first-year university writing. In S. Hidri (Ed.), English language teaching research in the Middle East and North Africa (pp. 179–199). Palgrave Macmillan. https://doi.org/10.1007/978-3-319-98533-6_9
Arias-Hermoso, R., Imaz Agirre, A., & Garro Larrañaga, E. (2024). A comparison between input modalities and languages in source-based multilingual argumentative writing. Assessing Writing, 60, 100813. https://doi.org/10.1016/j.asw.2024.100813
Arias-Hermoso, R., Garro Larrañaga, E., Imaz Agirre, A., & Dalton-Puffer, C. (2025a). Producing cognitive discourse functions in disciplinary Basque writing: developmental patterns across secondary education and L1 profiles. International Journal of Bilingual Education and Bilingualism, 28(7), 744–763. https://doi.org/10.1080/13670050.2025.2488736
Arias-Hermoso, R., Imaz Agirre, A., & Garro Larrañaga, E. (2025b). Multilingual disciplinary literacies: exploring developmental patterns of science writing across secondary education. Journal of Multilingual and Multicultural Development, 1–19. https://doi.org/10.1080/01434632.2025.2481200
Bauer-Marschallinger. (2022). CLIL with a capital I: Using cognitive discourse functions to integrate content and language learning in CLIL history education [Unpublished doctoral thesis]. University of Vienna.
Breeze, R., & Dafouz, E. (2017). Constructing complex Cognitive Discourse Functions in higher education: An exploratory study of exam answers in Spanish- and English-medium instruction settings. System, 70, 81-91. https://doi.org/10.1016/j.system.2017.09.024
Chan, S., & May, L. (2022). Towards more valid scoring criteria for integrated reading-writing and listening-writing summary tasks. Language Testing, 40(2), 410-439. https://doi.org/10.1177/02655322221135025
Chau, L.T., Leijten, M., Bernolet, S. & Vangehuchten, L. (2022). Envisioning multilingualism in source-based writing in L1, L2, and L3: The relation between source use and text quality. Frontiers in Psychology, 13, 01-20. https://doi.org/10.3389/fpsyg.2022.914125
Chen, M., & Zhou, H. (2022). Comparison-and-contrast in research articles of applied linguistics: A frame-based analysis. Lingua, 276, 103387. https://doi.org/10.1016/j.lingua.2022.103387
Cheong, C. M., Zhu, X. & Liao, X. (2018). Differences between the relationship of L1 learners’ performance in integrated writing with both independent listening and independent reading cognitive skills. Reading and Writing, 31, 779–811. https://doi.org/10.1007/s11145-017-9811-8
Cheong, C. M., Zhu, X. & Xu, W. (2021). Source-based argumentation as a form of sustainable academic skill: An exploratory study comparing secondary school students’ L1 and L2 writing. Sustainability, 132, 2869. https://doi.org/10.3390/su132212869
Coffin, C. (2009). Historical discourse: The language of time, cause and evaluation. Bloomsbury.
Connolly, T. (2019). Die Förderung vertiefter Lernprozesse durch Sachfachliteralität: Eine vergleichende Studie zum expliziten Scaffolding kognitiver Diskursfunktionen im bilingualen Chemieunterricht am Beispiel des Erklärens [] [Unpublished doctoral dissertation]. Johannes Gutenberg-Universität Mainz.
Crossley, S. A., Tian, Y., & Wan, Q. (2022). Argumentation features and essay quality: Exploring relationships and incidence counts. Journal of Writing Research, 14(1), 1–34. https://doi.org/10.17239/jowr-2022.14.01.01
Dalton-Puffer, C., & Bauer-Marschallinger, S. (2019). Cognitive discourse functions meet historical competences: Towards an integrated pedagogy in CLIL history education. Journal of Immersion and Content-Based Language Education, 7(1), 30–60. https://doi.org/10.1075/jicb.17017.dal
Dalton-Puffer, C., Bauer-Marschallinger, S., Brückl-Mackey, K., Hofmann, V., Hopf, J., Kröss, L., & Lechner, L. (2018). Cognitive discourse functions in Austrian CLIL lessons: Towards an empirical validation of the CDF construct. European Journal of Applied Linguistics, 6(1), 5–29. https://doi.org/10.1515/eujal-2017-0028
Dalton-Puffer, C., Hüttner, J., & Nikula, T. (2024). The conceptualisation of disciplinary literacies in CLIL. In J. Hüttner & C. Dalton-Puffer (Eds.), Building disciplinary literacies in content and language integrated learning (pp. 1-25). Routledge.
Dalton-Puffer, C. (2013). A construct of cognitive discourse functions for conceptualising content-language integration in CLIL and multilingual education. European Journal of Applied Linguistics, 1(2), 216–253. https://doi.org/10.1515/eujal-2013-0011
Dalton-Puffer, C. (2016). Cognitive discourse functions: Specifying an integrative interdisciplinary construct. In T. Nikula, E. Dafouz, P. Moore, & U. Smit. (Eds.), Conceptualising Integration in CLIL and Multilingual Education (pp. 29–54). Multilingual Matters.
deBoer, M. (2020). Teacher-based assessment of learner-led interactions in CLIL: The power of cognitive discourse functions. In M. deBoer & D. Leontjev (Eds.), Assessment and learning in content and language integrated learning (CLIL) classrooms (pp. 203–223). Springer. https://doi.org/10.1007/978-3-030-54128-6_10
del Pozo, E., & Llinares, A. (2021). Assessing students’ learning of history content in Spanish CLIL programmes: A content and language integrated perspective, in C. Hemmi, & D. L. Banegas (Eds.), International Perspectives on CLIL. International perspectives on English language teaching. Palgrave MacMillan. https://doi.org/10.1007/978-3-030-70095-9_3
del Pozo, E. (2024). Assessment in CLIL: the pending subject in bilingual education? A case study. Revista De Educación, 1(403), 231–257. https://doi.org/10.4438/1988-592X-RE-2024-403-605
Evnitskaya, N., & Dalton-Puffer, C. (2024). CLIL learners’ categorizations: Writing about history in English across three grade levels in Spanish bilingual schools. In J. Hüttner & C. Dalton-Puffer (Eds.), Building disciplinary literacies in content and language integrated learning (pp. 60-79). Routledge.
Evnitskaya, N., & Dalton-Pufrer, C. (2023). Cognitive discourse functions in CLIL classrooms: Eliciting and analysing students’ oral categorizations in science and history. International Journal of Bilingual Education and Bilingualism, 26(3), 311–330. https://doi.org/10.1080/13670050.2020.1804824
Fang, Z. & Coatoam, S. (2013). Disciplinary literacy: what you want to know about it. Journal of Adolescent and Adult Literacy, 56 (8), 627-632. https://doi.org/10.1002/JAAL.190
Gerns, P. (2023a). Building scientific knowledge in English: Integrating content, cognition and communication in secondary school CLIL biology. Journal of Language and Education, 9(3), 52–78. https://doi.org/10.17323/jle.2023.17569
Gerns, P. (2023b). Qualitative insights and a first evaluation tool for teaching with cognitive discourse function: “Comparing” in the CLIL science classroom. Porta Linguarum, 40, 161–179. https://doi.org/10.30827/portalin.vi40.26619
Granados, A., & Lorenzo, F. (2024). A functional description of disciplinary literacy in history: Applications of the Common European Framework of Reference for Languages to content and language integrated learning courses. In J. Hüttner & C. Dalton-Puffer (Eds.), Building disciplinary literacies in content and language integrated learning (pp. 83–100). Routledge.
Halliday, M. A. K., & Matthiessen, C. (2014). Halliday’s Introduction to Functional Grammar (4th ed.). Routledge.
Harsch, C., Koval, V., Kanistra, P. V., & Delgado-Osorio, X. (2024). Validating an integrated reading-into-writing scale with trained university students. Assessing Writing, 62, 100894. https://doi.org/10.1016/j.asw.2024.100894
Imaz Agirre, A., Arias-Hermoso, R., & Ipiña, N. (2025). The effect of an intervention focused on academic language on CAF measures in the multilingual writing of secondary students. International Review of Applied Linguistics in Language Teaching, 63(2), 1373–1396. https://doi.org/10.1515/iral-2023-0137
Jackson, D. O. (2024). The longitudinal development of argumentative writing in an English for academic purposes course in Japan. System, 103482. https://doi.org/10.1016/j.system.2024.103482
Jiménez-Aleixandre, M. P., & Erduran, S. (2007) Argumentation in science education: An overview. In S. Erduran & M. P. Jiménez-Aleixandre (Eds.), Argumentation in science education (pp. 3–27). Springer.
Kato, M. (2018). Exploring the Transfer Relationship of Summarizing Skills in L1 and L2. English Language Teaching Archives, 11(10). https://doi.org/10.5539/elt.v11n10p75
Kuiken, F., & Vedder, I. (2020). Scoring approaches: Scales/rubrics. In P. Winke & T. Brunfaut (Eds.), The Routledge handbook of second language acquisition and language testing (1st ed., pp. 10). Routledge. https://doi.org/10.4324/9781351034784
Landeta, J., & Lertxundi, A. (2024). Quality indicators for Delphi studies. Futures & Foresight Science, 6(1), e172. https://doi.org/10.1002/ffo2.172
Lázaro Cantabrana, J. L., & Gisbert Cervera, M. (2015). El desarrollo de la competencia digital docente a partir de una experiencia piloto de formación en alternancia en el Grado de Educación [Developing teachers’ digital competence through a pilot alternating training experience in the Bachelor’s degree in education]. EDUCAR, 51(2), 321-348. http://dx.doi.org/10.5565/rev/educar.725
Lersundi, A. (2022). Analysis of subject-specific literacies in a multidisciplinary project in upper-secondary education. Case Study. [Unpublished Doctoral thesis]. Mondragon Unibertsitatea. https://hdl.handle.net/20.500.11984/5964
Lesterhuis, M., Verhavert, S., Coertjens, L., Donche, V., & De Maeyer, S. (2017). Comparative judgement as a promising alternative to score competences. In E. Cano & G. Ion (Eds.), Innovative practices for higher education assessment and measurement (pp. 119-138). IGI Global Scientific Publishing. https://doi.org/10.4018/978-1-5225-0531-0.ch007
Li, D. (2022). A review of academic literacy research development: From 2002 to 2019. Asian-Pacific Journal of Second and Foreign Language Education, 7(1), 1-22. https://doi.org/10.1186/s40862-022-00130-z
Liu, F., & Stapleton, P. (2014). Counterargumentation and the cultivation of critical thinking in argumentative writing: Investigating washback from a high-stakes test. System, 45, 117-128. https://doi.org/10.1016/j.system.2014.05.005
Llinares, A., Morton, T., & Whittaker, R. (2024). Fostering language awareness for integration through teacher-researcher collaboration in a Spanish bilingual education context. Language Awareness, 1–21. https://doi.org/10.1080/09658416.2024.2385766
Llinares, A., & Morton, T. (2024). Assessing CLIL students’ expression of Explore across languages and school disciplines: An interdisciplinary approach, in J. Hüttner & C. Dalton-Puffer (Eds.), Building Disciplinary literacies in content and language integrated learning (pp. 101-118). Routledge. https://doi.org/10.4324/9781003403685
Lorenzo, F., Lorenzo, F., Cvikic, L., Llinares, A., De Boer, M., Adadan, E., Arias-Hermoso, R., Ćaleta, M., Demirkol Orak, S., Evnitskaya, N., Glasnović Gracin, D., Granados, A., Guzmán-Alcón, I., Kasprzak, M., Lehesvuori, S., Miloshevska, L., Özdemir, H., Piacentini, V., del Pozo, E., Roquet, H., Bagalová, D., and Ting, T. (2024). Assessing disciplinary literacy with CEFR descriptors: History, Mathematics and Science. A paper by CLILNetLE Working Group 2. V1. PHAIDRA repository, University of Vienna. https://doi.org/10.25365/phaidra.532_30
Lorenzo, F. (2017). Historical literacy in bilingual settings: Cognitive academic language in CLIL history narratives. Linguistics and Education, 37, 32-41. https://doi.org/10.1016/j.linged.2016.11.002
Lorenzo, F. (2023). Academic language as linguistic capital – A window to social justice: A commentary on “Midadolescents’ language learning at school: Toward more just and scientifically rigorous practices in research and education.” Language Learning, 73(1), 136-143. https://doi.org/10.1111/lang.12560
Meneses, A., Montenegro, M., Acevedo, D., Figueroa, J., & Hugo, E. (2023). Cross-disciplinary language changes in 4th graders as a predictor of the quality of written scientific explanation. Journal of Writing Research, 15(1), 105–132. https://doi.org/10.17239/jowr-2023.15.01.05
Moje, E. B. (2015). Doing and teaching disciplinary literacy with adolescent learners: A social and cultural enterprise. Harvard Educational Review, 85(2), 254–278. https://doi.org/10.17763/0017-8055.85.2.254
Morton, T., & Nashaat-Sobhy, n. (2024). exploring bases of achievement in content and language integrated assessment in a bilingual education program. TESOL Quarterly, 58(1), 5-31. https://doi.org/10.1002/tesq.3207
Morton, T. (2020). Cognitive discourse functions: A bridge between content, literacy and language for teaching and assessment in CLIL. CLIL Journal of Innovation and Research in Plurilingual and Pluricultural Education, 3(1), 7-17. https://doi.org/10.5565/rev/clil.33
Morton, T. (2022). Using cognitive discourse functions and comparative judgement to build teachers’ knowledge of content and language integration for assessment in a bilingual education program. Journal of Immersion and Content-Based Language Education, 10(2), 302-322. https://doi.org/10.1075/jicb.21017.mor
Nashaat-Sobhy, N., & Llinares, A. (2023). CLIL students’ definitions of historical terms. International Journal of Bilingual Education and Bilingualism, 9, 1–14. https://doi.org/10.1080/13670050.2020.1798868
Nikula, T., Nashaat-Sobhy, N., Minardi, S., Gülle, T., Yalçın, S., Duman, S.K., Bozbıyık, M., Delibegović Džanić, N., Ellison, M., Gerns, P., Gómez, E., Hanušová, S., Kováčiková, E., Kääntä, L., Lin, A., Llinares, A.; Yi Lo, Y., Lopriore, L., Meyer, O., Morton,T., Neville, C., Rannu, K., Sağlamel, H., Sula, G., Sulkunen, S., Pavičić Takač, V., Tiermas, A., Ting, T., Tsagari, D., Yüksel, G., & Žmavc, J. (2024). Towards an initial operationalisation of disciplinary literacies: A paper by CLILNetLE Working Group 1. PHAIDRA repository, University of Vienna. https://hdl.handle.net/11353/10.2050621
Paltridge, B., & Phakiti, A. (2015). Research Methods in Applied Linguistics: A practical resource. Bloomsbury.
Plakans, L., & Gebril, A. (2013). Using multiple texts in an integrated writing assessment: Source text use as a predictor of score. Journal of Second Language Writing, 22(3), 217-230. https://doi.org/10.1016/j.jslw.2013.02.003
Polias, J. (2016). Apprenticing students into science: doing, talking & writing scientifically. Lexis Education.
Pylonitis, C., & Meyer, O. (2024). Arguing for global citizenship: Mapping deeper learning in the language-as-discipline classroom. In S. Greco & L. Cinganotto (Eds.), Innovation in education for deeper learning (pp. 55–78). INDIRE-IUL Press.
Qin, J., & Karabacak, E. (2010). The analysis of Toulmin elements in Chinese EFL university argumentative writing. System, 38 (3), 444-456. https://doi.org/10.1016/j.system.2010.06.012
Rieder-Marschallinger, S. (2024). Cognitive discourse functions in history CLIL education: Insights from a design-based research study on conceptual links. In J. Hüttner & C. Dalton-Puffer (Eds.), Building disciplinary literacies in content and language integrated learning (pp. 195–215). Routledge.
Roca de Larios, J., Coyle, Y., & Garcia, V. (2023). The effects of using cognitive discourse functions to instruct 4th-year children on report writing in a CLIL science class. Studies in Second Language Learning and Teaching, 12(4), 597–622. https://doi.org/10.14746/ssllt.2022.12.4.4
Sampson, V., & Clark, D. (2008). Assessment of the ways students generate arguments in science education: Current perspectives and recommendations for future directions. Science Education, 92(3), 447–472. https://doi.org/10.1002/sce.20276
Schleppegrell, M. (2004). The language of schooling: a functional linguistic perspective. Mahwah: Erlbaum.
Sendur, K. A., van Drie, J., van Boxtel, C., & Kan, K. J. (2020). Historical reasoning in an undergraduate CLIL course: Students’ progression and the role of language proficiency. International Journal of Bilingual Education and Bilingualism, 25(6), 2058–2074. https://doi.org/10.1080/13670050.2020.1844136
Shanahan, T. & Shanahan, C. (2008). Teaching disciplinary literacy to adolescents: Rethinking content-area literacy. Harvard Educational Review, 78(1), 40-59. https://psycnet.apa.org/doi/10.17763/haer.78.1.v62444321p602101
Smith, S. (2019). Academic writing genres: Essays, reports & other genres. Evident Press.
Stapleton, P., & Wu, Y. (2015). Assessing the quality of arguments in students’ persuasive writing: A case study analyzing the relationship between surface structure and substance. Journal of English for Academic Purposes, 17, 12-23. https://doi.org/10.1016/j.jeap.2014.11.006
Steiss, J., Wang, J., Kim, S. G., & Olson, C. B. (2024). U.S. Secondary Students’ Source-Based Argument Writing in History. Written Communication. https://doi.org/10.1177/07410883241263549
Toulmin, S. (1958). The uses of argument. Cambridge University Press.
Trimble, L. (1985). English for science and technology: A discourse approach. Cambridge University Press.
Uludag, P., & McDonough, K. (2022). Validating a rubric for assessing integrated writing in an EAP context. Assessing Writing, 52, 100609. https://doi.org/10.1016/j.asw.2022.100609
van Weijen, D., Rijlaarsdam, G. & van den Bergh, H. (2019) Source use and argumentation behavior in L1 and L2 writing: A within-writer comparison. Reading and Writing, 32, 1635–1655. https://doi.org/10.1007/s11145-018-9842-9
Vercellotti, M. L., & McCormick, D. E. (2021). Constructing analytic rubrics for assessing open-ended tasks in the language classroom. TESL-EJ, 24(4).
Whittaker, R., & McCabe, A. (2023). Expressing evaluation across disciplines in primary and secondary CLIL writing: A longitudinal study. International Journal of Bilingual Education and Bilingualism, 5(2), 1–18. https://doi.org/10.1080/13670050.2020.1798869
Copyright (c) 2025 National Research University Higher School of Economics
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the Copyright Notice.
