A Mixed-Method Study on the Effectiveness of Using ChatGPT in Academic Writing and Students’ Perceived Experiences
Abstract
Introduction: The advent of artificial intelligence in education has brought forward tools like ChatGPT, which can potentially enhance students' academic writing abilities. However, there is limited empirical evidence examining its effectiveness and students' perceptions of its utility in academic contexts.
Purpose: This study aimed to measure the effect of using ChatGPT on students' academic writing abilities and to investigate students’ perceived experiences regarding the use of ChatGPT in their writing process.
Method: An explanatory mixed-method design was employed, incorporating a quantitative experiment followed by a qualitative investigation. The quantitative phase involved 102 fifth-semester students from an English education department at a university in Indonesia. These students were randomized into clusters based on their proximate writing test scores, resulting in two homogenous classes of 25 students each. These classes were then assigned to either an experimental group, which received 14 sessions using ChatGPT as a learning tool for academic writing, or a control group, which received 14 sessions using non-generative tools. Pre-tests and post-tests were administered to both groups. The qualitative phase involved interviews with 10 selected students from the experimental group to explore their perceived experiences with ChatGPT.
Results: The pre-test scores indicated homogeneity between the experimental and control groups, with scores of 57.15 and 56.35 respectively. After the intervention, the post-test scores revealed significant improvement in the experimental group, with an average score of 81.11 compared to 60.30 in the control group. Statistical analysis demonstrated a significant disparity between the two groups (p-value = 0.0000 < 0.05 for the experimental group and p-value = 0.067 > 0.05 for the control group), suggesting that the use of ChatGPT significantly enhanced students' academic writing abilities. The qualitative findings supported these results, with students reporting that ChatGPT facilitated idea generation, organization, and construction in their writing process.
Conclusion: The study concludes that ChatGPT significantly improves students' academic writing abilities, as evidenced by both quantitative and qualitative data. The tool's capacity to assist in the formulation and organization of ideas presents substantial potential for its use in academic research and writing. Given these findings, ChatGPT could be a valuable addition to the educational toolkit for enhancing academic writing skills.
Downloads
References
Al Yakin, A., Muthmainnah, M., Apriani, E., Obaid, A. J., & Elngar, A. A. (2023). Transforming online learning management: Generative models on ChatGPT to enhance online student engagement scale (OLE). Idarah (Jurnal Pendidikan Dan Kependidikan), 7(2), P. 135–148. https://doi.org/10.47766/idarah.v7i2.1514
AlAfnan, M. A., Dishari, S., Jovic, M., & Lomidze, K. (2023). Chatgpt as an educational tool: Opportunities, challenges, and recommendations for communication, business writing, and composition courses. Journal of Artificial Intelligence and Technology, 3(2), P. 60–68. https://doi.org/10.37965/jait.2023.0184
Alkaissi, H., & McFarlane, S. I. (2023). Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus, 15(2), P. 1-4. https://doi.org/10.7759/cureus.35179
AlMarwani, M. (2020). Academic Writing: Challenges and Potential Solutions. Arab World English Journal, 6, P. 114–121. https://doi.org/10.24093/awej/call6.8
Altunkaya, H., & Ayranci, B. (2020). The use of Edmodo in academic writing education. Dil ve Dilbilimi Çalışmaları Dergisi, 16(1), P. 89–103. https://doi.org/10.17263/jlls.712659
Amalia, S., Ramdhani, M. I., Apriani, E., & Boulahnane, S. (2024). The Effect of Learning Management System on Reading Comprehension Across 3 Types of Readers. LEARN Journal: Language Education and Acquisition Research Network, 17(1), P. 73-99. https://so04.tci-thaijo.org/index.php/LEARN/index
Apriani, E., Arsyad, S., Syafryadin, S., Supardan, D., Gusmuliana, P., & Santiana, S. (2022). ICT platforms for Indonesian EFL students viewed from gender during the COVID-19 pandemic. Studies in English Language and Education, 9(1), P. 187–202. https://doi.org/10.24815/siele.v9i1.21089
Apriani, E., Santiana, S., & Harmi, H. (2022). Investigating the role of ICT toward students’ higher order thinking in writing skills at Islamic university students. Al-Ishlah: Jurnal Pendidikan, 14(2), P. 2213–2224. https://doi.org/10.35445/alishlah.v14i2.2260
Apriani, E., Syafryadin, S., Inderawati, R., Arianti, A., Wati, S., Hakim, I. N., & Noermanzah, N. (2021). Implementing e-learning training toward English virtual lecturers: The process, perspectives, challenges and solutions. International Journal of Emerging Technologies in Learning (Online), 16(4), P. 240–255. https://doi.org/10.3991/ijet.v16i04.14125
Bagci Ayranci, B., & Mete, F. (2017). Evaluation of teacher candidates writing skills. Educational Research and Reviews, 12(24), P. 1253–1259. https://doi.org/10.5897/ERR2017.3430
Burkhard, M. (2022). Student perceptions of AI-powered writing tools: Towards individualized teaching strategies. 19th International Conference on Cognition and Exploratory Learning in Digital Age, p. 73–81. https://doi.org/10.33965/celda2022_202207l010
Chaves, A. P., & Gerosa, M. A. (2022). The impact of chatbot linguistic register on user perceptions: A replication study. In A. Følstad, T. Araujo, S. Papadopoulos, E. L.-C. Law, E. Luger, M. Goodwin, & P. B. Brandtzaeg (Eds.), Chatbot Research and Design (vol. 13171, pp. 143–159). Springer International Publishing. https://doi.org/10.1007/978-3-030-94890-0_9
Coniam, D. (2014). The linguistic accuracy of chatbots: usability from an ESL perspective. Text & Talk, 34(5), P. 545-567. https://doi.org/10.1515/text-2014-0018
Yu, C. H. (2008). Book Review: Creswell, J., & Plano Clark, V. (2007). Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage. Organizational Research Methods, 12(4), p. 801–804. https://doi.org/10.1177/1094428108318066
Dempere, J., Modugu, K., Hesham, A., & Ramasamy, L. K. (2023). The impact of ChatGPT on higher education. Frontiers in Education, 8, p. 1206936. https://doi.org/10.3389/feduc.2023.1206936
Dergaa, I., Chamari, K., Zmijewski, P., & Saad, H. B. (2023). From human writing to artificial intelligence generated text: Examining the prospects and potential threats of ChatGPT in academic writing. Biology of Sport, 40(2), p. 615–622. https://doi.org/10.5114/biolsport.2023.125623
Firat, M. (2023). What ChatGPT means for universities: Perceptions of scholars and students. Journal of Applied Learning and Teaching, 6(1), p. 57–63. https://doi.org/10.37074/jalt.2023.6.1.22
Frye, B. L. (2022). Should using an AI text generator to produce academic writing be plagiarism? Fordham Intellectual Property, Media and Entertainment Law Journal, Vol. 33, No. 4. P. 946-968.
Fuchs, K. (2022). The importance of competency development in higher education: Letting go of rote learning. Frontiers in Education, 7, p. 1004876. https://doi.org/10.3389/feduc.2022.1004876
Fuchs, K. (2023). Exploring the opportunities and challenges of NLP models in higher education: Is Chat GPT a blessing or a curse? Frontiers in Education, 8. https://doi.org/10.3389/feduc.2023.1166682
Fyfe, P. (2022). How to cheat on your final paper: Assigning AI for student writing. AI & Society, 38(4), P. 1395–1405. https://doi.org/10.1007/s00146-022-01397-z
Gocen, A., & Aydemir, F. (2020). Artificial Intelligence in Education and Schools. Research on Education and Media, 12(1), P. 13–21. https://doi.org/10.2478/rem-2020-0003
Gordijn, B., & Have, H. T. (2023). ChatGPT: Evolution or revolution? Medicine, Health Care and Philosophy, 26(1), P. 1–2. https://doi.org/10.1007/s11019-023-10136-0
Hsiao, J.-C., & Chang, J. S. (2023). Enhancing EFL reading and writing through AI-powered tools: Design, implementation, and evaluation of an online course. Interactive Learning Environments, P. 1–16. https://doi.org/10.1080/10494820.2023.2207187
Hutson, M. (2022). Could AI help you to write your next paper? Nature, 611(7934), P. 192–193. https://doi.org/10.1038/d41586-022-03479-w
Inderawati, R., Apriani, E., Jaya, H. P., Saputri, K., Wijayanti, E., & Muthmainnah, M. (2024). Promoting students’ writing by using essay writing GPT: A mix method. In Advanced Applications of Generative AI and Natural Language Processing Models (pp. 249–264). IGI Global. https://doi.org/10.4018/979-8-3693-0502-7.ch012
Hamp-Lyons, L. (1984). Book reviews : Jacobs, H.L., Zingraf, S.A. Wormuth, D.R., Hartfiel, V.F. and Hughey, J.B. 1981: Testing ESL composition: a practical approach. Rowley, Massachusetts: Newbury House. xii + 139 pp. ISBN 0-88377-225-6 (v.1). Language Testing, 1(2), P. 241–244. https://doi.org/10.1177/026553228400100210
Jiang, R. (2022). How does artificial intelligence empower EFL teaching and learning nowadays? A review on artificial intelligence in the EFL context. Frontiers in Psychology, 13, P. 1–8. https://doi.org/10.3389/fpsyg.2022.1049401
Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1(9), P. 389–399. https://doi.org/10.1038/s42256-019-0088-2
Joyce, A. (2023). Embracing AI: Using Cat-GPT to encourage classroom discussion. College Teaching, P. 1–3. https://doi.org/10.1080/87567555.2023.2251074
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., & Hüllermeier, E. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, P. 102274. https://doi.org/10.1016/j.lindif.2023.102274
King, M. R. (2023). The future of AI in medicine: A perspective from a chatbot. Annals of Biomedical Engineering, 51(2), P. 291–295. https://doi.org/10.1007/s10439-022-03121-w
Kirmani, A. R. (2023). Artificial intelligence-enabled science poetry. ACS Energy Letters, 8(1), P. 574–576. https://doi.org/10.1021/acsenergylett.2c02758
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health, 2(2), e0000198. P. 1-3 https://doi.org/10.1371/journal.pdig.0000198
La’biran, R., Garoga, M., Karuru, P., & Apriani, E. (2023). Development of listening and speaking textbook for general communication based on laboratory language operating software. Journal of English Education and Teaching, 7(2), P. 235–252. https://doi.org/10.33369/jeet.7.2.235-252
Liebrenz, M., Schleifer, R., Buadze, A., Bhugra, D., & Smith, A. (2023). Generating scholarly content with ChatGPT: Ethical challenges for medical publishing. The Lancet Digital Health, 5(3), P. 105–106. https://doi.org/10.1016/S2589-7500(23)00019-5
Lund, B. D., & Wang, T. (2023). Chatting about ChatGPT: How may AI and GPT impact academia and libraries? Library Hi Tech News, 40(3), P. 26–29. https://doi.org/10.1108/LHTN-01-2023-0009
Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023). ChatGPT and a new academic reality: Artificial Intelligence-written research papers and the ethics of the large language models in scholarly publishing. Journal of the Association for Information Science and Technology, 74(5), P. 570–581. https://doi.org/10.1002/asi.24750
Luo, Y. (2023). Understanding Chinese English-major students’ engagement with supervisory feedback on their undergraduate thesis: Insights from a multiple case study. Innovations in Education and Teaching International, 0(0), P. 1–13. https://doi.org/10.1080/14703297.2023.2238693
Macdonald, C., Adeloye, D., Sheikh, A., & Rudan, I. (2023). Can ChatGPT draft a research article? An example of population-level vaccine effectiveness analysis. Journal of Global Health, 13, P. 1–7. https://doi.org/10.7189%2Fjogh.13.01003
Maharrani, D., Saputri, K., Pratiwi, W. R., Apriani, E., Muthmainnah, M., Warsah, I., & Supardan, D. (2023). The era of e-learning and the challenges for rural areas. In Contemporary Challenges in Education: Digitalization, Methodology, and Management (pp. 224–240). IGI Global. https://doi.org/10.4018/979-8-3693-1826-3.ch017
Manohar, N., & Prasad, S. S. (2023). Use of ChatGPT in academic publishing: A rare case of seronegative systemic lupus erythematosus in a patient with HIV infection. Cureus, 15(2), P. 1-8. https://doi.org/10.7759/cureus.34616
McCarthy, J. (2022). Artificial intelligence, logic, and formalising common sense. In S. Carta (Ed.), Machine Learning and the City (1st ed., pp. 69–90). Wiley. https://doi.org/10.1002/9781119815075.ch6
Michel-Villarreal, R., Vilalta-Perdomo, E., Salinas-Navarro, D. E., Thierry-Aguilera, R., & Gerardou, F. S. (2023). Challenges and opportunities of generative AI for higher education as explained by ChatGPT. Education Sciences, 13(9), P. 856. https://doi.org/10.3390/educsci13090856
Molnár, G., & Szüts, Z. (2018). The role of chatbots in formal education. 2018 IEEE 16th International Symposium on Intelligent Systems and Informatics, P. 197–202. https://doi.org/10.1109/SISY.2018.8524609
Muthmainnah, M., Apriani, E., Seraj, P. M. I., Obaid, A. J., & Al Yakin, A. M. (2024). Nudging motivation to learn english through a ChatGPT smartphone-based hybrid model. In Advanced Applications of Generative AI and Natural Language Processing Models (pp. 26–42). IGI Global. https://doi.org/10.4018/979-8-3693-0502-7.ch002
Park, S., Thieme, A., Han, J., Lee, S., Rhee, W., & Suh, B. (2021, June). “I wrote as if I were telling a story to someone I knew.”: Designing Chatbot Interactions for Expressive Writing in Mental Health. In Proceedings of the 2021 ACM Designing Interactive Systems Conference (pp. 926-941). https://doi.org/10.1145/3461778.3462143
Rahman, M. M., Terano, H. J., Rahman, M. N., Salamzadeh, A., & Rahaman, M. S. (2023). ChatGPT and academic research: A review and recommendations based on practical examples. In M. Rahman, H.J.R. Terano, N. Rahman, A. Salamzadeh, & S. Rahaman, (2023). ChatGPT and academic research: A review and recommendations based on practical examples. Journal of Education, Management and Development Studies, 3(1), P. 1–12. https://doi.org/10.52631/jemds.v3i1.175
Rasul, T., Nair, S., Kalendra, D., Robin, M., de Oliveira Santini, F., Ladeira, W. J., Sun, M., Day, I., Rather, R. A., & Heathcote, L. (2023). The role of ChatGPT in higher education: Benefits, challenges, and future research directions. Journal of Applied Learning and Teaching, 6(1), P. 41-56. https://doi.org/10.37074/jalt.2023.6.1.29
Sallam, M. (2023). ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns. Healthcare, 11(21), P. 1–20. https://doi.org/10.3390/healthcare11060887
Shidiq, M. (2023, May). The use of artificial intelligence-based chat-gpt and its challenges for the world of education; from the viewpoint of the development of creative writing skills. In Proceeding of international conference on education, society and humanity (Vol. 1, No. 1, pp. 353-357).
Song, C., & Song, Y. (2023). Enhancing academic writing skills and motivation: Assessing the efficacy of ChatGPT in AI-assisted language learning for EFL students. Frontiers in Psychology, 14, P. 1260843. https://doi.org/10.3389/fpsyg.2023.1260843
Stokel-Walker, C. (2022). AI bot ChatGPT writes smart essays-should professors worry? Nature. https://doi.org/10.1038/d41586-022-04397-7
Taecharungroj, V. (2023). “What can ChatGPT do?” Analyzing early reactions to the innovative AI chatbot on Twitter. Big Data and Cognitive Computing, 7(1), P. 35. https://doi.org/10.3390/bdcc7010035
Taufik, M., Rijal, A. S., Dahniar, D., & Apriani, E. (2021). The effectiveness of learning English using LMS Google Classroom during covid-19 pandemic. Al-Ishlah: Jurnal Pendidikan, 13(2), P. 960–970. https://doi.org/10.35445/alishlah.v13i2.706
Thorp, H. H. (2023). ChatGPT is fun, but not an author. Science, 379(6630), P. 313–313. https://doi.org/10.1126/science.adg7879
Toba, R., & Noor, W. N. (2019). The current issues of Indonesian EFL students’ writing skills: Ability, problem, and reason in writing comparison and contrast essay. Dinamika Ilmu, 19(1), P. 57–73. https://doi.org/10.21093/di.v19i1.1506
Tuzlukova, V., Al Busaidi, S., Burns, S., & Bugon, G. (2018). EXPLORING TEACHERS’ PERCEPTIONS OF 21ST CENTURY SKILLS IN TEACHING AND LEARNING IN ENGLISH LANGUAGE CLASSROOMS IN OMAN’S HIGHER EDUCATION INSTITUTIONS. Journal of Teaching English for Specific and Academic Purposes, 6(1), P. 191. https://doi.org/10.22190/jtesap1801191t
Utami, D. M. A., Prihantoro, P., Apriani, E., Hidayah, J., & Handrianto, C. (2021). Empowering ICT potentials in English language teaching. Journal Polingua: Scientific Journal of Linguistics, Literature and Language Education, 10(2), 42–48.
Van Dis, E. A., Bollen, J., Zuidema, W., Van Rooij, R., & Bockting, C. L. (2023). ChatGPT: Five priorities for research. Nature, 614(7947), P. 224–226. https://doi.org/10.1038/ d41586-023-00288-7
Van Slyke, C., Johnson, R. D., & Sarabadani, J. (2023). Generative artificial intelligence in information systems education: Challenges, consequences, and responses. Communications of the Association for Information Systems, 53(1), P. 14. https://doi.org/10.17705/1CAIS.05301
Vanichvasin, P. (2021). Chatbot development as a digital learning tool to increase students’ research knowledge. International Education Studies, 14(2), P. 44–53. https://doi.org/10.5539/ies.v14n2p44
Vetter, M. A., Lucia, B., Jiang, J., & Othman, M. (2024). Towards a framework for local Interrogation of AI ethics: A case study on text generators, academic integrity, and composing with ChatGPT. Computers and Composition, 71, P. 1–12. https://doi.org/10.1016/j.compcom.2024.102831
Wilkenfeld, J. N., Yan, B., Huang, J., Luo, G., & Algas, K. (2022). “AI love you”: Linguistic convergence in human-chatbot relationship development. Academy of Management Proceedings, 2022(1), P. 17063. https://doi.org/10.5465/AMBPP.2022.17063abstract
Yan, D. (2023). Impact of ChatGPT on learners in a L2 writing practicum: An exploratory investigation. Education and Information Technologies, 28(11), P. 13943–13967. https://doi.org/10.1007/s10639-023-11742-4
Yu, S., & Liu, C. (2021). Improving student feedback literacy in academic writing: An evidence-based framework. Assessing Writing, P. 48. https://doi.org/10.1016/j.asw.2021.100525
Copyright (c) 2025 National Research University Higher School of Economics
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the Copyright Notice.
