GPTBot Development for Translation Purposes: Flowchart, Practical Case and Future Prospects
Abstract
Background: This paper explores the development and evaluation of a GPTBot tailored for institutional translation tasks. It addresses a gap in applied research on how generative AI can be adapted for domain-specific translation workflows, particularly in academic institutions.
Purpose: To design and implement UGRBot, a chatbot based on ChatGPT-4 that supports the translation of institutional texts at the University of Granada (UGR) while also outlining a structured and replicable methodology for creating specialised chatbots to enhance translation processes.
Method: The methodology includes: (1) chatbot development using a knowledge base of 57 bilingual institutional documents; (2) evaluation of output quality using BLEU scores, comparing UGRBot with DeepL and Google Translate; and (3) a focused assessment on the translation of 100 institutional terms.
Results: A reference corpus in English of 14,521 words was compiled from UGR administrative and regulatory documents, with human translations serving as the benchmark. BLEU scores were computed using the Natural Language Toolkit library in Python, employing 4-gram analysis for full-text evaluation and bigram analysis for terminology translation.
Conclusion: Results show that UGRBot outperformed both baseline systems in the translation of specialised institutional terminology, achieving the highest BLEU score in this area. However, limitations include lower performance across full-length texts. In conclusion, this research documents the development of a domain-specific GPTBot and its implementation in an institutional context, offering a transferable framework for integrating generative AI into specialised translation workflows.
Downloads
References
Callison-Burch, C., Osborne, M., & Koehn, P. (2006). Re-evaluating the Role of BLEU in machine translation research. Journal of Artificial Intelligence Research, 25, 191–218.
Curry, N., Baker, P., & Brookes, G. (2024). Generative AI for corpus approaches to discourse studies: A critical evaluation of ChatGPT. Applied Corpus Linguistics, 4(1). https://doi.org/10.1016/j.acorp.2023.100082
Dimitrova, B. E. (2010). Translation process. In Y. Gambier & L. van Doorslaer (Eds.), Handbook of translation studies (1st ed., vol. 1, pp. 406–411). John Benjamins Publishing Company.
Gao, R., Lin, Y., Zhao, N., & Cai, Z. G. (2024). Machine translation of Chinese classical poetry: A comparison among ChatGPT, Google Translate, and DeepL Translator. Humanities and Social Sciences Communications, 11. https://doi.org/10.1057/s41599-024-03363-0
Gao, Y., Wang, R., & Hou, F. (2023). How to design translation prompts for ChatGPT: An empirical study. arXiv:2304.02182. http://arxiv.org/abs/2304.02182
Ghassemiazghandi, M. (2024). An evaluation of ChatGPT’s translation accuracy using BLEU Score. Theory and Practice in Language Studies, 14(4), 985–994. https://doi.org/10.17507/tpls.1404.07
Hansen, G. (2013). Translation process as object of research. In C. Millán & F. Bartrina (Eds.), The Routledge Handbook of Translation Studies (1st ed., pp. 88–101). Routledge.
Hassani, H., & Silva, E. S. (2023). The role of ChatGPT in data science: How AI-assisted conversational interfaces are revolutionizing the field. Big Data and Cognitive Computing, 7(2), 62. https://doi.org/10.3390/bdcc7020062
Jiao, W., Huang, J., Wang, W., He, Z., Liang, T., Wang, X., Shi, S., & Tu, Z. (2023). ParroT: Translating during chat using large language models tuned with human translation and feedback. arXiv:2304.02426. http://arxiv.org/abs/2304.02426
Jiao, W., Wang, W., Huang, J., Wang, X., Shi, S., & Tu, Z. (2023). Is ChatGPT a good translator? Yes with GPT-4 as the engine. arXiv:2301.08745. http://arxiv.org/abs/2301.08745
Läubli, S., Sennrich, R., & Volk, M. (2018). Has machine translation achieved human parity? A case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 4791–4796). Association for Computational Linguistics. https://doi.org/10.18653/v1/D18-1512
Lilli, S. (2023). ChatGPT-4 and Italian dialects: Assessing linguistic competence. Umanistica Digitale, (16), 235–263. https://doi.org/10.6092/issn.2532-8816/18221
Mohsen, M. (2024). Artificial intelligence in academic translation: A comparative study of large language models and Google Translate. Psycholinguistics, 35(2), 134–156. https://doi.org/10.31470/2309-1797-2024-35-2-134-156
Mossop, B. (2000). The workplace procedures of professional translators. In A. Chesterman, N. Gallardo-San Salvador, & Y. Gambier (Eds.), Translation in Context: Selected papers from the EST Congress, Granada 1998 (pp. 39-48). John Benjamins Publishing Company. https://doi.org/10.1075/btl.39.07mos
Ngo Cong-Lem, S., Soyoof, A., & Tsering, D. (2024). A systematic review of the limitations and associated opportunities of ChatGPT. International Journal of Human–Computer Interaction, 40(6), 515–537. https://doi.org/10.1080/10447318.2024.2344142
Nord, C. (1991). Text analysis in Translation: Theory, Methodology, and Didactic Application of a Model for Translation-oriented Text Analysis (1st ed.). Rodopi.
OpenAI. (2023). GPT-4 technical report. http://arxiv.org/abs/2303.08774
Papineni, K., S. Roukos, T. Ward and W. Zhu. (2002). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (pp. 311–318). ACM. https://doi.org/10.3115/1073083.1073135
Parra-Galiano, S. (2006). La revisión y otros procedimientos para el aseguramiento de la calidad de la traducción en el ámbito profesional [Revision and other procedures for ensuring translation quality in the professional field]. Revue de Traduction et d’Interprétation – Journal of Translation Studies, 15(2), 11–48. http://hdl.handle.net/10481/7369
Rodríguez-de Céspedes, B. (2020). Beyond the margins of academic education: Identifying translation industry training practices through action research. Translation & Interpreting, 12(1), 115-126. https://doi.org/10.12807/ti.112201.2020.a07
Sahari, Y., Al-Kadi, A. M. T., & Ali, J. K. M. (2023). A cross sectional study of ChatGPT in translation: Magnitude of use, attitudes, and uncertainties. Journal of Psycholinguistic Research, 52(6), 2937–2954. https://doi.org/10.1007/s10936-023-10031-y
Sánchez-Gijón, P. (2022). What experts say about increasingly relevant translation technologies. Tradumàtica. Tecnologies De La Traducció, 20, 295–301. https://doi.org/10.5565/rev/tradumatica.322
Schneider, B. (2022). Multilingualism and AI: The regimentation of language in the age of digital capitalism. Signs and Society, 10(3), 362–387. https://doi.org/10.1086/721757
Siu, S. C. (2023a). ChatGPT and GPT-4 for professional translators: Exploring the potential of large language models in translation. Social Science Research Network, 1-36. http://dx.doi.org/10.2139/ssrn.4448091
Siu, S. C. (2023b). Revolutionizing translation with AI: Unravelling neural machine translation and generative pre-trained large language models. Social Science Research Network, 1-19. http://dx.doi.org/10.2139/ssrn.4499768
Vela-Valido, J. (2021). Translation quality management in the AI Age. New technologies to perform translation quality assurance operations. Tradumàtica. Tecnologies De La Traducció, 19, 93–111. https://doi.org/10.5565/rev/tradumatica.285
Yamada, M. (2023). Optimizing machine translation through prompt engineering: An investigation into ChatGPT’s customizability. arXiv. https://doi.org/10.48550/arXiv.2308.01391
Zappavigna, M. (2023). Hack your corpus analysis: How AI can assist corpus linguists deal with messy social media data. Applied Corpus Linguistics, 3(3). https://doi.org/10.1016/j.acorp.2023.100067
Zhao, B., Jin, W., Del Ser, J., & Yang, G. (2023). ChatAgri: Exploring potentials of ChatGPT on cross-linguistic agricultural text classification. Neurocomputing, 557, 126708. https://doi.org/10.1016/j.neucom.2023.126708
Copyright (c) 2025 National Research University Higher School of Economics
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the Copyright Notice.
