The Language of Praise in Russian Students’ Evaluation of Teaching

Recent decades have seen a dramatic rise in student evaluation of teaching (SET). However, they have overwhelmingly focused on quantitative ratings, neglecting students’ written feedback. This study addresses the lack of qualitative research on SET by applying a semantic theory and computational methods for analysing the language of positive feedback comments provided by students of the Higher School of Economics (HSE) in Russia. Analysing a corpus of student commentary about teaching also contributes to the theory of pragmatics as the approach to analysing qualitative evaluations of teaching is based on the premise that students’ positive feedback can be treated as a sort of the compliment/praise speech act reflecting cultural specificity. Our findings showed that quantitatively the most common semantic pattern used by HSE students is ACTOR + (AUGMENTOR) EVALUATOR + PHYSICAL/MENTAL ACTION PERFORMED BY THE ACTOR + (AUGMENTOR) EVALUATOR. Thus, HSE students tend to praise the teacher more often than the other components of the teaching process and the teacher’s behaviour, thoughts, and feelings are viewed as more important than skills and speech.


Introduction
Student evaluation of teaching (SET) is a common assessment tool used to measure teaching effectiveness and evaluate courses at colleges and universities. Studies of SET in higher education have dramatically risen over the past several decades; however they focus on quantitative ratings, assessing the reliability and validity of these measures, or improving their design (Cheng & Marsh, 2010;Richardson, Slater, & Wilson, 2007;Spooren, Brockx, & Mortelmanset, 2013). Apart from measurement scales in SET, students are invited to write open comments. Nonetheless, they are generally not taken into account, owing to the lack and challenges of systematic analysis and processing techniques for subjective commentary despite the fact that open-ended questions are more appropriate for formative evaluation (Alhija & Fresko 2009;Huxham et al., 2008). Thus, in spite of the growing body of literature on SET, student qualitative evaluations have been under-researched and there is a lack of theoretically grounded analysis of the language of SET. Besides, in the literature, more attention is given to negative commentary or criticism and their effects on teachers while students' positive comments are particularly more frequent than negative ones and generously praise teachers (Alhija & Fresko 2009). Students seem to regard qualitative feedback as more meaningful than quantitative feedback (Davison & Price 2009).
As Grainger (2018) showed, Brown and Levinson's theory of politeness (Brown & Levinson, 1987) and Gricean pragmatics (Grice, 1975) can be useful for the analysis of different types of interaction. We suggest that the study of student positive commentary about teaching may also benefit from these theories using some concepts of traditional pragmatics (Austin, 1962;Leech, 1983;Searle, 1975). We base this on the premise that students' positive evaluation of teaching can be treated as praise or compliment speech acts (Holmes, 1988) reflecting cultural values (Herbert, 1986;Manes, 1983). It is claimed that despite being semantic universals, positive student feedback varies from one language to another, from culture to culture, and from society to society (Wierzbicka, 1991;Wolfson & Manes, 1980). Thus, the study of the language of praise in student evaluation of teaching may fill a gap in the speech act theory.
Despite extensive research of compliments across cultures (Herbert, 1989;Holmes, 1988;Lin, Woodfield, & Ren, 2012;Maíz-Arévalo, 2012;Seifoori & Emadi, 2015;Tang & Zhang, 2009) and in the foreign/second language classroom (Allami & Montazeri, 2012;Bai, 2015;Holmes & Brown, 1987;Huth, 2006), there has been little attention paid to complimenting and praise between teachers and students. A few studies that investigated this issue either suggested guidelines for effective teacher praise (Brophy, 1981) or focused on praise used by teachers to soften criticisms (Hyland & Hyland, 2001). Burnett (2002) also looked into the relationships between teacher praise and students' perceptions of the classroom. Suketi (2014) analyzed students' response to the compliment given in consultation. No literature on student complimenting or praising behaviors toward teachers was found. In this study, we investigate student complimenting and praising behaviour in the context of teacher evaluation.
Students' evaluation of teachers, particularly in higher education, has become a widely discussed subject. There were attempts to determine the relationships between SET scores and such factors as students' gender (Amin, 1994;Centra & Gaubatz, 2000;Thawabieh, 2017), expected grades (Chen, Wang, & Yang, 2017;Sarwar, Dildar, & Hussain, 2017) or overall satisfaction with the course (Ahmad, 2015). Yet very few studies have systematically investigated the content of students' written feedback while students were found to be generally willing to provide such feedback (Spencer & Schmelkin, 2002), and it be both negative and demotivating as well as positive and encouraging (Light & Cox, 2001).
There are a small number of studies related to the content of student feedback (Blair & Noel, 2014;Steyn, Davies, & Sambo, 2018), but these works restrict themselves to conventional thematic analysis eliciting factors and themes of student qualitative feedback. "Attention to the … language used has potential to reveal insight into students' relationships with teachers and the institution" (Stewart, 2015, p. 1-2). Nevertheless, only scant research exists on the language of students' feedback. For instance, Zaitseva, Milsom, and Stewart (2013) interpreted the qualitative data from student satisfaction surveys utilizing concept mapping software. Stewart (2015) analysed the extent to which students managed language through intensification or moderation of views and highlighted the asymmetry between positive and negative feedback. Rajput, Rajput, Haider, and Ghani (2016) presented a lexicon-based sentiment analysis of students' open-ended textual feedback using word clouds visualization techniques. Our study seeks to expand the linguistic perspective of SET as, being a window into the mind, language is one of the best ways to provide insight into underlying concepts and inform on academic and interpersonal experience.

Natural Semantic Metalanguage Theory
Linguists point out that languages are culture-specific (Humboldt, 1999;Sapir, 1929;Whorf, 1982). However, it is important to find common ground for making reliable comparison between languages, the so-called "true tertium comparationis" (Wierzbicka, 1996, p. 16). Wierzbicka's (1996) natural semantic metalanguage theory has become a noteworthy approach with considerable longevity and "a substantial output within the field of contemporary linguistics" (Goddard, 2008, p. 1).
In Wierzbicka's understanding, there are indefinable elements or absolute particles of meaning in all languages, "that are indefinable not because they are considered elementary in a particular language but in any kind of human language system" (Dziedziul, 2017, p. 256). Wierzbicka (1996) and Goddard (2008) suggest using this semantic cognitive core as a metalanguage that will allow us to compare different languages and cultures. Nevertheless, Wierzbicka (1996) points out that the natural semantic metalanguage does not imply that all languages are equivalent. The theory of elementary semantic units remains culture sensitive. It assumes the presence of words in a language, that have no direct equivalents in others, and that the superficial unique elements can be compared by extracting the elementary particles of meaning shared by all languages. "In plain words: it is not the meaning that is unique, but the sequence of universal elements that stand behind the word" (Dziedziul, 2017, p. 257) The palette of the elementary units of meaning has been changing. However, despite wild fluctuations, their number has not exceeded 100 items (Tabakowska, 2001). Wierzbicka (1996) outlined the following elementary units of meaning (See Table 1). The present study aims to address the shortcomings of qualitative research on SET by applying a semantic theory and corpus-driven methodology. The research question to be answered is: What semantic patterns are commonly used by HSE students in their positive written comments about teaching?

Methodology Participants
This study describes semantic patterns used by students of the Higher School of Economics in Russia when presenting praise in an institutional satisfaction survey.
In our study, the data analysed were drawn from Teaching Quality Assessment (TQA), a tool available on the Higher School of Economics (HSE) website. TQA is conducted at the end of each module in order to improve the planning and organisation of the teaching process. Undergraduate students across Year 1, 2, 3, and 4 are offered the opportunity to assess courses in which they had an examination in the current module or received a grade as part of an interim assessment. The name of the course and the teacher who taught it are reflected in the evaluation form. Students are invited to assess different aspects of the courses and the teachers' performance on a 5-point scale as well as to leave free-form written comments in a separate field, with the direct wording of the instruction: "Your comments: ___".
We obtained ethical approval from the HSE's ethics committee with the stipulation that we not use any direct quotes from the respondents. We retrieved and analyzed 363 anonymous positive comments about teaching from first through fourth-year students across six Bachelor's programmes (Business Informatics, Economics, History, Law, Management, and Software Engineering) for the academic years 2015/2016 and 2016/2017. We differentiated between positive and negative feedback using a teacher's overall score. If a teacher received a positive score (4 and 5), the following comment was qualified as positive while the comments following the teacher rating of 3 or below were negative. It is acknowledged from the outset that the dataset investigated here may not be substantial enough to generalize findings beyond the sample. However, as this was not the required purpose, the dataset is useful for exploring semantic models of the positive feedback language used by HSE students in Perm.

Assessments and Measures
The analysis of students' positive feedback about teaching was carried out in several steps: data preprocessing (tokenization, punctuation and stop word removal, and lemmatization), part-of-speech tagging (POS tagging), counting the number of occurrences of different parts of speech, calculating word occurrence and frequency, extracting semantic concepts from the notional words, and building semantic patterns of HSE students' positive feedback about teaching (See Figure 1). We refer to the sequences of semantic concepts that represent entities appearing in SET as semantic patterns for simplicity.

Figure 1 Student positive feedback analysis
For the data analysis we applied Pymorphy2, a morphological analyser and generator for the Russian and Ukrainian languages. Pymorphy2 was implemented as cross-platform software in the Python programming language and is widely used in academic research. Pymorphy2 uses morphological word features and relies on large lexicons for the analysis of common words. For the Russian language, Pymorphy2 uses the OpenCorpora dictionary, available at OpenCorpora.org (∼ 5 * 106 word forms, ∼ 0.39 * 106 lemmas) and converted from OpenCorpora XML6 format to a compact representation optimized for morphological analysis and generation tasks. Pymorphy2 provides users with methods to analyze, inflect, lemmatize, or tag words as well as with data about the word gender, number, part of speech, and root. For data processing, it is necessary to choose the correct analysis from a list of possible analyses. It is generally required to take the word context in account.
The retrieved comments were categorised as unstructured and noisy owing to the informal writing style, grammatical and spelling mistakes, use of abbreviated shortcuts, and incomplete sentences. Therefore, we preprocessed the data removing unwanted and noisy data. This stage of analysis comprised tokenization (breaking a stream of text into a list of words), punctuation and stop words removal, and lemmatization (grouping together the inflected forms of a word and analysing them as a single item). The tokenizer split the input text into 5448 words. After that, we filtered function words and lemmatised the remaining 4193 notional words. The resulting data set was a list of 1294 lemmas serving as input for further processing steps.
Next was POS tagging. We marked up each lemma as corresponding to a particular part of speech and counted the number of occurrences for each part of speech elucidating their usage patterns. In our research, we also analysed word frequency by counting how many times each word was used in students' positive feedback comments. Finally, there was semantic analysis dividing the notional words into semantic categories and yielding semantic patterns of students' positive evaluation of teaching.

Results
As the first step of data processing, the morphological analyzer Pymorphy2 performed POS tagging marking up all the notional words as a noun, pronoun, adjective, verb, adverb, or numeral (See Figure 2).
Next, we computed the number of occurrences for each part of speech (See Table 2). As shown in Table 2, students' positive commentary about teaching was dominated by verbs, accounting for more than a half of all words in the dataset. Next, frequent were nouns (24%) and adjectives (12%), with adverbs, pronouns, and numerals lagging behind.
Further investigation narrowed down to the computation of word frequencies for each part of speech (See Tables 3-7). Next was semantic categorization of the notional words in positive SET. For this the semantic metalanguage particles, which either coincided with or helped create compound concepts of student positive comments, were used. According to the natural semantic metalanguage theory, substantives are universally divided into persons (Who is it?) and things (What is it?). Furthermore, there are categories describing those who act (Actor), the means by which they act (Tool), the objects they make or use (Artifact), and the properties of actors, tools, and artifacts (Property). Table 4 shows the semantic categories of the nouns used in students' positive feedback comments. The adjectives were classified into two big groups: those attributing people and things. Each of the two groups consists of three semantic categories: determiners serving to express the reference of a noun, descriptors describing or identifying a person or thing, and evaluators giving an opinion about a person or thing. Table 5 shows different trends for attributing animate and inanimate nouns in positive SET. Nouns denoting people are overwhelmingly attributed by evaluators while those for things are more commonly used with descriptors.

Example 4
Danny used for a person/object is referred to now the nearest in space/time used to differentiate between people/things Danny is a DETERMINER

Example 5
Prekrasny someone thinks about something that something is very good Prekrasny is an EVALUATOR

Example 6
Uchebny someone says about something that something is used for learning someone says how something is used someone describes the purpose of something Uchebny is a DESCRIPTOR Table 6 presents the semantic types of the verbs used in students' positive comments about teaching.  (Table 7) shows the semantic categories of the adverbs used by students to praise teachers. As seen in Table 7, students tended to intensify their positive opinion about teaching using augmentors.

Discussion
Our main finding is that students praised the teacher more often than the other components of the teaching process (e.g. means of teaching, teaching results). It aligns with Stewart's (2015) research and proves that an effective teacher plays a very important role in student achievement.
With reference to Wolfson (1980), our findings revealed that students tended to use more verbs than other notional parts of speech to compliment and praise teachers. To summarize the parts-of-speech patterns in Table 3, five semantic groups of verbs were found in students' positive feedback comments: actions/events, mental predicates, non-mental predicates, metapredicates, and speech verbs. Interestingly, the figures for actions, mental, and non-mental predicates were almost equal and considerably exceeded those for speech and metapredicate groups. This means that students value teachers for what they do, what and how they think, and how well they conduct a seminar or lecture rather than how they speak or what their personal expertise and skills are.
Both animate and inanimate nouns were attributed by adjectives of three semantic types: determiner, descriptor, and evaluator. The analysis showed that evaluators tended to attribute nouns denoting people while descriptors overwhelmingly referred to things. Evaluators were also numerous among adverbs. Quantitatively, the most common semantic models of students' positive evaluation of teachers are as follows:

Actor + Evaluator + Action/Mental predicate + Augmentor/ Evaluator Tool/ Artifact + Descriptor + Non-mental predicate + Augmentor/Space/Time
Although compliments are thought to be formulaic and lack originality in choice of lexical items and semantic structures (Manes & Wolfson, 1981;Wolfson, 1981), our results showed some unique characteristics of Russian compliments in the higher education context. One major difference was noted when examining the use of parts of speech. As mentioned above, we found that student positive evaluations of teachers in Russian were dominated by verbs, whilst previous research pointed out that the majority of Russian compliments were adjectival (Nguyen, 2013;Solodka, Sukhomlynsky, & Perea, 2018). Another difference that was noted in complimenting behaviour in the higher education context concerns thematic imperatives or topics chosen by Russian students to compliment teachers. Russian students tend to praise teachers for their actions (physical, mental, and non-mental) rather than their appearance, abilities, and personality traits occurring most in noneducational contexts (Issers, 2006;Kachevskaia, 2007;Shcheboleva & Sun', 2016).
The study of verbal representations of the complimenting speech act used by students of the Higher School of Economics in Russia offers insights into the system of values of the Russian student community. The prevalence of mental and non-mental verbs, expressing one's state (to be, to have, to like, to think, to understand, to want, etc.), represents Russian students' orientation value of "being" and "feeling". The orientation value of "doing" represented by action/event verbs (to do, to get, to give, to help, etc.) is less important for Russian students. Surprisingly, teaching delivery appears to be ranked the lowest in the verb classes. Russian students' complimenting behaviour is most often directed at interest, which manifests itself in the use of such lexemes as to be interested, interesting, and interestingly/with interest. These results compliment the study on the linkage between the teachers' emotional characteristics and personality type and students' estimation of the teachers' qualities (Busygina & Busygina, 2013).
However, this research has several limitations. It is based on feedback provided by students in just one campus of the Higher School of Economics at one point in time. So, no claims can be made that the obtained semantic patterns will be generalizable to all students' positive feedback, which was never the intention of this research. Different findings could be expected between different kinds of institutions and different cultures and languages, especially in weights of semantic groups in models. Nevertheless, addressing this limitation in our methodology may be an objective of future research. The study could also be extended to word collocations or sentence structures in student feedback and include age, gender, and nationality in the research. Moreover, the limited sample could not include negative comments, which might also contain the language of praise. It means that further studies should incorporate negative evaluations, to strengthen the validity of the results. Overall, the validity of quantitative ratings in SET and the improved design of evaluation measures can be tested in combination with the current findings and potential comparisons with negative comments.

Conclusion
Our study offers a new perspective on the theory of speech acts and reveals some patterns of interpersonal positioning in a single aggregated sample of student positive commentary. The findings provide information that concerns how expressive illocutionary acts are performed in the higher education context as well as how their implicit cultural meanings can be interpreted. It is expected that some aspects of students' evaluative language identified in this research may be universal.
The semantic models and their features can be applied in automatic reading of subjective commentaries to identify positive feedback and 'good' teachers. This application can fill the gap caused by the lack of systematic analysis and processing techniques for approaching open commentaries in students' evaluations. Because this study is on text analytics in semantic research, i.e. the application of the Python programming language and the OpenCorpora dictionary, its main contribution is processing a large number of user-generated texts as well as building semantic patterns of student positive commentary in the national language.
We also anticipate that a better understanding of the semantic structure and proposition of positive textual evaluation will allow educators to interpret students' subjective commentary more fully and improve both student learning and instructor performance, mainly through using them as guidance for the choice of instructors, knowing factors for faculty loads or rating biases, and students' expectations (Otto, Sanford, & Ross, 2008;Wongsurawat, 2011). Satisfaction with a teacher is assumed to facilitate learning and to positively impact learning outcomes. Particular recommendations for educational managers might include a better match of a teacher for a course and/or improvements in course selection by students.