Cognitive Load Approach to Digital Comics Creation: A Student-Centered Learning Case

Abstract

Featured Application

The present work has applications in the field of primary and secondary education. The work describes how educators can take advantage of digital comics creation for the learning of applied science in school. The study has implications not only for educators of applied sciences but also those of other educational disciplines. The study also outlines directions for future research to further clarify the appropriate instructional approach that could render digital comics an effective educational method.

Abstract

The use of comics and their creation is an especially promising tool to enable students to construct new knowledge. Comics have already been adopted in many applied sciences disciplines, as the combination of text and images has been recognized as a powerful learning tool. Educational activities and tools, however, must not create an overload on students’ working memory that could hinder learning. In the current study, we investigated, through pre-test and post-test performance, the effect of digital comics creation on students’ efforts to construct new knowledge. Furthermore, through the multidimensional NASA-TLX, we assessed the cognitive load imposed on students. The results were in favor of digital comics creation, ranking it as an efficient instructional activity. Specifically, the students’ performance after digital comics creation improved and the imposed load on students was normal. Also, studying the weighing procedure between the NASA-TLX dimensions, frustration and temporal demand were found to be the most aggravating dimensions. Finally, implications for teachers and future research recommendations are discussed.

1. Introduction

New technologies offer the contemporary classroom the possibility of using educational multimedia, which include combinations of at least two different media types, such as text (written or spoken), pictures, video, and animations [1]. The presentation of educational material in the form of multimedia is based on the fact that learning with words and images is more effective than learning with words or images alone [2]. However, instructional multimedia design might suffer from several conditions that potentially hinder students’ knowledge construction. These conditions arise from Cognitive Load Theory (CLT) and the Cognitive Theory of Multimedia Learning (CTML). Learning is an active process of knowledge construction in which instruction aims at guiding learners’ cognitive processes [3]. Sweller’s CLT describes the mental workload (MWL) as a result of the number of informational units that must be held in the working memory, which is limited in capacity [4]. Mayer’s CTML provides a useful means by which instructional efficiency can be analyzed because it sets out the principles that should govern educational multimedia in order to support learning. In this context, effectiveness is related to fostering learning, that is, the construction of knowledge [5]. Learning occurs when working memory successfully processes information, leading to new schema creation. The control of cognitive load is important for meaningful learning, and consequently, CLT and CMTL contribute to the generation of a suitable instructional design [6]. Multimedia designers, according to CLT and CMTL, should produce instructional materials, taking care not to overload the working memory [7].

A key element in the above theories is the measurement of the MWL imposed on learners by the educational activities they are asked to carry out. Reliable assessment of a learner’s MWL would enable a range of new and improved instructional activities [8]. The NASA-TLX, created by Hart and Staveland [9], is a widely used and reliable method to assess MWL [10,11]. The NASA-TLX is a subjective, self-reported, multidimensional method that assesses the MWL across six dimensions. A digital learning activity, with its specific technology, influences students’ cognitive load and is worthy of investigation under the theories of CLT and CMTL.

Research has shown the effectiveness of text and picture integration in school activities [12]. This integration is used, for example, in science textbooks that use different types of visual representations to support information reporting and explain content [13,14]. Similarly, Lee [15] noted that visual representations can aid the understanding of scientific ideas; thus, they are included in modern science textbooks. Mainali [16] also stated that most textbooks today, in order to promote understanding of mathematics, use a wide variety of diagrams and pictures.

Among different types of text and picture integration, comics have gained researchers’ interest. Visual narratives existing in comics present information in a sequence of images and are often combined with written text. Similarities in brain responses have been found in the processing of visual narrative sequences and sentences [17]. Hughes et al. [18] stated that comics, with their power of visual communication and narrative dialogue, can help students to deconstruct and reconstruct meaning. Comics creation, additionally, provides students with a popular and accessible medium to communicate their knowledge. Moreover, digital comics creation is based on computers, which appeal to students. Comics have been used as instructional tools in various disciplines, such as chemistry, computer science, biology, physics, nanotechnology, and programming, satisfying students and deepening their engagement [19,20,21,22]. Comics provide students with an easy way to access information and can motivate them [23], while comics creation maintains this motivation for a longer period [19]. Comics also enhance students’ active participation in classroom educational tasks and promote a positive attitude toward the learning process [24].

More research has focused on how comics might be used in a classroom than on comics creation. Students prefer creating digital comics in the classroom [25], but a significant aspect of using comics creation is whether it can benefit students in their knowledge construction. In addition, as we mentioned above, the MWL imposed on students by an educational activity is a decisive factor in students’ effort. Our research objectives included:

• The investigation of the effect of digital comics creation on students’ effort in knowledge construction and on the imposed MWL. We used the NASA-TLX method, which is acknowledged as a reliable method of assessing MWL [10,26].

• The examination of whether comics creation by students can lead to the satisfaction of the principles of effective design of artifacts, which combine text and pictures, and support learning.

The manuscript is structured as follows. The next two sections describe the theoretical framework and the research hypothesis that was investigated in the current study. The two subsequent sections describe the materials and the research methodology followed and the corresponding results. The last two sections discuss the findings of our research, its limitations, and some proposals for future work and finally present our conclusions.

2. Cognitive Load Theory, Cognitive Theory of Multimedia Learning, and Cognitive Load Assessment

Cognitive Load Theory is an educational theory that provides a conceptual model for the kind of cognitive processes that take place during instruction [3]. CLT explains how students’ ability to process new information and construct knowledge in long-term memory can be affected by the information processing load caused by the learning tasks [27].

CLT is concerned with the limitations of human cognitive processing in relation to learning [28] and plays a key role in connecting cognitive science and instructional practice. CLT assumes that the human cognitive architecture consists of a working memory that is limited in capacity and includes partially independent subcomponents to deal with auditory/verbal and visual information [28]. CLT argues that any learning task imposes a cognitive load on working memory and identifies different components of mental resources that compete for students’ working memory capacity [29]. This load depends on the characteristics of the learning task, the way the learning material is presented or the type of instructional activities that the learner participates in, and the process in which the learner relates relevant information from their long-term memory to information from the present learning task [30,31]. Hence, some learning tasks impose a higher cognitive load on working memory than others. Since there is a limit to working memory capacity, instructional design should intend to reduce the unnecessary working memory load in order to free capacity for learning-related processing, thereby enabling the transformation of currently attended information into long-term memory traces [32]. Conversely, if the cognitive load exceeds the working memory capacity, learning could be hampered [27,31,33].

Therefore, learning may occur when the cognitive load that is associated with an instructional design does not exceed the available processing capacity of the working memory resources. Learning materials have an inherent complexity that stems from the number of information units together with the number of connections between them [32]. Consequently, a balance must be found between the elements that comprise the cognitive load. This means that the load must be managed appropriately so that the simultaneous processing of all information elements leaves some spare cognitive capacity, allowing learners to invest available processing resources in schema acquisition and automation [6,28].

CLT has influenced CTML, which focuses on how to structure multimedia instructional messages and apply more effective cognitive strategies, aiming to help people to learn efficiently [5,34]. Mayer [2] defined a multimedia instructional message as communication containing words and pictures that intends to foster learning. The words can be spoken or written, and the pictures can correspond to any form of graphical imagery.

CTML is based on three assumptions, namely, the dual channel assumption, limited capacity assumption, and active processing assumption. According to these assumptions, humans have separate but interconnected channels with a limited capacity for processing auditory and visual information, and they construct knowledge by selecting appropriate incoming information, organizing it into a coherent mental representation, and integrating it with prior knowledge activated from the long-term memory. CTML considers three kinds of demands on students’ information processing system during learning: extraneous processing, which is caused by poor instructional design and impedes learning; essential processing, which concerns the essential information presented; and generative processing, which aims to make sense of the material presented [2,34].

Multimedia learning research has provided and evidenced several principles for the effective design of multimedia instruction [2,3,7,12,29,31]. Mayer [2], with the aim of reducing extraneous processing, managing essential processing, and fostering generative processing, formulated some multimedia instructional principles to be followed when multimedia instructional messages are created. Specifically, the aim of reducing extraneous processing involves the coherence principle, the signaling principle, the redundancy principle, the spatial contiguity principle, and the temporal contiguity principle. The aim of managing essential processing involves the segmenting principle, the pre-training principle, and the modality principle. Finally, the aim of fostering generative processing involves the multimedia principle, the personalization principle, and the voice principle. These principles state that people learn better (i) when extraneous material is excluded rather than included (coherence principle), (ii) when cues that highlight the organization of the essential material are added (signaling principle), (iii) when graphics and narration are used together rather than graphics, narration and printed text together (redundancy principle), (iv) when corresponding words and pictures are placed near each other rather than far from each other on the page or screen (spatial contiguity principle), (v) when corresponding words and pictures are presented at the same time (temporal contiguity principle), (vi) when a multimedia lesson is presented in learner-paced segments rather than as a continuous unit (segmenting principle), (vii) when they know the names and characteristics of key components in a multimedia message (pre-training principle), (viii) when graphics and narration are used rather than graphics and printed text (modality principle), (ix) when words and pictures are used rather than words alone (multimedia principle), (x) when the words in a multimedia presentation are in a conversational style rather than in formal style (personalization principle), and (xi) when the words in a multimedia message are spoken by a standard-accented human voice rather than a machine voice (voice principle).

Digital technology offers new opportunities for learning, but several design factors involved in digital learning can expose students to a higher cognitive load [29]. Therefore, in an educational environment, managing the cognitive load can help improve learning outcomes. Cognitive load can be measured by self-report measures. The NASA-TLX, developed by Hart and Staveland [9], is one of the most widely used measures of cognitive workload and has been proven to be a reliable and effective tool for evaluating the cognitive load in different fields of knowledge [10,11,26,35,36,37]. The NASA-TLX, being more sensitive than other measures [38], can capture the multidimensionality of cognitive load, assessing it in six dimensions [39]. Three of these analyze dimensions of requirements concerning individuals, specifically, the mental demand, which concerns the mental and perceptual activity; the physical demand, which concerns the amount of physical effort; and the temporal demand, which is associated with the perception of time. The other three analyze dimensions that are related to the willingness of individuals, specifically, performance, which concerns the degree of goal accomplishment; effort, which concerns the amount of effort; and the frustration level, which is associated with the feeling of insecurity, discouragement, irritation, and stress. When the NASA-TLX is administered, the information provided by the participants concerns both the score and the weight for each dimension. The weight of each dimension reflects the relevance of this dimension to the task. Weighing consists of 15 binary comparisons among the six dimensions and represents the subjective importance of each dimension. For each pair of comparisons, one element is chosen that the participant considers the most important source of his/her load. The total weight of each dimension depends on the number of times that dimension was selected in the binary comparisons. The weight of each dimension ranges from zero (if the dimension is never chosen) to five (if the dimension is chosen in every pair of comparisons). The weight and the score of each dimension are provided after task completion. The participant estimates, on a scale from 0 to 100, his/her subjectively sensed magnitude cognitive load with respect to each dimension. Subsequently, using the above data, a weighted index for each dimension is calculated by multiplying the score of each dimension by its corresponding weight. Finally, the global index is calculated by adding up all the weighted scores and by dividing this sum by 15 [9,10].

There are several studies that have pointed out the worth of evaluating each NASA-TLX dimension independently in order to assess the effect of each dimension [40,41]. The global score can provide information about the overall load of a task, while each dimension score can show where the load lies in this task.

3. Comics in Education

Comics, with their multimodal features [42], present a story using a set of images and words and can be placed in a narrow conceptualization of the multimedia area [2,31]. The frame or panel, the bordered illustration containing visual information, is the fundamental unit of a comic. The shape and size of the frame are designed to make the emotions more noticeable, affect the speed of the action, and emphasize what the creator thinks is important [43]. The gutter, the space between the frames, is sometimes a single dividing line, but at other times, it may imply action to connect the frames through closure [43,44]. Images in comics, whether in the background or the foreground, have various characteristics and can provide information about the characters in the story, where and how the story develops, feelings, attitudes, opinions, etc. In addition, through technical features such as color, camera angle, zoom, etc., images can make understanding of this information easier for readers. Text expressing speech, thoughts, and sounds in comics appears in speech and thought balloons, captions, and sound effects. Balloons can be of various shapes to convey the characters’ intentions, feelings, and situations. Captions convey the narrator’s voice, and sound effects present environmental sounds that cannot be represented graphically or by the characters’ or narrator’s voices.

Comics, by combining text and visual representations, exploit both systems for incoming information according to CTL and CTML. The comic reader is allowed to take control of time and space by ceasing reading to look at and study the images. Text and picture integration facilitate comprehension, enabling comics to be an interesting educational tool that helps students strengthen their ability to synthesize information [12,20,45].

Comics attract students’ interest and help them deconstruct and reconstruct meaning [18]. They have the ability to develop critical thinking, motivate students, and offer clues that helped students remember what they have learned [22,23,24,46,47]. Lin et al. [48] stated that science comics, which combine visual representations and scientific explanations, can provide learners with visualized learning. Comics have also the ability to make the subjects more alive and accessible [20] and, consequently, to transfer knowledge. Moreover, comics provide students with the opportunity to nurture some of their intelligence according to Gardner’s multiple intelligence theory [49].

Digital comics creation combines the medium of comics, which children are familiar with, and computers, which attract students and motivate them to participate in the learning activities. Students prefer creating digital comics in school [25], and comics creation keeps them active in the classroom [20]. By asking students to create comics, the processes of problem solving, planning, and revision are activated [49,50]. Students, during comics creation, identify the most essential content of the learning subject and then reform it and retell it in the new format. Students think of digital comics creation as a technology that is enjoyable, easy to use, and useful in a school environment that can offer them learning opportunities [25].

Therefore, in the current study, we tested the following null hypothesis:

H0: 

There is no difference in students’ performance before and after creating digital comics.

The impact of digital comics creation on students’ attempts was also examined through the measurement of the imposed cognitive load on their cognitive system.

4. Materials and Methods

In the current study, we investigated the efficiency of digital comics creation as a classroom activity in students’ knowledge construction attempts in a computer science course. High school students aged thirteen and fourteen years old attending a school in a suburb of Athens, Greece, participated in the survey. Our experiment followed a common cognitive load research design that was connected with learning tests to understand the extent to which digital comics creation contributes to successful learning [51].

A total of 42 students, 16 boys and 26 girls, participated in the experiment. Initially, in order to familiarize students with the comic elements and their digital creation, the students attended a presentation about the comic medium and its basic elements. Students also attended a demonstration of the software interface and the software functions. Subsequently, students created a comic strip with the teacher’s guidance in order to become familiarized with the software.

Next, the students attended a presentation describing the issues of the learning subjects. Two subjects from the national curriculum were used in the study: computer viruses and multimedia. Before the comics creation procedure, the students took a 10 min test that served as a pre-test for the digital comics creation effect in which they had to answer questions about issues of the learning subject. Next, the students created comic strips displaying all issues of the subject under study and discussed with the teacher any relevant questions that arose during the comics creation procedure, explaining these issues. After comics creation, the students took a 10 min test on the issues of the learning subjects, such as what a computer virus is, how a computer virus enters a computer, what forms of information representation are combined in multimedia, and what image resolution expresses. This test served as a post-test for the digital comics creation effect. Answers to each item for all tests were marked as entirely correct, partially correct, or entirely wrong, and a total mark with a maximum value of twenty was calculated. Missing answers were marked as entirely wrong.

Subsequently, the students received the NASA-TLX questionnaire and filled it out anonymously.

Students used the ComicsFun Software created by [25] to create their comics. To create a comic, they first chose the number of frames and the frame they were going to fill and then added the characters and the balloons they needed to narrate their story. Since research has shown, on the one hand, that when students create drawings, an excessive cognitive load is produced and, on the other hand, that students can benefit if they are given, for example, cut-out figures [52,53], the software helps students by providing them with a set of character and object images.

The software interface was simple enough to complicate the activity. The buttons had funny images to attract students’ interest, and, when possible, they had images similar to those from other applications that students were familiar with, enhancing the software’s ease of use. Buttons were grouped by functionality in order to help students to identify related functions easily. Button functionality was indicated by hints and more explanatory information presented in the status bar. Figure 1 presents a screenshot of the software interface. It was designed to be suitable to the students’ computer skills and provide an introduction to the activity [25]. Some of the students’ comics, translated into English, are shown in Figure 2.

5. Results

All data were analyzed using SPSS statistical software, version 20, with a significance level of 0.05. In our experiment, the data were not normally distributed. Students performed better in the post-test. Thus, in order to check whether there was a significant improvement in students’ performance, we conducted a Wilcoxon Signed Rank Test showing a significant difference between the performances in pre- and post-tests (p = 0.000). This indicated that digital comics creation helped students in their knowledge construction effort, consequently giving no support to hypothesis H0.

The students, after creating their digital comics, also filled in the NASA-TLX questionnaire. In order to simplify the comprehension of the study and achieve a more plausible analysis of its results, we used the scale defined by [10]. Specifically, between 0 and 20%, a very low workload was indicated; between 20 and 40%, a low workload was indicated; between 40 and 60%, a normal workload was indicated; between 60 and 80%, a high workload was indicated, and between 80 and 100%, a very high workload was indicated. According to the above scale, the overall MWL was normal (mean value = 42.21), which is considered a positive MWL in terms of motivation to keep students active. The mental demand was low to normal (mean value = 32.98), the physical effort was very low (mean value = 11.79), the temporal demand was low to normal (mean value = 36.79), the effort was normal (mean value = 45.36), the performance was high (mean value = 71.19), for the frustration was low to normal (mean value = 32.86). The results regarding the overall workload as well as those across the six dimensions were similar in general terms, except for physical demand and performance, implying that digital comics creation does not burden MWL and its dimensions unevenly. The physical demand scored the lowest, as expected, because digital comics creation does not demand any particular physical effort on behalf of the students. Performance scored the highest, implying that students’ attempts to create successful digital comics was the most demanding dimension. The results also showed that digital comics creation was not considered difficult, which is notably positive for instructional goal fulfillment.

Concerning the weighing procedure, we found the weights of frustration and temporal demand to be the highest (3.17, 3.12), indicating that students put great importance on the time dimension and the feelings of insecurity and discouragement through the frustration dimension. The weights of mental demand, performance, and effort were 2.71, 2.60, and 2.43, respectively. The physical demand weight was the lowest, with a value of 0.98.

Finally, to better understand the relationship between the workload and the dimensions as proposed by the NASA-TLX, we conducted a Spearman correlation analysis for higher robustness, and only significant correlations with a minimum absolute value of 0.3 are mentioned because such a correlation denotes at least a medium correlation strength [8]. Specifically, the following dimensions were significantly correlated: temporal demand and mental demand (r_s = 0.452, p < 0.01), indicating that when the time pressure increased, the mental demands increased; mental demand and effort (r_s = 0.458, p < 0.01), indicating that when the mental demand increased, the students tried harder to accomplish the task; mental demand and frustration (r_s = 0.424, p < 0.01), indicating that when the mental demand increased, the students felt more stress; temporal demand and performance (r_s = −0.305, p < 0.05), indicating that the limited time the students had to create the digital comics negatively influenced the successfulness of the comics; temporal demand and frustration (r_s = 0.396, p < 0.01), indicating that students felt more stress and insecurity due to the limited available time; and, finally, frustration and effort (r_s = 0.510, p < 0.01), indicating that the higher the frustration was, the higher the effort that students exerted.

6. Discussion

6.1. Study Observations and Lessons Learnt

The present study investigated the effectiveness of digital comics creation on students’ knowledge construction attempts in real classroom settings. In an educational setting, such as the classroom, the educational activity that teachers choose must provide students with a stimulating experience, leading to a better understanding of the concepts and helping them acquire knowledge. Students’ test performance before and after the digital comics creation was compared. We also used the multidimensional NASA-TLX to determine the MWL the students experienced while creating their digital comics.

The results showed a significant difference between pre-test and post-test performance, implying that digital comics creation can assist teachers’ efforts to improve knowledge construction. Digital comics creation gives students the opportunity to participate in the learning process, thus promoting higher engagement and, consequently, enhancing knowledge construction. This is in accordance with the literature showing how Information and Communication Technology (ICT) helps students construct new knowledge [54]. Linardatos and Apostolou [25] stated that students perceive digital comics creation as useful because they see it as a tool capable of providing learning opportunities. Digital comics creation, by determining the most important content from the subject under study, includes active processing of the materials. Students, while creating their digital comics, divided comic strips into frames, put images inside each frame, and added suitable text to speech or thought balloons. Digital comics creation, thereby, motivates students to select only the essential information from the lesson, expressing it in their own words and reorganizing it using their existing knowledge. According to research, this way of prompting students can enhance learning [7]. Digital comics creation offers students the opportunity to evaluate their knowledge levels of the learning subject, make the necessary modifications, and integrate new information with prior knowledge into a mental model.

However, all of these actions put strain on students’ working memory, affecting the MWL that they experienced. Since working memory has a limited capacity [7] we investigated MWL through the NASA-TLX. The results showed that students experienced a normal MWL, which influenced them to be engaged with the digital comics creation activity and gain its corresponding benefits. Each frame in a comic presents information that is a continuation of the information of the previous frame, giving students guided access to information. The unguided discovery of information would put a heavy burden on students’ cognitive load.

The load for the NASA-TLX dimensions had a wide range, from low to high. The low case was, as expected, on behalf of the physical demand dimension since digital comics creation is an activity that takes place exclusively in front of a computer. The high case concerned the performance dimension, which is related to personal effectiveness in creating the digital comics. Students are interested in their performance in a school environment and, consequently, in creating successful comics that present the subject under study. This is of great importance because students were engaged in the activity and tried to accomplish the tasks.

The mental demand, the temporal demand, and the frustration dimensions were low to normal, while the effort dimension was normal. As Nikulin et al. [10] mentioned, these values could be a positive indicator of motivation. The students, as creators, can now determine the information contained in the comics, thus influencing the number of interacting elements that need to be simultaneously processed in working memory. As the number of elements within a learning task and the interactions between them increase, the experienced cognitive load also increases.

Students also chose the most significant source of workload during digital comics creation, comparing the NASA-TLX dimensions. The frustration and temporal demand dimensions were the most weighted, and the physical demand was again, as expected, the least weighted. Students rated limited in-classroom time as the most aggravating factor in their attempts to create digital comics. Equally important to the students was the feeling of insecurity and discouragement. Students might have had some ideas about how to create their comics, but these ideas may have proven not to be implementable. This could have resulted in the loss of valuable time, thereby increasing frustration. Students need more time to create digital comics and need to feel less stressed. Teachers, in order to lessen these negative factors, might exemplify digital comics creation by using digital comics created by other students. These results also show that students need more quickly implementable in-classroom activities and activities that make them feel more secure. Teachers need to take these characteristics into account when instructional tasks are incorporated into the classroom.

The correlation analysis of the NASA-TLX dimensions pointed out a relationship between temporal demand and mental demand. The higher the temporal demand was, the higher the mental demand became. Since we used real classroom conditions, the students needed to create the comics correctly and comprehensively, presenting the issues under study, within the limited time period of the teaching hour. This caused an increase in mental demand. The analysis also pointed out a relationship between mental demand and effort. When the digital comic is mentally demanding, the students are impelled to try harder to create the digital comics and present the necessary information, gaining more benefits from the activity.

Another relationship was between mental demand and frustration. The more mentally demanding the comics the students had to create were, the more pressure was put on students, which made them feel insecure about being able to create them. Choosing specific characters or creating a story that ended up not being implementable also could have added to the frustration dimension. However, as can be seen from the results, this worked to the benefit of the students, motivating them to continue their efforts and helping them reap the benefits of digital comics creation. The temporal demand was also related to the performance dimension. The limited available time the students had to create their comics negatively affected the performance they wanted to achieve. Possibly, they wanted to create more complex comics in terms of the story and characters. However, the more complex comics may have imposed an additional load on the students unrelated to the subject being studied, affecting the effectiveness of digital comics creation. This needs further investigation.

The temporal demand was also related to the frustration dimension. Due to limited time, students might have felt pressure. But, as mentioned above, offering more time might negatively affect the efficiency of digital comics creation. If students had seen some comics created by other students, they might have felt less pressure and insecurity. The same effect might have been observed if students had not been creating digital comics for the first time. This also needs further research. Finally, a relationship between frustration and effort was noticed. The greater the pressure and stress the students felt, the harder they tried to create their comics and present the information in the comic format.

Approaches to learning that combine text and images can enhance learning. However, in order to obtain these benefits, students need to be engaged in active cognitive processing. Students, nevertheless, often face difficulties with this process. Research on multimedia learning has provided some design principles, enabling beneficial instructional messages to be constructed [2,3,5,7,12,55,56]. All these principles aim to avoid problematic designs that would hinder learning and overload the student’s cognitive system. The students’ digital comics satisfied some of these principles, keeping cognitive load ata normal level. The students’ digital comics were simple enough and usually dialogic, mostly containing characters with spoken words. These comics mainly consisted of three frames (which was the maximum number of the available frames they were asked to create), enabling students to focus on key information. Students, as creators, depending on the way they created the comics, influenced the degree that these principles were applied.

Specifically, digital comics created by students follow the coherence principle because they contain the information that students consider necessary and appropriate to express the issue under study, thus removing any nonessential information. In addition, students had at their disposal a set of character images along with images of objects to include in their comics, minimizing any extra load that could be imposed by the drawing procedure.

Furthermore, the signaling principle was followed. The speech balloon (or thought balloon) is linked with the speaker (or thinker), directing the student’s attention to the character’s words (or thoughts) that contain the essential learning information. Moreover, students could change the font size and underline or bold words that they considered important to understand the comics.

The redundancy, spatial contiguity, and temporal contiguity principles were also applied to students’ comics. Students, due to the limited time and space they had, did not include superfluous or unnecessary information that would impede learning. Furthermore, due to the available images the students could use, the included images and text representations did not overlap. In comics, text usually corresponds to characters’ words. Therefore, it is placed close to the characters’ images, inside the same frame. Thus, the student does not try to combine information presented in previous frames with the frame they currently see.

Additionally, concerning the segmenting principle, students present the subject under study in frames, segmenting the information where they think it is appropriate, thus reducing the cognitive load they process. Segmentation, in our study, was student-paced, and therefore, it allowed the student to process as much information as he/she was able to, controlling the rate at which he/she received information. Some students may apply different segmentation than others depending on their abilities and prior experience. Further research is needed to determine the characteristics of this segmentation process.

The pre-training principle was also applied in our study. Students, before making the comics for the subject under study, had already made comics about an issue they liked in order to be familiar with the comic creation environment. Further research will determine exactly how long this familiarization should be. Moreover, before students created their comics, they attended a presentation describing the issues of the learning subject. Teachers can also use comics created by other students to exemplify the process. Further research is also needed on this case.

Finally, the modality, multimedia, personalization, and voice principles are applied to students’ comics. Presenting words as spoken text empowers students to learn better. Words in comics are presented as a simulating speaking activity by the characters’ mouths. Comics, by nature, follow the multimedia principle because they are a combination of text and images that can serve as a scaffold enabling students to construct their mental models. In comics, text is expressed by characters’ words, which are included in the same frame as the characters’ images. Therefore, the words are in a conversational style rather than a formal style, and they do not have mechanical speech characteristics but follow human speech.

From the above, we see that the students’ comics, along with the design of the activity (e.g., a comic strip with a maximum of three frames and limited available time) helped students apply these principles, positively affecting the cognitive load they experienced. Students need to create a clear and effective presentation of the information on the subject under study. Comics, by combining text and images and breaking the information into frames, allow students to benefit from the proven value of this combination and allow them to determine the way they access the necessary information.

6.2. Limitations and Future Work

Research on digital comics creation efficiency needs to be applied to other learning conditions as well. Since students can create digital comics using online applications, further research is needed to investigate how this way of creating digital comics could enhance learning. In addition, research in various disciplines with more participants of various ages and from different levels of education will shed more light on the impact of digital comics creation on students’ MWL.

It is worth noting that there were few cases of students whose MWL was particularly low or particularly high. Determining the cause requires further investigation. Students who experienced a high load may not have been experienced computer users, or the specific process requiring the creation even of a simple story may have been too taxing. This could potentially be solved if a comic was given with the first frame completed and the students had to continue its development, or if the students were given suggestions for the story beginning and chose one. Students might need more guidance to overcome the difficulties they might have faced with the software or the way the information was presented. Exemplifying the process could help them. Students who experienced a low load might have perceived digital comics creation as too simple compared with the applications they usually use outside of school. They might want an activity that could also incorporate other media, such as sounds, videos, or animations. Further research must be carried out to shed light on these issues.

In addition, students might have been further motivated if they could create more complex comics or draw their characters’ images. However, these comics may impose an additional load on students unrelated to their learning needs. Also, comics with more frames and, consequently, more information might need more signaling. Students might experience different MWL during these cases depending on their prior experience. A longer familiarization period and the use of comics created by other students might lessen the above differences and affect the way principles like segmenting and signaling are applied. Further research is needed to clarify the above cases.

Finally, other instruments assessing the cognitive load, individual assessment of the various tasks that the digital comics creation consists of (e.g., the story creation, finding characters’ images), or a dual-task methodology might reveal more information about the resources that burden students’ working memory and thus affect the cognitive load.

7. Conclusions

In this paper, we investigated the educational efficiency of digital comics creation in a computer science course in a real classroom setting. Digital comics creation was found to help students in their knowledge construction attempts. This finding is compliant with prior research stating that deep understanding occurs when students are encouraged to engage in productive learning activities [52]. During digital comics creation, the students process the subject under study and present its issues in a new format—the comic format. Hagaman and Reid [57] concluded that a paraphrasing strategy could lead to reading comprehension improvement, and Leopold and Leutner [53] stated that students learned better with pictorial summaries.

We also used the six NASA-TLX dimensions (mental demand, physical demand, temporal demand, effort, performance, and frustration) to assess the MWL and obtain knowledge of what causes this load. The average load and the loads on most dimensions imposed by digital comics creation were normal. The load on the performance dimension was high, indicating that students considered creating successful comics very important. The load on the physical demand dimension was very low since digital comics creation does not require physical strain on behalf of the students. Additionally, digital comics creation required low to normal mental demand and effort. Also, the frustration and temporal demand were the most aggravating on the students’ attempts.

Moreover, comics display information using the content forms of text and images. Mayer [2], stating that multimedia can be as simple as a still image with words, presented some principles that educational materials containing different content forms must satisfy in order to help students to construct knowledge. Students, as comic creators, can apply these principles to their comics, reaping the corresponding benefits. If digital comics creation is combined with appropriate instructional design, students’ efforts can be facilitated and become more effective.

Finally, our study, despite its limitations, shows that digital comics creation can help students in a real classroom achieve learning goals. Students might be stimulated to invest important effort and thus improve their learning by offering them an active role in their own learning process. The appropriate use of ICT can help teachers transform the learning environment into a student-centered one [1,58,59], and digital comics creation might contribute to this. Of course, further research is needed to replicate the results and determine parameters that will enhance the positive effect of digital comics creation.

Author Contributions

Conceptualization, D.A. and G.L.; Methodology, D.A. and G.L.; Validation, D.A. and G.L.; Formal Analysis, D.A. and G.L.; Investigation, D.A. and G.L.; Resources, D.A. and G.L.; Data Curation, D.A. and G.L.; Writing—Original Draft Preparation, G.L.; Writing—Review and Editing, D.A. and G.L.; Visualization, D.A. and G.L.; Supervision, D.A.; Project Administration, D.A. and G.L. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

The procedure performed in this study was in accordance with the ethical standards of the university and with the Declaration of Helsinki.

Informed Consent Statement

This study does not contain the personal data of the participants. The information obtained from participants was anonymized. By participating in the procedure, participants accepted that the authors could use the information to conduct research.

Data Availability Statement

The data presented in this study are available upon reasonable request from the corresponding author.

Conflicts of Interest

The authors declare no conflict of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures

Figure 1. ComicsFun Software.

Figure 2. Examples of students’ created comics.

References

1. Abdulrahaman, M.D.; Faruk, N.; Oloyede, A.A.; Surajudeen-Bakinde, N.T.; Olawoyin, L.A.; Mejabi, O.V.; Imam-Fulani, Y.O.; Fahm, A.O.; Azeez, A.L. Multimedia tools in the teaching and learning processes: A systematic review. Heliyon; 2020; 6, e05312. [DOI: https://dx.doi.org/10.1016/j.heliyon.2020.e05312] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33195834]

2. Mayer, R.E. The Cambridge Handbook of Multimedia Learning; 2nd ed. Cambridge University Press: New York, NY, USA, 2014.

3. Kizilcec, R.F.; Bailenson, J.N.; Gomez, C.J. The instructor’s face in video instruction: Evidence from two large-scale field studies. J. Educ. Psychol.; 2015; 107, pp. 724-739. [DOI: https://dx.doi.org/10.1037/edu0000013]

4. Harris, D.; Wilson, M.; Vine, S. Development and validation of a simulation workload measure: The simulation task load index (SIM-TLX). Virtual Real.; 2020; 24, pp. 557-566. [DOI: https://dx.doi.org/10.1007/s10055-019-00422-9]

5. Fyfield, M.; Henderson, M.; Phillips, M. Improving instructional video design: A systematic review. Australas. J. Educ. Technol.; 2022; 38, pp. 155-183. [DOI: https://dx.doi.org/10.14742/ajet.7296]

6. Kirschner, F.; Kester, L.; Corbalan, G. Cognitive load theory and multimedia learning, task characteristics, and learner engagement: The current state of the art. Comput. Hum. Behav.; 2011; 27, pp. 1-4. [DOI: https://dx.doi.org/10.1016/j.chb.2010.05.003]

7. Castro-Alonso, J.C.; de Koning, B.B.; Fiorella, L.; Paas, F. Five Strategies for Optimizing Instructional Materials: Instructor- and Learner-Managed Cognitive Load. Educ. Psychol. Rev.; 2021; 33, pp. 1379-1407. [DOI: https://dx.doi.org/10.1007/s10648-021-09606-9]

8. Gjoreski, M.; Kolenik, T.; Knez, T.; Luštrek, M.; Gams, M.; Gjoreski, H.; Pejović, V. Datasets for Cognitive Load Inference Using Wearable Sensors and Psychological Traits. Appl. Sci.; 2020; 10, 3843. [DOI: https://dx.doi.org/10.3390/app10113843]

9. Hart, S.G.; Staveland, L.E. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Adv. Psychol.; 1988; 52, pp. 139-183. [DOI: https://dx.doi.org/10.1016/S0166-4115(08)62386-9]

10. Nikulin, C.; Lopez, G.; Piñonez, E.; Gonzalez, L.; Zapata, P. NASA-TLX for predictability and measurability of instructional design models: Case study in design methods. Educ. Technol. Res. Dev.; 2019; 67, pp. 467-493. [DOI: https://dx.doi.org/10.1007/s11423-019-09657-4]

11. Umair, M.; Sharafat, A.; Lee, D.-E.; Seo, J. Impact of Virtual Reality-Based Design Review System on User’s Performance and Cognitive Behavior for Building Design Review Tasks. Appl. Sci.; 2022; 12, 7249. [DOI: https://dx.doi.org/10.3390/app12147249]

12. Yum, Y.N.; Cohn, N.; Lau, W.K. Effects of picture-word integration on reading visual narratives in L1 and L2. Learn. Instr.; 2021; 71, 101397. [DOI: https://dx.doi.org/10.1016/j.learninstruc.2020.101397]

13. Dimopoulos, K.; Koulaidis, V.; Sklaveniti, S. Towards an analysis of visual images in school science textbooks and press articles about science and technology. Res. Sci. Educ.; 2003; 33, pp. 189-216. [DOI: https://dx.doi.org/10.1023/A:1025006310503]

14. Tang, K.S. Distribution of visual representations across scientific genres in secondary science textbooks: Analysing multimodal genre pattern of verbal-visual texts. Res. Sci. Educ.; 2023; 53, pp. 357-375. [DOI: https://dx.doi.org/10.1007/s11165-022-10058-6]

15. Lee, V.R. Adaptations and continuities in the use and design of visual representations in US middle school science textbooks. Int. J. Sci. Educ.; 2010; 32, pp. 1099-1126. [DOI: https://dx.doi.org/10.1080/09500690903253916]

16. Mainali, B. Representation in teaching and learning mathematics. Int. J. Educ. Math. Sci. Technol.; 2021; 9, pp. 1-21. [DOI: https://dx.doi.org/10.46328/ijemst.1111]

17. Cohn, N. Your brain on comics: A cognitive model of visual narrative comprehension. Top. Cogn. Sci.; 2020; 12, pp. 352-386. [DOI: https://dx.doi.org/10.1111/tops.12421]

18. Hughes, R.; Hawkins, M.; Lopez, K. History, Literacy, and Popular Culture. Using Graphic Novels to Teach the Struggle for Racial Justice. Educating Through Popular Culture. You ’re Not Cool Just Because You Teach with Comics; Janak, E.; Sourdot, L.A. Lexington Books: Lanham, MD, USA, 2017; pp. 21-40.

19. Walton, G.; Childs, M.; Jugo, G. The Creation of Digital Artefacts as a Mechanism to Engage Students in Studying Literature. Br. J. Educ. Technol.; 2019; 50, pp. 1060-1086. [DOI: https://dx.doi.org/10.1111/bjet.12785]

20. Cha, J.; Kim, H.B.; Kan, S.Y.; Foo, W.Y.; Low, X.Y.; Ow, J.Y.; Chandran, P.D.B.; Lee, G.E.; Yong, J.W.H.; Chia, P.W. Integrating organic chemical-based socio-scientific issues comics into chemistry classroom: Expanding chemists’ toolbox. GreenChem. Lett. Rev.; 2021; 14, pp. 689-699. [DOI: https://dx.doi.org/10.1080/17518253.2021.2005153]

21. Hoffman, A. Comics and medicine: Using graphic narratives in pharmacy Education. Am. J. Pharm. Educ.; 2023; 87, ajpe8797. [DOI: https://dx.doi.org/10.5688/ajpe8797] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/34933900]

22. Grande-de-Prado, M. Quantitative review of articles about comics & education in Ibero-America. J. Graph. Nov. Comics; 2022; 13, pp. 287-301. [DOI: https://dx.doi.org/10.1080/21504857.2021.1898428]

23. Lazarinis, F.; Mazaraki, A.; Verykios, V.S.; Panagiotakopoulos, C. E-comics in teaching: Evaluating and using comic strip creator tools for educational purposes. Proceedings of the 2015 10th International Conference on Computer Science & Education (ICCSE); Cambridge, UK, 22–24 July 2015; pp. 305-309. [DOI: https://dx.doi.org/10.1109/ICCSE.2015.7250261]

24. Hosler, J.; Boomer, K.B. Are comic books an effective way to engage nonmajors in learning and appreciating Science?. CBE—Life Sci. Educ.; 2011; 10, pp. 309-317. [DOI: https://dx.doi.org/10.1187/cbe.10-07-0090]

25. Linardatos, G.; Apostolou, D. Investigating high school students’ perception about digital comics creation in the classroom. Educ. Inf. Technol.; 2023; [DOI: https://dx.doi.org/10.1007/s10639-023-11581-3]

26. Virtanen, K.; Mansikka, H.; Kontio, H.; Harris, D. Weight watchers: NASA-TLX weights revisited. Theor. Issues Ergon. Sci.; 2022; 23, pp. 725-748. [DOI: https://dx.doi.org/10.1080/1463922X.2021.2000667]

27. Sweller, J.; van Merriënboer, J.J.G.; Paas, F. Cognitive Architecture and Instructional Design: 20 Years Later. Educ. Psychol. Rev.; 2019; 31, pp. 261-292. [DOI: https://dx.doi.org/10.1007/s10648-019-09465-5]

28. Paas, F.; Renkl, A.; Sweller, J. Cognitive load theory: Instructional implications of the interaction between information structures and cognitive architecture. Instr. Sci.; 2004; 32, pp. 1-8. [DOI: https://dx.doi.org/10.1023/B:TRUC.0000021806.17516.d0]

29. Skulmowski, A.; Xu, K.M. Understanding cognitive load in digital and online learning: A new perspective on extraneous cognitive load. Educ. Psychol. Rev.; 2022; 34, pp. 171-196. [DOI: https://dx.doi.org/10.1007/s10648-021-09624-7]

30. Sweller, J. Element interactivity and intrinsic, extraneous, and germane cognitive load. Educ. Psychol. Rev.; 2010; 22, pp. 123-138. [DOI: https://dx.doi.org/10.1007/s10648-010-9128-5]

31. Anmarkrud, Ø.; Andresen, A.; Bråten, I. Cognitive load and working memory in multimedia learning: Conceptual and measurement issues. Educ. Psychol.; 2019; 54, pp. 61-83. [DOI: https://dx.doi.org/10.1080/00461520.2018.1554484]

32. Sweller, J.; van Merrienboer, J.J.G.; Paas, F.G.W.C. Cognitive architecture and instructional design. Educ. Psychol. Rev.; 1998; 10, pp. 251-296. [DOI: https://dx.doi.org/10.1023/A:1022193728205]

33. Zu, T.; Hutson, J.; Loschky, L.C.; Rebello, N.S. Using eye movements to measure intrinsic, extraneous, and germane load in a multimedia learning environment. J. Educ. Psychol.; 2020; 112, pp. 1338-1352. [DOI: https://dx.doi.org/10.1037/edu0000441]

34. Sorden, S. The cognitive theory of multimedia learning. The Handbook of Educational Theories; Irby, B.J.; Brown, G.; Lara-Alecio, R.; Jackson, S. Information Age Publishing: Charlotte, NC, USA, 2013; pp. 155-167.

35. Ruan, L.; Xiong, Z.; Jiang, L.; Zhou, X. Comparison between digital and paper note-taking based on NASA-TLX. Proceedings of the 2015 IEEE International Conference on Progress in Informatics and Computing (PIC); Nanjing, China, 18–20 December 2015; pp. 221-225. [DOI: https://dx.doi.org/10.1109/PIC.2015.7489841]

36. LaBara, L.M.A.; Meloni, L.; Giusino, D.; Pietrantoni, L. Assessment Methods of Usability and Cognitive Workload of Rehabilitative Exoskeletons: A Systematic Review. Appl. Sci.; 2021; 11, 7146. [DOI: https://dx.doi.org/10.3390/app11157146]

37. Ro, H.; Byun, J.-H.; Park, Y.J.; Lee, N.K.; Han, T.-D. AR pointer: Advanced ray-casting interface using laser pointer metaphor for object manipulation in 3D augmented reality environment. Appl. Sci.; 2019; 9, 3078. [DOI: https://dx.doi.org/10.3390/app9153078]

38. Emerson, L.; MacKay, B. A comparison between paper-based and online learning in higher education. Br. J. Educ. Technol.; 2011; 42, pp. 727-735. [DOI: https://dx.doi.org/10.1111/j.1467-8535.2010.01081.x]

39. Buchner, J.; Buntins, K.; Kerres, M. A systematic map of research characteristics in studies on augmented reality and cognitive load. Comput. Educ. Open; 2021; 2, 100036. [DOI: https://dx.doi.org/10.1016/j.caeo.2021.100036]

40. Mingardi, M.; Pluchino, P.; Bacchin, D.; Rossato, C.; Gamberini, L. Assessment of implicit and explicit measures of mental workload in working situations: Implications for industry 4.0. Appl. Sci.; 2020; 10, 6416. [DOI: https://dx.doi.org/10.3390/app10186416]

41. Galy, E.; Paxion, J.; Berthelon, C. Measuring mental workload with the NASA-TLX needs to examine each dimension rather than relying on the global score: An example with driving. Ergonomics; 2018; 61, pp. 517-527. [DOI: https://dx.doi.org/10.1080/00140139.2017.1369583]

42. Jiang, S.; Smith, B.E.; Shen, J. Examining How Different Modes Mediate Adolescents’ Interactions During Their Collaborative Multimodal Composing Processes. Interact. Learn. Environ.; 2021; 29, pp. 807-820. [DOI: https://dx.doi.org/10.1080/10494820.2019.1612450]

43. Morrison, T.G.; Bryan, G.; Chilcoat, G.W. Using student-generated comic books in the classroom. J. Adolesc. Adult Lit.; 2002; 45, pp. 758-767.

44. McCloud, S. Understanding Comics: The Invisible Art; HarperCollins: NewYork, NY, USA, 1993.

45. Tsene, L. Using comics as a media literacy tool for marginalized groups: The case of Athens comics library. Media Commun.; 2022; 10, pp. 289-293. [DOI: https://dx.doi.org/10.17645/mac.v10i4.5716]

46. Nagata, R. Learning biochemistry through manga—Helping students learn and remember, and making lectures more exciting. Biochem. Educ.; 1999; 27, pp. 200-203. [DOI: https://dx.doi.org/10.1016/S0307-4412(99)00052-7]

47. Suh, S.; Lee, M.; Xia, G.; Law, E. Coding strip: A pedagogical tool for teaching and learning programming concepts through comics. Proceedings of the 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC); Dunedin, New Zeeland, 10–14 August 2020; pp. 1-10. [DOI: https://dx.doi.org/10.1109/VL/HCC50065.2020.9127262]

48. Lin, S.-F.; Lin, H.-S.; Lee, L.; Yore, L.D. Are science comics a good medium for science communication? The case for public learning of nanotechnology. Int. J. Sci. Educ. Part B; 2015; 5, pp. 276-294. [DOI: https://dx.doi.org/10.1080/21548455.2014.941040]

49. Kirtley, S.E.; Garcia, A.; Carlson, P.E. A Once and Future Pedagogy. With Great Power Comes Great Pedadogy; Kirtley, S.E.; Garcia, A.; Carlson, P.E. University Press of Mississippi: Jackson, MI, USA, 2020; pp. 3-19.

50. Wiegerova, A.; Navratilova, H. Let’s Not Be Scared of Comics (Researching Possibilities of Using Conceptual Comics in Teaching Nature Study in Kindergarden). Procedia-Soc. Behav. Sci.; 2017; 237, pp. 1576-1581. [DOI: https://dx.doi.org/10.1016/j.sbspro.2017.02.248]

51. Krieglstein, F.; Beege, M.; Rey, G.D.; Ginns, P.; Krell, M.; Schneider, S. A systematic meta-analysis of the reliability and validity of subjective cognitive load questionnaires in experimental multimedia learning research. Educ. Psychol. Rev.; 2022; 34, pp. 2485-2541. [DOI: https://dx.doi.org/10.1007/s10648-022-09683-4]

52. Schwamborn, A.; Thillmann, H.; Opfermann, M.; Leutner, D. Cognitive load and instructionally supported learning with provided and learner-generated visualizations. Comput. Hum. Behav.; 2011; 27, pp. 89-93. [DOI: https://dx.doi.org/10.1016/j.chb.2010.05.028]

53. Leopold, C.; Sumfleth, E.; Leutner, D. Learning with summaries: Effects of representation mode and type of learning activity on comprehension and transfer. Learn. Instr.; 2013; 27, pp. 40-49. [DOI: https://dx.doi.org/10.1016/j.learninstruc.2013.02.003]

54. Parody, L.; Santos, J.; Trujillo-Cayado, L.A.; Ceballos, M. Gamification in Engineering Education: The Use of Classcraft Platform to Improve Motivation and Academic Performance. Appl. Sci.; 2022; 12, 11832. [DOI: https://dx.doi.org/10.3390/app122211832]

55. Mayer, R.E. Evidence-based principles for how to design effective instructional videos. J. Appl. Res. Mem. Cogn.; 2021; 10, pp. 229-240. [DOI: https://dx.doi.org/10.1016/j.jarmac.2021.03.007]

56. Noetel, M.; Griffith, S.; Delaney, O.; Harris, N.R.; Sanders, T.; Parker, P.; del Pozo Cruz, B.; Lonsdale, C. Multimedia Design for Learning: An Overview of Reviews with Meta-Meta-Analysis. Rev. Educ. Res.; 2021; 92, pp. 413-454. [DOI: https://dx.doi.org/10.3102/00346543211052329]

57. Hagaman, J.L.; Reid, R. The Effects of the Paraphrasing Strategy on the Reading Comprehension of Middle School Students at Risk for Failure in Reading. Rem. Spec. Educ.; 2008; 29, pp. 222-234. [DOI: https://dx.doi.org/10.1177/0741932507311638]

58. Coleman, L.O.; Gibson, P.; Cotten, S.R.; Howell-Moroney, M.; Stringer, K. Integrating computing across the curriculum: The impact of internal barriers and training intensity on computer integration in the elementary school classroom. J. Educ. Comput. Res.; 2016; 54, pp. 275-294. [DOI: https://dx.doi.org/10.1177/0735633115616645]

59. Nieto-Márquez, N.L.; Baldominos, A.; Martínez, A.C.; Pérez-Nieto, M.Á. An exploratory analysis of the implementation and use of an intelligent platform for learning in primary education. Appl. Sci.; 2020; 10, 983. [DOI: https://dx.doi.org/10.3390/app10030983]