Analyzing Characteristics of Experts in the Context of Stoichiometric Problem-Solving

Abstract

To gauge the variability in expert problem-solving strategies for stoichiometry problems, a set of experts in different career tracks were studied with the cohort including 17 graduate students in chemistry, three college chemistry instructors, and seven college graduates working in the industry. The goal of the study was to determine whether variability would be observed based upon experience and career trajectories. The data were collected using interviews and analyzed qualitatively and quantitatively using the COSINE (Coding System for Investigating Sub-problems and Network) method. Although the method was developed for the analysis of undergraduate problem-solving, it appeared to be effective in examining experts’ problem-solving in chemistry as well. The study revealed similar abilities for succeeding at solving a series of problems, but the strategies were variable for the three cohorts of experts. Specifically, the amount of information used to solve the problems differed across the three cohorts with graduate students focusing more upon each of the specific subproblems within each problem compared to industry chemists utilizing the big-picture approach in lieu of breaking down each problem into respective subproblems. Familiarity with the question types and ability to chunk information were common characteristics observed consistently for the expert participants, which is consistent with existing research.

1. Introduction

Problem-solving, whether explicitly stated or not, is always an underlying goal of STEM courses. This focus is driven because of the necessity to solve a wide scope of problems in STEM-oriented careers [1]. In chemistry education, significant research has focused upon developing methodologies for effectively teaching problem-solving [2,3,4,5,6]. Similar to prior reports by Bodner [7] and Pastell [8], which outlined frustrations students experience with problem-solving, current articles are outlining similar difficulties despite attempts for scaffolding or unpacking problems for students [9,10], incorporating of active learning [11,12,13], online assignments [14], and several other teaching strategies [15,16,17,18,19]. Student difficulties with problem-solving are persisting with students having difficulty with problem-solving due to the use of ineffective strategies and lack of confidence in their problem-solving abilities [14,20]. As a deeper probe into problem-solving, comparisons between expert and novice problem-solving pathways [21,22] have been widely studied and reported across a range of subjects including physics [23] and chemistry [24,25]. The literature on experts’ problem-solving strategies, their overall success in problem-solving, and their differences from novices are quite prevalent throughout the academic community. However, there is little literature that focuses on how experts attempt to do sub-problems within a question. Specifically, what subproblems are chunked by experts and how does this vary based upon experience. A previous study [25] investigated how an expert solves algorithmic and conceptual problems but only focused on the time spent in 1 of 5 stages, providing little information about the details of problem-solving and how subproblems are effectively chunked by experts. This study aims to analyze how experts, with variable backgrounds and careers, approach solving stoichiometry exercises. Note the term exercises is used to define these activities because experts should view these as exercises with a known pathway to the solution in lieu of problems. The goal is to gain insight into the strategies as well as the expert knowledge structure in order to consider and design potential future interventions.

Experts are defined as individuals who “have acquired extensive knowledge that affects what they notice and how they organize, represent, and interpret information in their environment” [26]. DeGroot [26] reported that expert chess players analyze certain permutations of a chessboard and developed strategies far different than those of novice chess players. The quality of chess moves of expert players was higher than the novices because of their abilities to chunk elements together, that is their ability to recognize familiar patterns and form more sophisticated strategies to win. Chunking [22] comes from the prior knowledge that can be fluently recalled when needed, on the other hand, novices chunk content to a lesser extent because they lack such strong connections. Previous research studies have focused on key aspects of expert thinking. One of these characteristics is expert knowledge structure. The term “knowledge structure,” also referred to as a “cognitive structure,” describes the mental representation created as someone learns and conceptually connects related information [27,28]. An expert’s knowledge structure of a specific domain provides a comprehension representation of the concepts and their interconnections [27,28].

Chunking abilities stem from knowledge structures, which can be assessed in various ways, and in turn, can be useful in assessing skills such as problem-solving [29]. Although experts tend to have more extensive and organized declarative knowledge structures [30], the problem-solving ability is based not only on the quantity of knowledge but also on knowing how information is organized and applied to problems [31]. Experts’ more complex organization of knowledge and related concepts can enable them to be more effective problem-solvers compared to novices [31]. While making observations about the knowledge structures of experts, it is important to acknowledge the existence of distortions due to expert bias, which is defined as the inability for a person who mastered a skill or a concept to appreciate how useful that skill or concept can be to a novice [31]. Understanding these biases is important particularly in teaching or communicating with novices because key information may be omitted, thereby, causing challenges for students and individuals who are first learning content [31]. Think-aloud protocols are common research approaches utilized to minimize the extent of expert bias in research studies [31]. By using think-aloud protocols with verbal cues from the researchers to capture missing pieces of information, we can identify not only problem-solving strategies utilized by experts, but we can also identify potential concepts and ideas that might not be fully conveyed in lecture because of bias.

Experts’ problem-solving abilities also depend on various cognitive variables including working memory, attentional control, and context. Acting as a mental storage system, working memory preserves information while also simultaneously processing other immediate information [32]. The success of working memory in recalling and processing information is subject to attentional control [33]. Therefore, if knowledge is not retrieved and processed as chunked pieces, working memory capacity can become a barrier to problem-solving abilities. The complexity [21] of the problem is another determinant in the ability to recall, as well as, apply the knowledge. Examples of strategies include using algorithms, changing the point of view, working backwards, developing and using models, or relating to analogous problems [34]. Full development of these processes and hierarchies give rise to high potency in successful problem-solving. These cognitive functions can be developed through an individual’s internalization of knowledge gained from different resources, and his or her externalization of the internal representations [21]. Analysis of the strategies utilized by experts will provide greater insight not only into their strategies, but the range of different successful approaches observed within the stoichiometry domain [29]. Experts also utilize analogies to break problems down into subproblems and identify the main concepts and laws associated with the problem [35]. The analogies [22] and other approaches used to design subproblems or chunk information are pertinent for understanding and tailoring for the development of interventions.

In addition, expert problem-solving is impacted by their affective domain. One’s affective domain can be defined as the values, interests, attitudes, and motivations they have that can impact how they solve problems [36]. Experts have an increased cognitive and affective functioning ability as they have a more positive attitude towards the problem in hand, often admiring the subject and therefore being more inclined to be perseverant compared to novices [37]. Therefore, during the interviews, it was expected to see a greater level of confidence when responding to each of the prompts and describing their strategies.

2. Methodology

2.1. Research Questions

This study aimed to document the differences between different expert groups at a level that have not been achieved in other studies and provide additional information about the characteristics of experts. The study was guided by two research questions:

(1)

How does different expert groups’ success with sub-problems differ from that of overall problem?

(2)

What characteristics do experts show while dealing with a wide range of stoichiometry questions?

The first research question is particularly important given the lack of focus on experts’ success with sub-problems in the literature. It will also highlight the influence of different expertise and the role of interaction with students on expert groups’ problem-solving performance. The COSINE method [29] has not been applied to expert problem-solving previously, and our goal was also to assess whether key attributes of expert problem-solving can accurately be captured using this coding system. More information regarding the COSINE method will be further discussed in a later part of this section. The second research question will provide greater insight into expert qualities for guiding novices to become effective problem solvers and developing new approaches for teaching stoichiometry.

2.2. Participants and Settings

The participants for this study were selected based on their knowledge and perceived mastery of chemistry. For purposes of recruiting participants, three cohorts were identified. Graduate students, who participated in the study, had between one and four years of experience in graduate school. A total of 17 graduate students were recruited. Three chemistry instructors (two chemistry lecturers who teach general chemistry on a regular basis and one organic chemistry professor) were recruited. These individuals have Ph.D. degrees in chemistry. Finally, six chemistry graduates, who were working in the industry were recruited. An email request was sent to the graduate students in the department for recruitment. The professors were recruited individually through personal communications. Finally, the six industrial chemists were also recruited on an individual basis through personal communication. The industry professionals had five to twenty years of experience in addition to their educational backgrounds ranging from an undergraduate degree to a doctoral degree. All participants volunteered to participate and consented to the use of voice recordings as part of the think-aloud protocol. All interviews were held in quiet and secluded spaces to limit disruptions and distractions. Participants were compensated for their contribution with gift cards of modest monetary value.

2.3. Instruments and Design

Five closed-ended questions were selected that ranged in difficulty, which was discerned based on the problem type: computational or application focused. These problems were standard textbook-style problems with the focus of each question being different such that the background knowledge required to solve them varied. The problems have been used for other studies that are in preparation for publication. There was an approximate time allotted for each interview, about 30--35 min, and this time constraint was based on the idea that experts would not take much time to answer the five questions. The rigor and number of questions, as well as the time allowed for problem-solving, were decided based on previous studies on problem-solving and the same conditions were used for all expert problem-solving sessions [29]. Eight interviewers were trained by the first author through several mock interviews in which each student experienced the think-aloud process and learned how to administer the protocol. After inquiring information on participants’ field of work, educational background, and teaching experience, they were explained the importance of thinking aloud while solving questions and including all of their thought processes. In addition, if necessary, previously agreed upon hints were provided for the first step in the question to prompt the problem-solving process.

The interviews were recorded and transcribed for data analysis. To assess the approach taken, specifically for each subproblem, the Coding System for Investigating Subproblems and Network (COSINE) was applied. To apply COSINE, the problems were broken into sub-problems, occasionally referred as steps, which were associated with different stoichiometry topics such as writing chemical equations (WEQ), balancing chemical equations (BEQ), mole concept (MC), stoichiometric or mole-to-mole ratio (SR), mass conversion (MassC) and limiting reactant (LR). The COSINE codes were then utilized to assess the participants’ solutions, find the sources of differences between expert groups, and better understand their common challenges. One of the seven codes is assigned to each subproblem with the individual codes being grouped into three categories: successful, neutral, and unsuccessful. The code S (Successful), which is the only code in the successful category, is used when a participant completes a subproblem correctly. One of three neutral codes, NR (Not Required), DD (Did not know to Do), and DSE (Did Something Else), are assigned when a participant does not attempt a subproblem for various reasons. The NR code is assigned when participants skip over a subproblem by using a different strategy that does not involve that specific step. The DD (Did not know to Do) is assigned when a participant skips over a required step or omits information using a piece of data. This is different than NR in that the DD code is assigned when the predicted procedure (with the proposed problems) is being used to the solve problem. The DSE (Did Something Else), is assigned when a participant does not complete the required subproblem but instead does something else. For example, a participant may subtract the number of moles for no reason using the predicated procedure. The unsuccessful group of codes includes cases where the participant attempts the subproblem but does not complete it successfully. The code UDI (Unsuccessful Did Incorrectly) is assigned when a participant correctly identifies and attempts a subproblem, but the attempt is not successful. For example, the participant realizes a balanced equation is needed, but incorrectly balances the equation. The code UG (Unsuccessful Guessed) refers to a subproblem where the participant attempts the subproblem by guessing. Finally, the CD code is assigned when participants indicate they know what sub-problem is needed but mention that they cannot carry out that calculation. The evidence for this code was particularly obtained from the transcripts. These codes are assigned carefully using think-aloud protocols during interviews and analyzing the work provided while solving problems. A more detailed description of the COSINE codes is available in the article by Gulacar et al. [29]. The inter-coder reliability was evaluated by calculating Cohen’s Kappa coefficient, which was 0.769.

3. Results and Discussions

The following sections outline findings related to each question. The analysis provides great insight into the differences between expert groups and general characteristics of experts in the context of stoichiometric problem-solving.

3.1. Analyzing Experts’ Success with Sub-problems Involving Individual Stoichiometric Topics:

In order to quantify and highlight the differences between experts’ success with questions and, more importantly, with individual sub-problems, the COSINE method was utilized to analyze the data that were collected through think-aloud protocols. The COSINE codes described in the Methods section were applied to each sub-problem which was identified for all the questions utilized in the study. With the use of the COSINE codes, attempt success rate (ASR) and complete success rate (CSR) were calculated. ASR is a measure of a participant’s true ability to do individual sub-problems, which is determined by using the clear evidence from his solutions and the things said while solving questions. This formula incorporates only the successful and unsuccessful codes (Equation (1)). CSR is calculated similarly to ASR, but, additionally, incorporates the neutral codes (DD and DSE) as well (Equations (2)). It pertains to how the participants connect their sub-problems together or chunking⁠—a higher CSR is more indicative of a greater ability to chunk information. For novices, the high number of DD and DSE is an indication of poor conceptual understanding and result of overloading working memory capacity. ASR and CSR were both calculated, as shown in Equations (1) and (2), and averaged for each stoichiometric category by expert groups, with an overall percentage for both rates.

$$\mathrm{ASR}=\frac{S}{S+UDI+URH+CD+UG}$$

(1)

$$\mathrm{CSR}=\frac{S}{S+UDI+URH+CD+UG+DD+DSE}$$

(2)

A comparison of the ASR and CSR values, for each of the subproblems, for graduate students, college chemistry instruments, and industrial chemists is provided in Figure 1 and Figure 2, respectively. Figure 1 represents the average ASR of all subproblems involved in the questions that were administered in the think-aloud protocols. ANOVA across the three cohorts for the six subproblems yielded a [F _(2,15) = 0.31, p = 0.73].

The average CSR values for all subproblems involved in administered questions are represented in Figure 2. Again, ANOVA was run to calculate the F value for the three expert cohorts and the six subproblems. The test yielded a (F _(2,15) = 1.37, p = 0.28).

Our goal was to analyze the performance of three cohorts of experts each having differing backgrounds and careers at a sub-problem level. When comparing the analysis of variance across the three cohorts, no statistical differences were observed between the three groups for the ASR (p = 0.73) and the CSR (p = 0.28). The statistical comparison supports the three cohorts as having identical abilities to successfully solve the problems based upon ASR and similar chunking abilities based upon CSR. Graduate students had the highest overall Attempt Success Rate (73.7%), followed by professors (67.4%), and finally the industry participants (66.5%).

While statistical differences were not observed with regard to CSR, we can rationalize the trend observed in which graduate students scored higher in CSR than professors who scored higher than industrial chemists. Due to a lower TA to student ratio, compared to a professor to student ratio, graduate students are frequently expected to provide detailed answers to the questions similar to those used in the study. Professors should exhibit the same level of knowledge, or perhaps higher, since they teach the subject and must have the relevant knowledge to answer questions. However, the discrepancy in the data could potentially be attributed to the limitation of having a low number of professor participants, with one of the three professors routinely teaching organic chemistry. The industrial chemists scored lower but this could be attributed to different problem-solving demands of their careers [38]. Industrial chemists consider measurement, chemical equations, and related content daily, but the content is often applied to more open-ended problems necessitating the use of a wider range of problem-solving approaches. This may also explain why the industry members, as more specialized experts in fields beyond general chemistry, exhibited relatively less prowess when navigating the problems. These observations are supported by comparing the ASR and CSR values, which show graduate students having the smallest difference between ASR and CSR (6.62%), followed by professors (7.19%), and then industry participants (13.71%). This expert behavior promotes the ability of grouping, in other words, chunking fragmented pieces of knowledge around the underlying principles, which leads to more efficient use of working (short-term) memory [21,33,39].

Due to experiences of graduate students and professors in teaching stoichiometry, generally with systematic approaches, this is reasonable that these cohorts will be more detailed and meticulous when including and solving the subproblems. In fact, the graduate students and professors used a more instructional approach⁠—as though they were teaching how to solve the problems during the think-aloud interviews. This observation is supported given the fact that the problems used in the research, closely match those that one would expect on a general chemistry exam. However, the gap is wider for the industrial chemists because they likely focus upon different problems and do not consider each subproblem out of necessity to teach others. Therefore, when considering chunking analyses, one should consider how background may introduce bias in the studies—in lieu of chunking information, graduate students and professors may have provided extra information out of habit from teaching undergraduate students.

3.2. Identifying the Characteristics of Experts in the Context of Stoichiometric Problem-Solving

While statistical analysis of expert problem-solving performance measured by success rates is valuable, there were many observations noted during the think-aloud interviews were not fully captured by the success rate analysis. Therefore, participant transcripts were qualitatively analyzed considering the reported expert-typical problem-solving traits as reported in the literature. These behaviors were recorded as they revealed patterns in expert thought processes that may prove useful and warrant closer attention in future studies. Regardless of background, the three cohorts of experts approached problems similarly in that they carefully reviewed the instructions and vocalized their strategies in moving from the beginning to end of each problem. Comprehensive problem perception has been previously observed in experts and this understanding of the interconnection and application of concepts, while problem-solving can be related back to experts’ extensive and organized knowledge structures [28,39].

The use of extensive knowledge was observed in two forms, chunking and familiarity. It was operationalized based on observations that demonstrated going beyond the superficial and procedural steps of the problem and instead examining deeper meaning and concepts being utilized. This was demonstrated by one participant as they stated “how many g of N gas form? Ok, so this I mean when I see that they give me amounts of 2 of the starting materials I automatically know it’s a limiting reactant problem. So the first thing I have to do is find out which one is limiting” suggesting that the participant categorized the problem as a limiting reagent problem and grouped the given information under this category to recall the relevant information from the long term memory and process it using short-term memory with high-efficiency success [21,33]. Previous studies have supported the significance of the organization of knowledge, chunking, and working memory that are closely related to and affect one’s efficiency in problem-solving [40]. Cognitive Load Theory postulates that the limited capacity and duration of information in one’s working memory does not apply to information that has been retrieved from the long-term memory, experts having relevant knowledge already stored in their long-term memory in the form of schemata may contribute to their working memory capability [41,42]. Therefore, because of chunking experts’ working memories compared to novices’ are superior because of the efficient organization of knowledge [43], which reduces cognitive load.

The experts in our study also explored the use of extensive knowledge by displaying familiarity with the concepts. This was recognized several times while analyzing the participant transcripts, as many of the experts used their familiarity with the content, stoichiometry, to help them solve the questions. The use of familiarity was identified by when participants recognized a particular part of the questions and related them to methods they have already encountered. An example of this is when a participant mentioned “so it’s asking for the molecular formula and the molecular formula is simply just the ratio between atoms whereas they are giving you percents so you have to convert your percents, I guess, into some number and you can convert that into a ratio. That’s my strategy, I guess.” Familiarity with the subject allows participants to be able to recognize a particular type of question and more easily solve it by using steps they have applied to similar questions they have worked within the past. Furthermore, participants noted several values they deemed important before starting the problems, one even explaining that “[he] really like[s] to write all units and be super clear about it because when you’re doing those big stoic problems its’ easy to get lost. So write out your units, I think that’s really important for people.” Another observation related to working memory capacity is that many of the participants in this study seemed to manipulate units during their calculations and perform dimensional analysis automatically and with little effort and almost no error. This observation is in contrast with novices’ use of units to solve problems [44,45].

Along with the characteristics related to cognitive aspects of problem-solving, some important qualities were noted regarding affective domain during the think-aloud interviews, which are equally important variables for determining one’s success with problem-solving [36]. Perseverance is one of them and described as a participant continuing to attempt a problem even when confronted with a roadblock such as a mistake or a piece of knowledge that they were unsure of or lacked. During the interview, one participant demonstrated perseverance when starting problem 5, “so I’m thinking about how I am going to combine these right now to solve for everything because... uh... it’s not innately obvious how to get to the answer to me. Umm, so I guess we don’t really know what these masses are, um, but, um, we do know the molecular weights so I am just going to keep on transforming the equation into something that’s meaningful.” Experts have greater resilience to challenges and continue to strive for solutions [36]. High sense of self-efficacy is also related to being persistent and has been observed to be correlated with increased cognitive performance and problem-solving success [46]. When teaching, we need to emphasize that solutions presented in the textbook are not always easily acquired—as may be depicted from the confined solutions provided. As scientists, we have many false starts and experience challenges. However, solutions as presented in the textbook and generally through our lectures fail to emphasize hurdles and challenges and may attribute to students’ lack of perseverance and sense of failure when they cannot immediately produce a solution similar to the notes or book.

The qualitative expert characteristics revealed in this study help illuminate the patterns of expert thinking. However, limitations can be identified with the research. It is important to note that even experts were observed to have trouble vocalizing all of their thoughts in the think-aloud protocol. As discussed in other papers, the think-aloud method has downsides such as increasing cognitive load and not capturing unconscious processes [24]. For some participants, their verbalization was especially sparse when they were doing something they deemed very simple, such as writing out a reaction equation. Other times they would say what they were doing or writing out but not explain why. Additionally, some participants would hardly explain any of their thoughts at all, despite continued prompting from the interviewers. This may reflect the idea discussed above that some of these thought processes and schemata have become automated due to the experience of these expert participants, making such thoughts less conscious. One additional factor that could have affected the findings was the fact that a limited amount of time was given for each problem, leading to participants feeling rushed (which may have affected their success). These points demonstrate the limitations of the think- aloud method of measuring participant thought processes. However, think-aloud protocol as a whole continues to provide important insights into expert thought processes and problem-solving strategies that merely observing their work and success could not capture [27,28]. Future studies should closely examine any differences between expert participant case-based episodic knowledge, the type of knowledge involving in the formation of a mental model that can later be applied to similar situations and problems, and proximity to the material [46,47].

4. Conclusions

The results of this study conclude that experts are very successful in problem-solving which is indicated by the high attempted solve rate (ASR) and completed solve rate (CSR) values, as well as minor discrepancies between the ASR and CSR values. As previously stated, ASR refers to a given participant’s ability and knowledge to deal with an individual sub-problem, while CSR pertains to the interconnections of sub-problems. Graduate students and professors had higher CSR scores collectively, which could be explained given their interactions in explaining and discussing content with students. An ANOVA test indicated there were no significant differences between the means of the expert groups for either ASR or CSR. This suggests that none of the expert groups outperformed nor underperformed relative to the other groups.

This study and future studies will aid in developing these findings, with the hopes of applying them to the curriculum as well as textbooks throughout all levels of education. An emphasis on critical thinking using multi-concept questions and topics will be beneficial to all students. Our study suggests that the most successful groups of experts had extensive amounts of practice in the kinds of questions we asked, given that they both solve questions that are similar frequently and teach students how to tackle with them. This suggests that students can benefit from increased amounts of repetition of a particular exercise as well as learning in the form of teaching others. In addition, revealing the problem-solving strategies of experts has implications for students because they can see what leads to successful problem-solving and grants them insight into how to approach questions in more effective ways. Furthermore, given the use of analogies and the connections to chunking and recalling information, a future study will explore the impact of analogies on ASR and CSR results, as well as, qualitative characteristics observed during problem-solving.

The study supported the use of the COSINE method for describing the differences in experts’ performances with individual sub-problems (ASR) and overall problems (CSR) in addition to the analysis of undergraduate data [44]. With this method, we were able to highlight experts’ chunking ability of sub-problems which is attributed to successful problem-solving strategies. Therefore, as expertise is developed, one can expect a smaller variability in ASR and CSR because sub-problems will be connected more successfully and less number of DD (Did not Know to Do) and DSE (Did Something Else) cases will be encountered.

Finally, lectures in general chemistry often are content-specific and while we work problems, there is a limited discussion of the pathways and the fact that true problems often will involve several strategies, often may have false starts, and generally will not result in a perfectly confined solution. Discussions of problem-solving procedure during lecture and noting that not every solution may be acquired from exactly the same pathway may help promote a stronger sense of perseverance for the undergraduate students.