Validation of a Pre- and Post-Evaluation Process: A Tool for Adult Training in Food Handling

Abstract

Education in food safety is a well-recognized health intervention, which allows the prevention of a wide range of diseases. Among the strategies of control and prevention of foodborne diseases, it is indicated that food safety education has the double advantage of having low costs and high potential effectiveness, as long as it is carried out with the active participation of food handling workers. In many countries, the Food Code has made compulsory the sanitary training of food workers. However, like in many other disciplines, food science educators receive minimal training on instructional techniques before becoming teachers. One of the important questions of the problem here presented is the issue related to the methodologies of pre-evaluation and final evaluation. We describe two indices to validate the training in food safety, which could be used for the quantification of educational intervention. The results show that a better learning process involves the active participation of both the students and the educators. We concluded that the evaluation process is more complex than the single instance of accreditation though a final evaluation.

1. Introduction

Worldwide, most of the reported foodborne disease outbreaks start from food prepared outside home, in both public and private foodservice establishments. It is widely recognized that inadequate food preparation practices and food service of community kitchens and dining rooms are strongly associated with poor microbiological quality. Education in food safety is a well-recognized (although still underutilized) health intervention which allows preventing a wide range of diseases with different etiologies [1]. For example, differences between restaurants in which outbreaks had occurred and those in which no outbreaks had occurred showed that a qualification in the kitchen management, including training in food hygiene, was the only factor that influenced the results [2].

Therefore, among the strategies of control and prevention of foodborne diseases, it is indicated that food safety education has the double advantage of having low costs and high potential effectiveness, as long as it is carried out with the active participation of food handling workers [3,4]. However, like educators in many other disciplines, food science educators receive minimal training on instructional techniques before becoming teachers [5].

Among pedagogic methodologies, test analysis techniques examine how the test items are performed as a set, whereas item analysis techniques investigate the performance of items considered individually, either in relation to some external criterion, or in relation to the remaining items on the test [6]. However, our experience indicates to us that some of the so called best practices in item and test analysis are not frequently used in food worker training design. Classically, test analysis refers to the application of statistical methods for the characterization of each of the items.

The aim of the present work was to analyze alternatives for the evaluation of the training process of food workers. We also aimed to find out whether only one exam is enough as a tool for the accreditation of this evaluation. To this end, two instances of evaluation were analyzed and some of their properties were quantified to find a criterion to improve them as training tools. In addition, we compared these two instances of evaluation in order to analyze the results obtained through the training process by the item analysis use. We found tools to provide food safety educators with good criteria to evaluate the performance of training activities.

2. Methods

2.1. Training Activity

The training activity is a full course on food handling topics. The methodology used for the development of the course was a series of educator-trainee sessions with workshop modality. The complete activity consisted of eight sessions. The subjects approached considered different aspects ranging from the particular rules of food hygiene to general topics in food security. Following previous experiences, different practical activities and the work with a reading guide were part of the course. Interactive dissertations consist of the topics’ exhibition for the teacher and the active incorporation of personal experiences, additional concepts, questions and doubts of the workers.

The sample consisted of 101 workers who had participated in interactive dissertations on the training for food handlers between 2005 and 2007 (42% men), aged 16–78 years old (average: 23 years old). The participants had varying degrees of literacy, and their education ranged from incomplete primary education to complete tertiary education.

The activities were developed in groups of a maximum of 20 people, with different levels of experience in food handling. The literacy level varied from a low level and little knowledge on food production or elaboration outside home to groups with professional skills. Only 18% were active workers, meaning that a few of them were employees with current and permanent activity in establishments of food production. The tested groups were not discriminated, taking the total number of students independently of their abilities.

Audiovisual resources designed to explain the topics were present in all the activities. These resources included videos, slides, illustrations and cartoons. Some hands-on activities like the use of thermometers, the washing of hands, the detection of microbes and the direct observation of parasites were also developed.

Two evaluations were carried out, one at the beginning and the other at the end of the course. They were called pre- and post-evaluation, respectively. The same kind of questions was used in the pre-evaluation and the post-evaluation test, being answered through multiple choice options. All the questions were developed based on previous own experience in food handling hygiene courses.

2.2. Exam Structure and Item Analysis

The set of questions for the pre-evaluation and the post-evaluation tests included the same topics, grouped into five areas: (a) general concepts of food hygiene; (b) storage and preservation of food; (c) acquisition, preparation and foodservice; (d) worker hygiene and (e) infrastructure hygiene.

The exam had a total of 51 multiple-choice questions (26 of them in the pre-evaluation test, and 25 in the post-evaluation test), including mostly primary concerns in food hygiene. Each one of the questions had five options to choose as an answer. In order to test specific aspects, some questions were about minor topics which were also discussed during training activities. The designs of questions considered the previous experience looking to achieve a good evaluation, including also some test distracting. Finally, some questions were repeated in both tests.

The students were all scored in a percentage scale, with Equation (1), where Gi is the number of questions answered correctly by student i, and T is a total number of questions. Then, Qi is the percentage of questions answered correctly by each student, which means that the Qi for student i is his score over 100 points.

(1)

In the test analysis techniques, two properties are frequently considered for the characterization of the questions: the Facility Value (Fv) and the Discrimination Index (Di).

The Fv quantifies the possibility of a question to be correctly answered by students. The item difficulty is simply the percentage of students taking the test who have answered it correctly. The algorithms to calculate the Fv are direct and intuitive. When all the items of the test are extremely difficult, the great majority of the test scores will be very low. When all the items are extremely easy, most of the test scores will be extremely high. In either case, test scores will show very little variability and will not represent real situations or differential behavior of students. That is why items with the highest and lowest Fvs are recommended to be removed.

(2)

In our work, the Fv for a question is a difficulty index which varies between 0 and 1, being 1 the value assigned to a question with maximum difficulty. The Fv for each question is described according to Equation (2), where Sc is the number of students who answered the question correctly, and n is the total number of students.

(3)

The Di quantifies the possibility that a question be correctly answered by the student that obtains a good score and, simultaneously, wrongly answered by a student that obtains a low score. This index shows the selectivity of a particular question. A question with high capacity of discrimination between the students was assigned with a high value of Di, whereas a question with low Di was assigned to a relatively confusing question, whose result could not be clearly associated with the general result of the evaluation. There are some algorithms to calculate discrimination indicators, based on statistical criteria and are generally not intuitive. High values of discrimination indices are recommended to be used in all the items.

(4)

In our work, the Di for a given set of exams was determined by the range between the highest and lowest score obtained in the group of students evaluated. According to the algorithm developed by Córica [7], Di could be described by Equation (3), where Ci is the Córica index for each student and each question, and n is the total number of students.

The final value of Di is described by Equation (4), which means that the value for Di for a single question is the Ci mean over all students (n).

3. Results and Discussion

Adult training implies the use of different tools that contribute to the development of participation instances. The tools should not only take into account the previous experience of the students, but also govern the activities of the educators. In this sense, the incorporation of the experience of the student in food handling is one of the most important tools in the development of training activities [8]. Also, many researchers [9] have evaluated how individual and organizational factors influence a range of immediate outcomes (i.e., worker’s knowledge, worker’s self-efficacy, worker’s competencies, and commitment to overcome organizational barriers) and ultimate outcomes (i.e., safe food handling practices, inspection scores, incidence of foodborne diseases).

Consequently, for a training program to be effective, it should be based on appropriate adult education theory and should incorporate some activities which support the development of relevant skills to real life situations, where workers can put the information into practice [10,11].

Significant differences have been observed in the understanding and acquisition of concepts, when the exams at the beginning and at the end of the courses are compared with pre- and post-evaluation techniques. Literacy is defined not only by someone’s writing and reading abilities, but also by the proper understanding and transmission of his own reality.

After carrying out training activities, such differences have been found to be related to variations in both the previous knowledge and literacy of the evaluated students [12]. In many countries, especially in those of low and middle income, marginal literacy of workers of food production establishments is a common circumstance. This is due to the fact that the education degree is not a condition for their employment. Moreover, a great amount of food is produced in non-formal establishments without state habilitation. Thus, we postulate that in order to generate effective actions in sanitary policy, the initiatives of training must consider these previous circumstances. However, contrary to this point of view, other authors suggest the use of homogeneous training codes [13].

It has been observed that workers with many years of working experience in the food sector or with previous food safety education give correct answers more frequently than those without [14]. In this sense, the workshop in which the best interaction between the individuals is verified, the knowledge of the group is benefited in two different ways. That means it is benefited by the previous individual experience of each student and by the concepts developed by the educator.

In Argentina, the sanitary training of workers which develop their tasks in direct contact with foods has been made compulsory, by the Argentinean Food Code since 2000. However, each province has regulated it with different levels of attention. Generally, Latin-American regional laws only emphasize the contents of food hygiene and the qualification degree of the educators. They should be specialized in food technology and food hygiene without the requirement for training on instructional techniques.

Nevertheless, as it has been previously discussed, the training of adults requires not only flexible contents that consider the methodology with which the specific concepts are developed but also other contents that articulate those specific concepts with the daily tasks of food handlers. As a result, the task of educators is more than the single presentation of contents. They have to make use of suitable pedagogic tools. This implies the handling of techniques of social communication and carrying out a previous diagnosis of the group to be trained [15].

Unfortunately, neither the aspect of the flexibility of contents nor the aspect of the pertinent methodologies is currently taken into account seriously. This is also observed in the regulations applied by official agencies that force the training of food handlers. This aspect is clear, for example, in many of the training manuals published by the State Health Authority of Buenos Aires, Argentina [16,17]. At present, this situation turns the norm useless.

Most studies assess the acquisition of knowledge of workers by analyzing pre- and post-training methods [18]. One of the most important aspects of the problem is the methodologies of diagnosis and of final evaluation. Is it possible to evaluate changes in knowledge, behavior, attitude and practices at the workplace? The written examination has been profusely used as an accreditation tool of food workers [19]. However, this tool mainly evaluates knowledge acquisition.

Nevertheless, what really allows a better evaluation of the results of the training is the follow-up of the changes in behavior in the effective accomplishment of sanitary procedures [3]. For this reason, it is necessary to find out the best validation method for an adequate written examination as the only accreditation tool [20].

In our knowledge, the general idea is to consider that a final written examination should be compulsory as a supporting document of the achieved training. Nevertheless, how this exam should be made is not specified far beyond the fact that it is a crucial document. This is true for example in the national application of the 21° Article of the Argentinean Food Code.

The training of food handlers in food safety should not be seen only as a simple training in techniques and procedures for the sanitary maintenance of food hygiene. It should also be considered under global concepts, including the social context and networks that support the practices, the insertion of the workers in the productive system, and the recovery of their role as learning subjects in a non-formal system.

The ability of the current food safety training programs to change behavior has been questioned. To increase the effectiveness of these programs, it is important to consider the current behavior of food handlers and to understand how this behavior interacts with their beliefs and the level of knowledge [21]. From a critical point of view, it is necessary to reflect on how valid and precise a way of transmission is to accredit the acquisition of ability and knowledge. Thus, it is important to analyze the relevance of the evaluation and accreditation methodology.

In this work, the scores (Qi) of the post-evaluation test were between 25 and 92, with an average of 67 and a standard deviation of 14.4, whereas those of the pre-evaluation test ranged between 19 and 73, with an average of 46 and a standard deviation of 13.9. The differences between the results of the post-evaluation and those of the pre-evaluation (∆qi) showed a value of 28 with a standard deviation of 16.4.

The item analysis of the different questions was carried out through Fv and Di values. Table 1 shows the questions with extreme Fv and Di values for the pre- and post-evaluation tests.

In regards to the performance of the set of questions, there was a linear and positive correlation between both indices in the pre-evaluation test (Figure 1). The Fv index ranged from 0.13–0.86, with an average of 0.54 and a standard deviation of 0.18, whereas the Di index ranged from 0.48–0.76, with an average of 0.61 and a standard deviation of 0.08.

In contrast, in the post-evaluation test, there was a clear linear and negative correlation between both indices for the set of questions considered (Figure 2). The values of Fv ranged from 0.06–0.75, with an average of 0.30 and a standard deviation of 0.20, whereas the values of Di ranged from 0.44–0.70, with an average of 0.62 and a standard deviation of 0.07.

Scores obtained in our post-evaluation data set reflect a strong educational intervention. A difference of almost 30 points in average was found in the performance of the students when compared with the pre-evaluation scores. However, Fv was notably different between the pre- and post-evaluation, whereas Di was not. It is argued that as Fv and Di are closely related, there is a better discrimination capacity with medium facility values (0.4–0.6). This relation can offer a lot of information about the competence among students [22].

We also observed an important modification in the linear relationship of Di and Fv (Figure 1 and Figure 2). Whereas the representation corresponding to the pre-evaluation shows a positive correlation between both indices, with an important dispersion in Fv, the representation of the stage of the post-evaluation shows a clear linear and negative correlation.

Table 1. Selected questions for the pre- and post-evaluation activities, with a respective qualification on Facility Value (Fv) and Discrimination Index (Di).

Question	Choices	Pre-		Post-
		Fv	Di	Fv	Di
Which is the most frequent cause of transmission of foodborne diseases	(a) Contamination by insects and rodents; (b) Contamination by poor handling of kitchen utensils; (c) Poor hand washing; (d) Poor general personal hygiene.	0.86	0.51	0.69	0.49
Why is it necessary to keep waste recipients closed and far from the handling food zone?	(a) Because it is easier; (b) Because waste is a contamination source; (c) Because waste produce bad smell; (d) Because waste near the food is not adequate.	0.20	0.53	0.06	0.69
Raw eggs …	(a) Should be washed and brushed before storage; (b) Should be washed and brushed before use; (c) Should be disinfected before use; (d) Should be stored and used without treatment.	0.86	0.55	0.75	0.44
The best way to wash your hands is …	(a) With hot water and soap; (b) With hot water and soap, then drying hands with a dishtowel; (c) With hot water and liquid soap, then drying with one-use paper; (d) With cold water and disinfectant soap.	0.35	0.49
To purchase tins, it is important:	(a) To purchase the larger ones in order to assure food quality; (b) To verify the sell-by date and that they do not have dents; (c) To always purchase products from large companies; (d) To be sure to keep products refrigerated.	0.13	0.51
How should foods be kept in a refrigerator?	(a) Cleanliness is not important, but the refrigerator must be cold; (b) Raw meat underneath or far from cooked meat; (c) So as to have easy access to dried foods; (d) Checking that the temperature is right.	0.72	0.58
Aluminum recipients should not be used …	(a) To preserve soups, stews or fruit juices which are to be used the next day; (b) To cook food with a high amount of sugar; (c) To prepare dough or other foods like bread; (d) To dress salads.			0.06	0.68
Which of the following sanitary practices are useful in the kitchen?	(a) Do not clean until all the dirt is dried up; (b) Use the equipment as least as possible; (c) Cleaning while working; (d) To keep waxed floors.			0.07	0.69

Table 1. Selected questions for the pre- and post-evaluation activities, with a respective qualification on Facility Value (Fv) and Discrimination Index (Di).

Question	Choices	Pre-		Post-
		Fv	Di	Fv	Di
Which is the most frequent cause of transmission of foodborne diseases	(a) Contamination by insects and rodents; (b) Contamination by poor handling of kitchen utensils; (c) Poor hand washing; (d) Poor general personal hygiene.	0.86	0.51	0.69	0.49
Why is it necessary to keep waste recipients closed and far from the handling food zone?	(a) Because it is easier; (b) Because waste is a contamination source; (c) Because waste produce bad smell; (d) Because waste near the food is not adequate.	0.20	0.53	0.06	0.69
Raw eggs …	(a) Should be washed and brushed before storage; (b) Should be washed and brushed before use; (c) Should be disinfected before use; (d) Should be stored and used without treatment.	0.86	0.55	0.75	0.44
The best way to wash your hands is …	(a) With hot water and soap; (b) With hot water and soap, then drying hands with a dishtowel; (c) With hot water and liquid soap, then drying with one-use paper; (d) With cold water and disinfectant soap.	0.35	0.49
To purchase tins, it is important:	(a) To purchase the larger ones in order to assure food quality; (b) To verify the sell-by date and that they do not have dents; (c) To always purchase products from large companies; (d) To be sure to keep products refrigerated.	0.13	0.51
How should foods be kept in a refrigerator?	(a) Cleanliness is not important, but the refrigerator must be cold; (b) Raw meat underneath or far from cooked meat; (c) So as to have easy access to dried foods; (d) Checking that the temperature is right.	0.72	0.58
Aluminum recipients should not be used …	(a) To preserve soups, stews or fruit juices which are to be used the next day; (b) To cook food with a high amount of sugar; (c) To prepare dough or other foods like bread; (d) To dress salads.			0.06	0.68
Which of the following sanitary practices are useful in the kitchen?	(a) Do not clean until all the dirt is dried up; (b) Use the equipment as least as possible; (c) Cleaning while working; (d) To keep waxed floors.			0.07	0.69

Figure 1. Relationship between Fv and Di in pre-evaluation questions. The linear correlation found shows a positive slope of 2.46432 (±0.17179) and a coordinate to the origin of −0.95341 (±0.10476). r² = 0.895540; SD = 0.06544; p < 0.0001; N = 26.

Figure 1. Relationship between Fv and Di in pre-evaluation questions. The linear correlation found shows a positive slope of 2.46432 (±0.17179) and a coordinate to the origin of −0.95341 (±0.10476). r² = 0.895540; SD = 0.06544; p < 0.0001; N = 26.

Figure 2. Relationship between Fv and Di in final evaluation questions. The linear correlation found shows a negative slope of 3.0363 (±0.20208) and a coordinate to the origin of 2.11685 (±0.1213). r² = 0.92698384; SD = 0.06278; p < 0.0001; N = 25.

Figure 2. Relationship between Fv and Di in final evaluation questions. The linear correlation found shows a negative slope of 3.0363 (±0.20208) and a coordinate to the origin of 2.11685 (±0.1213). r² = 0.92698384; SD = 0.06278; p < 0.0001; N = 25.

The negative slope in the linear correlation obtained in the set of questions of the post-evaluation stage indicates that the greater discrimination capacity and therefore the higher evaluation value could appear in the easiest questions, i.e., in those with small indices of difficulty. In a direct interpretation, we observed that the increase in the individual scores obtained in the final evaluation with respect to those obtained in the pre-evaluation stage are related to an individual assimilation of both the theoretical and practical concepts.

Nevertheless, considering the questions posed in both stages, the correlation founded between both indices was improved. This could reflect an integral modification of the ideas incorporated from the interactive dissertations along the course.

It is observed that, before the interactive dissertations, the students approached the subject from a multiplicity of directions, derived from previous and personal experiences. In contrast, at the evaluation stage, they showed a common directionality to answer the questions, which was reflected in the correlation obtained. If compared with previous studies carried out by others, our studied group was trained during a longer period [18,23]. This may indicate a harder formative intervention in the courses reported in our study influencing the assimilation of the knowledge and this should be taken into account to obtain objective and quantitative indices.

It is interesting to point out that the least difficult questions should be selected to define a set of questions with a high level of evaluation capacity during the post-evaluation stage. On the other hand, the hardest questions are not ideal to discriminate between better or worse trained students. In other words, the evaluation stress, associated with the need to accredit the knowledge and abilities acquired, does not need to be enhanced by the difficulty in the questions. This is because this difficulty does not contribute to the purpose of the evaluation, which is exactly to accredit this knowledge and abilities.

These results allowed us to design better pre- and post-evaluation tests, including objective parameters. We can describe a good pre-evaluation as that which includes a set of questions with a wide range of difficulty. The best value for the Fv index does not exist. Besides, a good post-evaluation test includes a set of questions with a high index of discrimination and a low to intermediate difficulty. Similarly, although ideal maximum values were preferred for Di, the best value for this index for an individual question does not exist either.

Then, we can use these tools to eliminate questions that do not meet minimum standards for difficulty and discrimination in both the pre- and post-evaluative stages. The final objective is to have better pre- and post-evaluation tests for each group of workers after an iterative methodology. This is finally an alternative procedure for testing and validation of evaluation instances.

The present study was performed within the framework of an activity defined by the requirements of the training of workers in the area of food handling. However, the parameters defined in this work can be generalized in other non-formal health training contexts. That is, they can be used to quantify the utility of the choose question for the pre- and post-evaluation of courses in which a modification in the concepts of the students in relation to a structured base of practical knowledge is desired.

The difficulty and discrimination indices used in the present work were good tools in this context. In this way, the marginal literacy of some participants can call into question the efficacy of the indices to establish changes in the participants’ knowledge. Nevertheless, these indexes are in use for qualifying the groups as a complex entity, and not only by making an individual evaluation of each worker. Therefore, the behavior of the group is first validated and then, if it is adapted, the individual behavior. At this point, it is necessary to mention that the workers generally recover their tasks in a group. In this way, they also reach or not the desired quality standards for the products that they prepare.

Without doubt, food safety training increased knowledge and improved attitudes about hand hygiene practices. However, very few studies have been conducted to evaluate the impact of food safety training on food handlers’ attitudes about good hand hygiene practices [24]. Our indices are not directly related to the changes in sanitary behavior of workers; it is necessary to find additional information for this essential evaluation.

The learning process that involves the active participation of both the students and educators in workshops allow us to consider the evaluation as a process of greater complexity than the single instance of accreditation by a final evaluation. This work is an approach to consider an integral point of view on the accreditation aspect, incorporating the use of objective evaluation indices of the training activity performance. This fact is not exclusive of other accreditation conditions.

4. Conclusions

The quantification of the properties of the questions used in the pre- and post-evaluations stages of training courses of food handling workers showed a remarkable and opposite correlation between discrimination and difficulty indices.

This phenomenon allows establishing an objective quantification of the degree of educational intervention over the groups of students. On the other hand, it allows validating the use of the proposed indices for this purpose. This might be used to qualify the performance of any training intervention in similar contexts, independently of the applied methodology.

We propose a methodology for a gradual selection of better questions for a pre- and post-evaluation for each group of food workers in an iterative procedure. That is a way to find a better methodology on sanitary food initiatives.

Finally, the heterogeneity of the groups studied and the forcefulness of the correlations obtained allow us to propose this method as a more general tool in the evaluation of the performance of adult training activities. Though, the methodology here discussed is an important general issue for any type of work training, and it is particularly significant in the application of food workers training because there is a particular context. That context includes: heterogeneous literacy conditions of participants, presence of workers with food experience on the subject, their role as sanitary agents, and how the relevance of training results in public health quality.