Evaluation of Human Perception Thresholds Using Knowledge-Based Pattern Recognition

Abstract

This paper presents research on determining individual perceptual thresholds in cognitive analyses and the understanding of visual patterns. Such techniques are based on the processes of cognitive resonance and can be applied to the division and reconstruction of images using threshold algorithms. The research presented here considers the most important parameters that affect the determination of visual perception thresholds. These parameters are the thematic knowledge and personal expectations that arise at the time of image observation and recognition. The determination of perceptual thresholds has been carried out using visual pattern splitting techniques through threshold methods. The reconstruction of the divided patterns was carried out by combining successive components that, as information was gathered, allowed more and more details to become apparent in the image until the observer could recognize it correctly. The study being carried out in this way made it possible to determine individual perceptual thresholds for dozens of test subjects. The results of the study also showed strong correlations between the determined perceptual thresholds and the participants’ accumulated thematic knowledge, expectations and experiences from a previous recognition of similar image patterns.

1. Introduction

New classes of cognitive systems described in [1] have been developed for image recognition and their meaningful classification. These systems have been dedicated to the semantic classification of different types of image patterns, including, for example, medical and biometric patterns. Cognitive systems are based on selected models of human visual perception and can also be used to evaluate individual visual perception thresholds for individuals who will make visual assessments of patterns and images in order to recognize and understand them [2,3].

The correct interpretation of a visual signal such as an image, as well as the ability to understand its meaning and carry out its correct evaluation, depends on parameters such as accumulated experience, acquired knowledge and generated expectations [4]. Therefore, this paper will present the research on the use of a knowledge-based approach to determine individual visual perception thresholds for people who recognize images and interpret selected classes of visual patterns. To this end, a methodology will be presented that allows for individual determinations of the perceptual thresholds necessary to fully understand and correctly interpret the evaluated image pattern, depending on one’s knowledge, experience and expectations of the images being evaluated.

In the research described, a method for determining the visual perception thresholds will be presented, depending on the knowledge and experience of the participants related to the analysis of selected types of images. Very often, when reproducing image patterns (e.g., after division or compression), we are faced with a situation where a given image is unreadable to the observer, and its correct interpretation requires certain associations or guesses. Improving the quality of such images or providing additional information (e.g., subsequent parts of a split image) can increase its readability. In such situations, at a certain stage of the image reconstruction, a visual assessment will be possible and will enable the correct interpretation of the image. It is worth noting, however, that when reproducing an image, people who have thematic knowledge or certain assumptions about the content or meaning of the image will be able to correctly recognize the image slightly earlier than people who do not have thematic knowledge or previous experience resulting from observing the same or similar images. This results from the model of human visual perception called knowledge-based perception, in which it is possible for the observer to recognize only patterns that he/she has seen before [1].

The main motivation for the research undertaken is the desire to determine the relationship between the individual threshold of visual perception of each person, depending on his/her thematic knowledge and the experience in recognizing different visual patterns. This is an extremely important task from the point of view of the cognitive processes taking place in the human mind when observing and understanding images. To this end, an approach based on cognitive resonance models will be presented, using both expectations and acquired knowledge during the interpretation of new image patterns. The ability to determine the aforementioned thresholds of visual perception makes it possible to quantify the relationship between the accumulated knowledge and expectations and the actual observed features of recognized patterns. The results in this regard can be of great importance in the development of new solutions in the field of machine learning methods and computer techniques for understanding image patterns.

2. Perceptual Inference Model

As already mentioned, the perceptual inference approach to assessing and understanding the semantics of image patterns is based on the model of human visual perception, i.e., knowledge-based perception [5]. This model is based on the assumption that image recognition and its correct understanding are possible when the observer has certain experiences, expectations or knowledge related to the content of the assessed image. Without this knowledge and previous experience, it is not possible to correctly classify or understand the image, because it is a completely new pattern to be recognized. Previously gained experience allows you to compare new observed patterns with previously recognized ones and refer to them in the process of understanding [6]. In the described model of knowledge-based perception, when trying to recognize patterns that are already known but are being shown in an unusual situation (due to the expectations associated with such a pattern), it is also difficult to make a correct classification, because the expectations generated in the cognitive model are completely different. This is due to the fact that in the visual perception model, when trying to understand a new pattern, parts of the human mind generate certain expectations as to the semantic content and meaning of the new image.

The expectations generated in this way primarily result from previous experiences in pattern recognition, but also from general thematic knowledge of a given area to which the newly assessed image refers. When expectations are subconsciously generated, the visual system also registers the actual features of the observed object and compares them in the process of cognitive resonance with the generated expectations and previously acquired knowledge [4]. As a result of resonance, those features that are consistent with expectations gain importance, while those that are inconsistent with expectations lose their importance. In this way, cognitive reasoning processes allow us to understand the observed image. It is therefore clear that the ability to correctly understand and evaluate images largely depends on the knowledge and experience gained while recognizing previous patterns. In such a situation, it is possible to determine the perception threshold at which an observer with appropriate experience and knowledge will be able to correctly assess and semantically understand the analyzed images [7].

By comparing important image features with acquired knowledge and previous experiences, cognitive resonance methods also allow for the prediction of future possibilities in the area of understanding image patterns. Thanks to such properties, they can be used in research on human intelligence and AI. Since the process of inferring meanings is based on the comparisons of actual features with patterns representing previous experiences, the complexity of the comparative procedure is relatively low and is at the level of a low-degree polynomial.

A knowledge-based model of visual perception and cognitive processes can be implemented in computer systems using parsers for formal grammars. Such grammars will store the accumulated knowledge in the sets of derivation rules, which the system will then be able to compare with the actual, observed features. The comparison will be performed using a syntactic analyzer implemented for the defined formal grammar.

Of the many classes of formal grammars available, context-free grammars are the best suited for implementing cognitive resonance processes. They have great descriptive power, and at the same time, there are deterministic parsers for them with a polynomial complexity. This is of great importance when creating and effectively implementing cognitive analysis systems. An additional advantage of using this class of formal grammars is the ability to use programs to automatically analyze the rules of the defined grammar and create parsers for this class of grammars.

The next part of the work will present a method of determining individual perception thresholds for people who have different experiences in visual pattern recognition, accumulated knowledge and expectations related to the analyzed images.

3. Visual Pattern Understanding

The accumulated knowledge, experience and expectations regarding the contents of the analyzed images are of the greatest importance in the processes of understanding images and their classification. Thematic knowledge and accumulated experiences may refer to a specific area of knowledge to which the assessed patterns may also belong [8]. An example here would be medical diagnostic images showing various structures and disease lesions occurring in them. The assessment of the meaning of such images is only possible by medical professionals, who should have not only knowledge related to the structures visible in the image, but also diagnostic experience in assessing the pathological changes that may appear there.

The same situation occurs if we recognize other image patterns, including photos, posters, landscapes, cars, famous persons, etc. Each image, regardless of the thematic area it represents, will contain a lot of information for the selected observers, while for others, it may be completely new and incomprehensible. This applies to situations in which images refer to areas in which observers do not have sufficient knowledge.

Knowledge and experience therefore play a large role in the assessment of paintings or visual graphics, and also allow us to understand the essence of the observed image or work of art. In the case of artistic paintings, the lack of experience and knowledge means that we can only enjoy the impressions related to colors.

Figure 1 presents two selected heart medical visualizations. The first one is very popular and presents heart muscle on echocardiography images (Figure 1A), but the second is much more difficult to recognize and presents the left main coronary artery in a color-coded, transthoracic Doppler echocardiography (Figure 1B).

The assessment of the meanings of these two images, as well as their medical interpretations, mainly depend on specialist medical knowledge, but also on experience and expectations related to the content of the assessed images.

Without specialized knowledge of such images, it is impossible to understand what they represent and recognize them correctly. If the observer has certain expectations but does not have specialized knowledge, the observed images can only be assessed in a general way from an aesthetic point of view. This would be the case if they were assessed by people without medical knowledge or by people who were unfamiliar with diagnostic imaging [8]. Their assessment would mainly consist of the assumption that they resemble the structure of the heart muscle. Therefore, most people without medical knowledge recognize the first image from the two images presented in Figure 1 as being typical of the structure of the heart. A slightly different situation occurs when the person assessing the images has specialized medical knowledge and skills in diagnostic imaging. By looking at these images, the person will be able to correctly recognize that both structures represent health problems, even if this person were to be looking at a blurry visualization. Specialized knowledge, without a priori expectations, will enable the correct recognition of the image only if it is easy to interpret, i.e., without any modifications and distortions [9].

It is therefore clear that the ability to understand an image depends primarily on subject knowledge, but also on certain assumptions made before the visual assessment process begins. Therefore, if we consider a situation in which an image is subjected to a threshold division operation, as a result of which we obtain several visual parts that cannot be interpreted independently, it is also possible to reconstruct it by adding the separated parts [3]. This combination of parts causes, at a certain stage, important fragments of its content to begin to appear in the incomprehensible image, which enable the observer to correctly recognize the entire image. This is only possible if you have specialist knowledge and expectations related to the possible contents of the image. In this way, it is possible to determine the individual perception threshold for a given observer; by reaching this threshold, he/she will be able to correctly recognize the image he/she is viewing, while others will not be able to do so yet and will have to keep adding more parts of the split image to fully reproduce its content.

An example of threshold division and the reconstruction of an image is presented in Figure 2.

In Figure 2, we present a well-known Mona Lisa painting. The determined visual parts of the picture are visible as seven grey strips in the generated shadows frame.

The seven visual parts visible in Figure 2 are the so-called shadows obtained by dividing the image using the threshold division procedure. In Figure 2, these parts are visible as grey bars that make it impossible to determine the content of the image. In threshold algorithms, assembling several such shares allows for the reconstruction of the original image, provided that the required number of such shares are combined. During the image restoration, adding subsequent shares allows us to obtain the original image in an increasingly better quality, as presented in Figure 2. At the bottom, several stages of reconstruction are shown for the original image. Depending on the number of collected parts, it is possible to see more and more details [10]. The left blurred image is too difficult to recognize immediately. The middle picture allows us to identify the Mona Lisa contour. The image on the right is a fully reconstructed picture, in which one can easily recognize the Mona Lisa.

The complexity of the image division and reconstruction procedures depends on the algorithms used. If visual cryptography techniques are used for this purpose, the complexity will remain polynomial. However, if cryptographic threshold division techniques are used, the complexity of the entire procedure will also be at the polynomial level.

4. Methods

In the conducted experiments, 60 people representing various areas of thematic specialization took part to demonstrate the dependence of visual perception thresholds on the knowledge and expectations regarding the contents of the images.

The respondents were 60 students (30 men and 30 women) aged up to 24 years and represented various fields of study, i.e., 12 art students, 20 technical engineering students, 18 natural sciences students, 5 history students and 5 social sciences students. The level of expertise for all participants was at the level of higher academic studies in various fields of study. This means that each person was comfortable recognizing images correctly and quickly that were thematically related to his or her scientific field, while recognizing images from other fields required knowledge of the basics of another discipline or a complete reconstruction of a previously divided image that was already easy to recognize.

The persons taking part in the experimental research were shown sets containing 30 different images in two stages. The set of 30 presented images included images thematically related to the fields of study of some of the people taking part in the test, which meant that some of them were more easily recognized by the people with knowledge of these subjects, and other images were more difficult to recognize due to the participants’ lack of specialized knowledge or thematic knowledge related to their content.

In the conducted experiments, each image had been previously divided into 10 image parts using threshold methods. The reconstruction of each image consisted of adding subsequent constituent parts to those shown earlier, until the original image was completely recreated. When the participant could already recognize the content of the image, he or she informed us of it.

Experiments with the division and reconstruction of images were carried out using a program specially implemented for this purpose, which is presented in Figure 2. During the first series of presentations, none of the participants identified the contents of the recognized images, which were completely new to them and could become familiar to them for the first time. At this stage, the participants had no expectations regarding the contents of the images and could not guess what they would represent.

In the second series of presentations, all images were assessed by subsequent participants in such a way that the subsequent components were added to the reproduced images, thus revealing further details with varying degrees of detail. When the participant could correctly recognize the entire image, his or her individual visual perception threshold was determined. Because the images represented different thematic areas, their correct recognition required knowledge from different fields, so each participant in the experiments had a different visual perception threshold for different images. These thresholds depended on the subject knowledge of a given participant, which also included paintings.

In the second stage of the experiment, participants recognized the same images, but shown in a random order. At this stage, they already had some knowledge of what the threshold-reproduced images might contain. The expectations had a significant impact on the determined perception thresholds, and recognition usually occurred much earlier than when the first series of images were shown.

5. Results

To conduct experiments on a selected group of participants, specially developed software implemented in the JAVA language was used, an example of which is presented in Figure 2. This system allows for the loading and threshold division of images, creating a selected number of parts into which the image was divided. These parts are visible as gray bars located in one of the application windows in Figure 2. The system then allows for the restoration of the original image by showing each user the divided parts and gives the ability to add more of them to restore the original image. Adding more image shares allows the user to display an increasingly accurate image without distortion or blurring. At a certain stage, the image becomes clear enough to be recognized correctly, and at this point, the visual perception threshold for the participant is set.

The general methodology for determining visual perception thresholds is as follows:

• The selection of the algorithm implementing the threshold distribution of images: For this purpose, it is possible to use any cryptographic information partitioning algorithm that also allows for the division of image data;
• The implementation of a procedure for the distribution of divided data, with the verification of its correct operation by the splitting and restoration of shared images;
• The determination of the thresholds of visual perception depending on the levels of knowledge possessed by the participants of the protocol: For this purpose, two experiments are to be carried out on a selected group of participants, in which each participant will observe the images played sequentially and provide answers as to what they represent. The correct answer will determine the perceptual threshold of a given participant. In this way, the minimum visual perception thresholds for a person’s image recognition ability will be determined. In the next demonstration of the same set of visual data, the participants already have the knowledge of what the presented and shared images can refer to. Thus, during the playback of the image, they can correctly recognize the image in question at an earlier stage, and it is possible to do so with a lower perceptual threshold.

Figure 3 presents a graph of the dependence of the determined perception thresholds obtained during image classification by individual participants on the knowledge and expectations of the participants.

In image recognition tests using threshold procedures, the quality of the observed image plays a very important role. Decreasing the quality of the images may result in an increase in the threshold values and lead to a situation in which the participant must have more shares to properly recognize the image (Figure 3). Therefore, it is important to base experiments on clear images without noise and interference, which make their analysis difficult. On the other hand, the ability to recognize an image should not depend on the content of the image, because the correct recognition of a visual pattern is only possible when the observer has previous experience in recognizing similar images [11].

When determining perception thresholds, increasing the number of iterations during the presentation of the images allows participants to improve their memory and gives the ability to recognize specific patterns even after a long time. In this way, you can also increase the recognition rates of larger sets of images, which will become more recognizable after many presentation iterations.

6. Discussion

As a result of the research, it was shown that there are strong relationships between the participants’ visual perception thresholds and their knowledge related to the content of the images. Similar relationships have also occurred between previous experiences and expectations generated during image recognition.

Having specialized knowledge lowers participants’ perception thresholds by about 25%, especially in situations where they interpret images whose subject matters are related to their knowledge (Figure 4).

Thanks to this knowledge, test participants were able to recognize the image earlier and correctly, even when the image was blurry or showed select details from its content. A similar situation occurs when certain expectations arise before recognition but related to the content of the image. Generated expectations may also lower perception thresholds by an average of 28%. This is because experience tells participants what may be hidden in the images they judge, even when not all the details are shown.

The above-mentioned factors, i.e., thematic knowledge and generated expectations, have a direct impact on lowering the perception thresholds at which the observer completely and correctly recognizes the observed image. This happens even when it is blurred or does not fully show important details.

As a result of the experiments carried out, it was also shown that having only specialist knowledge, but without expectations, allows you to correctly recognize an image when it is fully reproduced using all its components. Having expectations alone, without substantive knowledge, may or may not be enough to correctly evaluate the image, because it may depict an object that is simply unknown to a given person, and therefore, he or she cannot classify it correctly. This situation only gives the opportunity to generally evaluate the image in terms of its content and determine whether the evaluator likes it.

The thresholds for visual perception were determined in empirical studies. This means that they can depend on the group of participants studied, as well as their size. In such studies, the representativeness of the group plays an important role, which depends on such characteristics as the age of the study participants, experience, expertise, etc. However, the main purpose of our experiments was to test the relationship between experience and expertise and the threshold values at which participants can correctly interpret imaging patterns. It was also an important parameter to see how much influence the first display of patterns has on the ability to recognize incomplete patterns in the second display of images. According to the study, these capabilities increase by more than 25%.

7. Conclusions

The purpose of this paper was to conduct an experimental study to determine individual human visual perception thresholds, using a cognitive approach based on thematic knowledge and expectations. As a result of the experiments conducted, the existence of strong correlations between visual perception thresholds and the knowledge, experience and expectations generated during image recognition were confirmed. It is noteworthy that in the present study, a threshold model of visual perceptual reproduction was applied during the conducted experiments. This was realized through the procedures of threshold data partitioning and the ability to reproduce the data by assembling successive shares so that the reproduced image became increasingly clear. Once the participant could correctly recognize the image (its content or meaning), the perceptual threshold for the user was determined. The greater his or her experience in recognizing such images or knowledge related to previous observations of such patterns, the more likely that correct recognition was possible at an earlier stage. This was confirmed by the results of the conducted experiments on the test data.

The topic presented in the work is extremely important and interesting not only from a scientific perspective, but also from a practical point of view. The obtained research results can be used in teaching processes and educational or training activities. They can also be used to develop computer cognitive systems based on artificial intelligence and vision systems. Such work is being carried out in the area of autonomous and humanoid robots, which can prove very useful in many areas of human activity including face recognition, hand gestures and survival activities [12,13,14,15,16,17].

In the context of the presented research, an extremely interesting issue is the analysis of the way of perceiving important details when recognizing images, which determine the possibility of the correct understanding of images [18,19,20]. A cognitive analysis may allow for the detection of the so-called distinctive features that clearly determine the content of the examined image. During an analysis using threshold division algorithms, it is the determination of distinctive features that influences the moment of recognition and the determination of perception thresholds. Future research will address the issue of the automatic extraction of distinctive features using human–machine visual interaction interfaces.