The Uncertainty–Certainty Matrix for Licensing Decision Making, Validation, Reliability, and Differential Monitoring Studies

Abstract

This research article proposes the use of an uncertainty–certainty matrix (UCM) for licensing decision making in the human services, which is the decision to issue a license to operate. It is a proposed study protocol and conceptual framework; it is not an empirical study. It shows how the matrix can be used in rule decision making and how it clearly shows when decision making has gone awry when bias is introduced into the decision making. It is also proposed to be used to make decisions in differential monitoring and in validation and reliability studies. This proposal presents a potential blueprint on how the UCM can be used within human services licensing as a decision-making tool.

1. Introduction

This research proposal takes the Contingency Table, which is a well-known metric in the statistical decision-making research literature [1], and refocuses it on regulatory science within the context of the definition of regulatory compliance and licensing measurement. It also deals with the policy implications of this particular metric. In this study protocol, it is proposed that the Uncertainty–Certainty Matrix (UCM) is a fundamental building block to licensing decision making from a measurement perspective. The Contingency Table, as demonstrated by a 2 × 2 matrix, is utilized in regulatory compliance and is the center piece for determining licensing key indicator rules [2], but it is also a core conceptual framework in licensing measurement and ultimately in program monitoring and reviews [3].

The reason for selecting this matrix is the nature of licensing data: it is binary or nominal in measurement. Either a rule/regulation is in compliance or out of compliance. Presently, most jurisdictions deal with regulatory compliance measurement in this nominal level or binary level. There is to be no gray area; this is a clear distinction in making a licensing decision about regulatory compliance. The UCM also takes the concept of Inter-Rater Reliability (IRR) a step further in introducing an uncertainty dimension that is very important in licensing decision making which is not as critical when calculating IRR. Inter-Rater Reliability is a real concern in the human services licensing field in that in many cases it is difficult for jurisdictions to maintain a high degree of consistency when comparing individual licensing inspectors to each other. Part of the problem is a fundamental measurement issue; it is hoped that the addition of the UCM will help to mitigate this problem [4]. Licensing measurement is dominated by nominal measurement: either a rule is in compliance or it is out of compliance. A proposal has been suggested in which an ordinal scale based upon licensing rule violations would be utilized called the Regulatory Compliance Scale (RCS) [3]. This new RCS scale shows promise, but it needs additional validation studies in order to be used on a regular basis for making human services licensing decisions (2a).

The reason for even suggesting this matrix is the high level of dissatisfaction with the levels of reliability in the results of program monitoring reviews as suggested in the previous paragraph. If the dissatisfaction was not so pronounced, it would not be an issue, but with it being so high, the field of licensing needs to take a proactive role in determining the best possible way to deal with increasing inter-rater reliability among licensing inspectors. Hopefully, this organizational schema via the UCM Matrix will help to think through this process related to licensing measurement and monitoring systems. The author has been conducting regulatory compliance studies for the past 50 years and has determined that the validity and reliability of these studies needs a more robust model for making licensing decisions via more accurate measurements of regulatory compliance. This led to the creation and proposing of the UCM Metric [5,6,7].

Over the past 50 years, it has been well documented by the National Association for Regulatory Administration (NARA) how the licensing field has changed in moving from a one-size-fits-all licensing and monitoring approach to one of differential or targeted licensing and monitoring (https://www.naralicensing.org/key-indicators, accessed on 24 April 2025). NARA has led this transition in the human services licensing and regulatory administration field, which has produced a much more productive, effective, and efficient licensing inspection system. The UCM and RCS are the latest pieces in the puzzle to accomplishing this new licensing decision-making framework.

The key pieces to the UCM are the following: the decision (D) regarding regulatory compliance and actual state (S) of regulatory compliance. Regulatory Compliance of individual Rules: Plus (+) = In-compliance, or Minus (−) = Out of compliance. As such, the matrix can be built as follows (

Table 1. Uncertainty–Certainty Matrix (UCM) Logic Model.

UCM Matrix Logic		Decision (D) Regarding	Regulatory Compliance
		(+) In Compliance	(−) Not In Compliance
Actual State (S) of Compliance	(+) In Compliance	Agreement	Disagreement
	(−) Not In Compliance	Disagreement	Agreement

):

The above UCM matrix demonstrates when agreement and disagreement occur, which establishes a level of certainty (Agreement Cells) or uncertainty (Disagreement Cells). In a perfect world, there would only be agreements and no disagreements between the decisions made about regulatory compliance and the actual state of regulatory compliance. However, from experience, this is not the case. This is based up reliability testing carried out in the human services licensing research field in which a decision is made regarding regulatory compliance with a specific rule or regulation, and then that is verified by a second licensing inspector observer who generally is considered the measurement standard.

Disagreements raise concerns in general, but the disagreements are of two types: false positives and false negatives. A false positive is when a decision is made that a rule/regulation is out of compliance when it is in compliance. This is not a good thing, but its twin disagreement is worse. With false negatives, it is decided that a rule/regulation is in compliance when it is out of compliance. False negatives need to be avoided because they place clients at extreme risk more so than a false positive. False positives should also be avoided, but it is more important to deal with the false negatives first before addressing the false positives.

2. Uncertainty–Certainty Matrix for Validation and Reliability Studies

This part of the research proposal is to explore the possibility of utilizing the Uncertainty–Certainty Matrix (UCM) as depicted in Table 1 in validation and reliability studies in licensing decision making. The UCM has been proposed for use in licensing decision making, but this would be an extension of this thinking to studies that involve validating licensing decisions, such as when key indicators/predictor rules are used in comparison with comprehensive reviews of rules [5] and in reliability studies to determine individual inspector bias in regulatory compliance [8,9].

The basic premise of the UCM is that individual decision making matches reality. When it comes to regulatory compliance decision making. a 2 × 2 matrix can be drawn with the possible outcomes as indicated in the following table (

Table 2. Uncertainty–Certainty Matrix (UCM) Logic Model applied to Validation Studies.

UCM Matrix Logic	For Validation Studies	Decision Regarding	Regulatory Compliance
		(+) In Compliance	(−) Not In Compliance
Actual State of Compliance	(+) In Compliance	Agreement (++)	Disagreement (+−)
	(−) Not In Compliance	Disagreement (−+)	Agreement (−−)

), which is based upon the logic of Table 1.

In using this table, the hope is that the decision regarding regulatory compliance matches the actual state of compliance where the coefficient is as close to +1.00 as possible; in other words, perfect agreement. The agreement cells are heavily weighted (++) and (−−). We do not want to see all the cells, both agreement and disagreement cells, equally weighted (++), (+−), (−+), (−−). That would indicate a random response rate and a coefficient close to 0.00.

However, there is another possibility which involves bias on the part of the licensing inspector in which they have certain biases or tendencies when it comes to making regulatory compliance decisions about individual rules. Consequently, it is possible that decisions made regarding regulatory compliance could be either overall (+) positive In-Compliance or (−) negative Not-In-Compliance when in reality, the actual state of compliance is more random.

The UCM can be used for both reliability and validity testing as suggested in the above table (Table 2). For validity, false positives (+−) and negatives (−+) should either be eliminated or reduced as well as possible, and the remaining results should show the typical diagonal pattern as indicated by the agreement cells.

For reliability, the same pattern should be observed as in the validity testing above, but there is an additional test in which bias is tested for. Bias is ascertained if the patterns in the results indicate a horizontal or vertical pattern in the data with little or no diagonal indication. Bias can be found at the individual inspector level, as well as at the standard level or the actual state of compliance. This could provide a helpful visual for licensing administrators regarding how decisions are being made about program regulatory compliance in the field.

In both reliability and validity testing, random results in which each of the cells are equally filled are not desirable either. Obviously, additional training involving licensing inspectors would need to occur in order to make the data collection efforts both reliable and valid. Monitoring of regulatory compliance history data would need to occur on an ongoing basis to make sure that biases did not return or if new biases developed within the regulatory compliance system.

The following

Table 3. Valid and Reliable Results.

Valid and Reliable Results	(+) In Compliance	(−) Not In Compliance
(+) In Compliance	Agreement (++)	Disagreement (+−)
(−) Not In Compliance	Disagreement (−+)	Agreement (−−)

,

Table 4. Random Results.

Random Results	(+) In Compliance	(−) Not In Compliance
(+) In Compliance	Agreement (++)	Disagreement (+−)
(−) Not In Compliance	Disagreement (−+)	Agreement (−−)

,

Table 5. Positive Bias Results Individual Assessor.

Positive Bias Results Individual	(+) In Compliance	(−) Not In Compliance
(+) In Compliance	Agreement (++)	Disagreement (+−)
(−) Not In Compliance	Disagreement (−+)	Agreement (−−)

,

Table 6. Negative Bias Results Individual Assessor.

Negative Bias Results Individual	(+) In Compliance	(−) Not In Compliance
(+) In Compliance	Agreement (++)	Disagreement (+−)
(−) Not In Compliance	Disagreement (−+)	Agreement (−−)

,

Table 7. Positive Bias Results Standard.

Positive Bias Results Standard	(+) In Compliance	(−) Not In Compliance
(+) In Compliance	Agreement (++)	Disagreement (+−)
(−) Not In Compliance	Disagreement (−+)	Agreement (−−)

and

Table 8. Negative Bias Results Standard.

Negative Bias Results Standard	(+) In Compliance	(−) Not In Compliance
(+) In Compliance	Agreement (++)	Disagreement (+−)
(−) Not In Compliance	Disagreement (−+)	Agreement (−−)

depict the above relationships with results highlighted in red:

Table 3, Table 4, Table 5, Table 6, Table 7 and Table 8 demonstrate the different results based upon individual response rates when making regulatory compliance decisions about rules. Table 5 is what needs to be attained and Table 4, Table 5, Table 6, Table 7 and Table 8 need to be avoided. Only in Table 3 are false negatives and positives eliminated or avoided. In Table 4, Table 5, Table 6, Table 7 and Table 8, false negatives and/or false positives are introduced, which is not desirable when making validity or reliability decisions.

Table 4 results clearly indicate that a great deal of randomness has been introduced in the regulatory compliance decision making in which the individual licensing inspector decisions do not match reality. Table 5 and Table 6 demonstrate bias in the decision-making process either positively (inspector always indicates in compliance) or negatively (inspector always indicates out of compliance). It is also possible that the standard being used has bias built into it; this is less likely but is still a possibility. The results in Table 7 and Table 8 demonstrate where this could happen.

All these scenarios need to be avoided and should be monitored by agency staff to determine if there are patterns in how facilities are being monitored.

3. Uncertainty–Certainty Matrix for Differential Monitoring Studies

The purpose of this part of this research proposal is to explore the possibility of utilizing the Uncertainty–Certainty Matrix (UCM) not only in validation and reliability studies in licensing decision making, but also with differential monitoring studies. The UCM has been proposed for use in licensing decision making, but this would be an extension of this thinking to studies that involve validating licensing decisions, such as when key indicators are used in comparison with comprehensive reviews of rules and in the development of risk rules as part of the risk assessment methodology [4]. This new Differential Monitoring 2 × 2 Matrix can also be used to depict the relationship between full and substantial regulatory compliance and the nature of rulemaking.

The basic premise of the DMM: Differential Monitoring Matrix is similar to the original thinking with the UCM Matrix Logic as depicted in Table 1, but there are some changes in the formatting of the various cells in the matrix (see

Table 9. DMM—Differential Monitoring Matrix.

DMM Matrix	High Group (+)	Low Group (−)
(+) Rule is In Compliance	(++)	(+−)
(−) Rule is Not In Compliance	(−+)	(−−)

). When it comes to regulatory compliance decision making, a 2 × 2 matrix can be drawn with the possible outcomes as is indicated in Table 9 where each individual rule is either in (+) or out (−) of compliance. Additionally, there is the introduction of a high regulatory compliant group (+) and a low regulatory compliant group (−), which is different from the original UCM.

By utilizing the format of Table 9, several key components of differential monitoring can be highlighted, such as key indicators and risk assessment rules, as well as the relationship between full and substantial regulatory compliance.

Regulatory compliance is grouped into a high group (+); generally, this means that there is either full or substantial regulatory compliance with all rules. The low group (−) usually has 10 or more regulatory compliance violations [4]. Individual rules being in (+) or out (−) of regulatory compliance is self-explanatory.

Table 10. Key Indicators/Predictor Rules.

Key Indicators	High Group (+)	Low Group (−)
(+) Rule is In Compliance	(++)	(+−)
(−) Rule is Not In Compliance	(−+)	(−−)

,

Table 11. Risk Rules/Place Clients at Increased Risk.

Risk Rules	High Group (+)	Low Group (−)
(+) Rule is In Compliance	(++)	(+−)
(−) Rule is Not In Compliance	(−+)	(−−)

,

Table 12. Full Compliance with All Rules.

Full Compliance	High Group (+)	Low Group (−)
(+) Rule is In Compliance	(++)	(+−)
(−) Rule is Not In Compliance		(−−)

,

Table 13. Substantial Compliance with All Rules.

Substantial Compliance	High Group (+)	Low Group (−)
(+) Rule is In Compliance	(++)	(+−)
(−) Rule is Not In Compliance	(−+)	(−−)

,

Table 14. Very Difficult Rules.

Very Difficult Rule	High Group (+)	Low Group (−)
(+) Rule is In Compliance	(++)	(+−)
(−) Rule is Not In Compliance	(−+)	(−−)

,

Table 15. Poor Performing Programs.

Poor Performing Programs	High Group (+)	Low Group (−)
(+) Rule is In Compliance	(++)	(+−)
(−) Rule is Not In Compliance	(−+)	(−−)

and

Table 16. Terrible Rule.

Terrible Rule	High Group (+)	Low Group (−)
(+) Rule is In Compliance	(++)	(+−)
(−) Rule is Not In Compliance	(−+)	(−−)

below demonstrate the following relationships:

Table 10 depicts the key indicator relationship between individual rules and the high/low groups as indicated in red. In this table, the individual rule is in compliance with the high group and is out of compliance with the low group. This result occurs on a very general basis and should have a 0.50 coefficient or higher with a p value of less than 0.0001.

Table 11 depicts what most rules look like in the 2 × 2 DMM. Most rules are always in full compliance since they are standards for basic health and safety for individuals. This is especially the case with rules that have been weighted as high-risk rules. Generally, one never sees non-compliance with these rules. There will be a substantial number of false positives (+−) found with high-risk rules, but that is a good thing.

Table 12 depicts what happens when full compliance is used as the only criterion for the high group. Notice that the cell right below (++) is eliminated (−+). This is highly recommended since it eliminates false negatives (−+) from occurring in the high group. As is seen in Table 12, when substantial compliance is used as part of the high group sorting, false negatives are re-introduced. If possible, this should be avoided; however, in some cases, because of the regulatory compliance data distribution, this is not always possible where not enough full compliant programs are present.

Table 13 depicts what occurs when substantial compliance is used as part of determining the high group. False negatives can be reintroduced into the matrix which needs to be either eliminated or reduced as best as possible. If substantial compliance needs to be used in determining the high group, then there is a mathematical adjustment that can be made, which will impact the equation and essentially eliminate false negatives mathematically.

Table 14 depicts what happens if the individual rule is particularly difficult to comply with. Both the high performers as well as the low performers are out of compliance with the rule.

Table 15 depicts a situation where the programs are predominantly in a low group with few at full or substantial regulatory compliance, which is indicative of poor performing programs. Very honestly, this is generally not seen in the research literature, but it is a possibility and one to be in tune with.

Table 16 depicts a terrible individual rule which predicts just the opposite of what we are trying to do with programs. Obviously, this rule would need to be rewritten so that it fits with the essence of regulatory compliance in helping to protect individuals.

The following Table 10, Table 11, Table 12, Table 13, Table 14, Table 15 and Table 16 depict the above relationships with results highlighted in red.

Table 10, Table 11, Table 12, Table 13, Table 14, Table 15 and Table 16 demonstrate the different results based on the relationship between individual regulatory compliance and if a program is either a high performer or a low performer. These tables are provided as guidance for understanding the essence of differential monitoring and regulatory compliance, which has various nuances when it comes to data distribution. This research proposal for a UCM hopefully can be used as a guide in determining from a data utilization point of view how to make important regulatory compliance policy decisions, such as which rules are excellent key indicator rules, which are performing as high risk rules, the importance of full compliance, what to do when substantial compliance needs to be employed, are there difficult rules to comply with, how well are programs performing, and do we have less than optimal rules that are in need of revision.

4. Conclusions

The Uncertainty–Certainty Matrix (UCM) should provide a useful tool for assessing the effectiveness of licensing decision making in the human services via validation and reliability studies within differential monitoring systems by visually inspecting cell proportions to determine if the appropriate results are depicted in the above matrices.

It is hoped that licensing researchers and regulatory scientists will experiment with it and test it out in different arenas beyond early care and education programs. It appears to have broad applicability across regulatory disciplines. The matrix has helped to identify the need to address false positives and negatives in the human services licensing decision-making process which undermines the effort of protecting clients.

The UCM also appears to provide a framework to identify reliability issues across licensing inspectors carrying out evaluations of individual programs. This issue of reliability is a big issue in the human services licensing field where there is a great deal of inconsistency when it comes to measuring regulatory compliance [10,11,12]. The UCM could be applied to existing regulatory compliance history data to determine if bias is present or not. It provides a clear visual demonstration of when regulatory compliance history data have gone awry and are not performing as they should. This can be a useful tool for licensing administrators in making changes to their overall licensing system, as well as for which individual rules/regulations/standards are most effective in protecting clients or might need revision.

The major limitation of the UCM is that as of this writing, it has not been empirically tested to see if this conceptual framework is really helpful to licensing policymakers and researchers. The UCM is a theoretical model at this point that needs to be verified. At the same time, it holds promise for the human services licensing field because the field as it relates to regulatory science has a measurement problem when it comes to reliability and validity. Without a solid measurement structure, it is the old adage of “Garbage In, Garbage Out”. Hopefully, the UCM is a first step to rectifying this issue.

Clearly, for future research, there needs to be additional expansion beyond the childcare and early education field to all of human services and then beyond this scope to other regulatory areas to determine if a UCM approach is relevant. It is obvious that in clinical studies within the medical field that the UCM would be very appropriate in order to avoid false negatives where a drug’s side effects would be more detrimental than the potential benefits from taking the particular drug. We need additional real-life examples where the UCM model can be tested to see how useful it would be in other regulatory settings.