Strategies for Error Reduction: Why More Stringent Premarket Evaluations Do Little to Prevent Laboratory Errors and Traffic Accidents

Abstract

Laboratory testing is a complex process with a significant error rate. Studies of laboratory errors have found that the major causes are preanalytical factors, interferences, and process errors. Efforts by regulatory agencies to improve quality via more stringent premarket evaluations of laboratory tests therefore have poor prospects of reducing laboratory errors and improving test quality. Efforts toward increasing the regulation of laboratory tests are analogous to preventing traffic accidents by increasing the premarket evaluation of automobiles. This analogy illustrates how increased premarket evaluation has limited prospects for quality improvement and, in some cases, actually contributes to errors and lower quality. Tools that are used by laboratories to detect, prevent, and address analytical errors are discussed, and the increased implementation of such tools offers approaches that can be used to improve laboratory quality.

1. Introduction

Clinical laboratory tests are processes and not devices. Most tests use devices and reagents, but these are just components of the overall testing process. Understanding that laboratory testing is a process is important for developing strategies for improving test quality and safety and for understanding why increased premarket evaluation of laboratory tests, such as recently proposed regulations by the United States Food and Drug Administration (US FDA) for laboratory-developed tests (LDTs) [1,2,3], are unlikely to contribute substantially to the stated goal of increased safety of testing. Regulatory agencies sometimes take the view that laboratories are manufacturers of tests, but laboratories are providing a service and not a physical product. Strategies for improving laboratory test quality need to address the testing process and not just the testing device.

Most laboratory errors and safety issues are not related to faulty test devices [4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]. A simple analogy might clarify this important point. Performing laboratory tests is a process, similar to driving, and clinical laboratories are service organizations analogous to car services, providing transportation/test results. Clinical laboratories are not manufacturers but are service providers. Automobiles/test devices are important tools in the transportation/testing process, but transportation and testing involve many other elements—driver/technologist training, driver’s licenses/certification, driver/technologist skill, checking the speedometer/quality control, road conditions/lab facilities, and external factors such as weather/specimen integrity.

Studies of laboratory errors have found that most errors are related to preanalytical problems and, less often, to analytical problems [4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]. A more extensive bibliography is provided in reference [16]. Similarly, few traffic accidents are related to mechanical defects of automobiles. Data from Great Britain show that fewer than 1% of accidents are related to vehicle defects [20]. The few mechanical issues are most often related to faulty tires or brakes, issues that probably result from wear and not initial design defects. The major causes of accidents in Great Britain are driver error (40%), reckless driving (13%), injudicious actions such as speeding (12%), impaired or distracted driving (9%), and road conditions (6%). Inadequate initial validations or premarket reviews of automobiles and test devices represent a very small proportion of accidents/errors. Moreover, most vehicle/device failures do not show up until after extended use. Notable vehicle defects such as faulty airbags [21] or the exploding gas tanks of Ford Pintos [22] were not apparent in premarket reviews and only showed up later. These problems did not result from any customization or changes after the vehicles were manufactured.

Let us extend the analogy regarding automobile safety further with the following parable about whether traffic accidents could be prevented and automobile safety improved, through a more thorough premarket review of customized automobiles. There may be some parallels with proposed FDA regulations of LDTs.

2. A Tale of Preventing Traffic Accidents through the Premarket Review of Driver-Modified Cars (DMCs)

The National Highway Safety Agency (NHSA) notes that the US FDA has proposed new regulations for lab-developed tests (LDTs), and administrators at NHSA see some parallels with automobile safety and traffic accidents. Traffic accidents probably kill and injure more people than laboratory errors, so this topic is clearly of high priority. Some automobile accidents could be related to automobile customization such as the tinting of windows, thus reducing driver visibility, or adding new rims and nonstandard tires that might affect stopping distances; after all, we are all familiar with the tragic death of Princess Diana in a limousine that was probably a customized vehicle. It has also been noted that there is continuing growth in the customization of cars and the use of electronics devices. Many people are using phone-linked navigation and other electronic devices that are not original equipment. What about the possibility of a driver closely following the directions of their navigation system and driving off a bridge under construction?

The NHSA develops modernized regulations that will prevent traffic accidents and promote automobile safety through the review of any driver-modified conveyances (DMCs), often termed driver-modified cars. To spearhead this program, the NHSA hires a team of former FDA employees based on their regulatory expertise. The approval of DMCs is required before their use. Any car customization must be submitted for review. If you tint your windows, add signage to your car (causing a potential distraction to other drivers), add different tires, or rebuild a 1954 Chevy (constituting a possible grandfather-clause-based exemption depending on when the rebuild was performed), then your DMC requires review before being driven. After all, traffic accidents are a huge problem, and the NHSA’s mission is to improve traffic safety. An online submission process with an informative 3 h tutorial guiding one through the application makes it easy to submit applications for reviews and complies with paperwork reduction rules.

The OberVan car service has six aging Dodge Caravans in their fleet. OberVan has decided to replace their engines and replace their old, dented bumpers with new, stronger bumpers with absorbent rubber cushions to reduce damage to other cars in minor collisions. These changes require a risk-based NHSA review. These modifications are classified as potential risks to safety. The new engines are more powerful and might encourage speeding. It is unclear how well the new bumpers will work, as the fact that they are stronger does not guarantee better safety. OberVan performs mechanical inspections of the modified Caravans and drives them on a test track. In one case, a new engine had a defective fuel injector that was replaced and now works fine. OberVan sends in specifications, test results, and added fees for expedited safety review.

These potentially hazardous changes to DMCs present challenges for the NHSA staff because they have not received identical requests previously. There is a lack of evidence of the efficacy of the new bumpers on this vehicle. No crash test results are available for this bumper on Caravan minivans. Notice is sent back to OberVan that further efficacy data are needed. OberVan performs full-frontal crash testing on one of the Caravans with the new bumper and finds better crash resistance than the original equipment. The results are resubmitted for review. These limited data are not completely satisfactory, however, because testing was only performed on one Caravan at a single speed and no partial offset crash results were provided. After three months of review, the NHSA team approves the use of OberVan’s five remaining DMCs. However, to ensure safety, important labeling instructions are provided. The DMCs are not to be operated while under the influence of alcohol or drugs and should not be driven at a speed of more than 40 mph (66 kph). Since efficacy data on the bumpers were limited, the driver and all passengers should wear NHSA-certified helmets. These cars should not be operated on snowy or icy roads or in hazardous conditions.

OberVan puts the DMCs into service but now realizes that winter is approaching, with the prospect of snowy and icy roads. OberVan considers installing snow tires or tire chains but opts not to, supposing that these customizations would require additional reviews. OberVan also learns that some customers want to transport Christmas trees home on the roofs of the DMCs. OberVan is unsure whether this would violate the intended use of the DMC and does not submit this issue for review.

In December, OberVan receives 1000 requests for transport. Problems are experienced with 60 of these requests: 40 trips were canceled by clients. In 12 trips, problems with traffic delays, becoming lost en route, drunken passengers, going to the wrong address, flat tires, or mechanical problems were encountered. Some of these problems required dispatching a backup vehicle to complete the trips. During five trips, the Caravans lacking snow tires became stuck in snow and required tow trucks to complete the trip. There were three traffic accidents.

Two of the accidents resulted from drivers texting about their next pickup. Drivers failed to stop in time and collided with the cars in front of them, resulting in minor damage and no injuries. The new bumpers helped minimize damage. However, the third accident was a different matter and was very serious.

The serious accident occurred shortly before Christmas. An OberVan customer requested transport of a large Christmas tree that was tied securely onto the roof of a Caravan DMC. During the trip to the customer’s home, there was a sudden snowstorm. With the snowy conditions and lack of snow tires, the DMC lost traction, slid at an angle into oncoming traffic, and collided with a bus. There was a massive crash, and the Christmas tree flew from the roof of the DMC and smashed through the front window of the bus. The driver and passenger of the DMC suffered head injuries but, fortunately, survived, possibly due to the added crash resistance provided by the new bumper.

Fears of the hazards of DMCs by NHSA were confirmed by this accident and added to the DMC accident database. Out of the 960 completed trips in December, there was one serious accident, amounting to a relatively high rate. Root cause analysis was performed. Although the investigators found that the driver was not intoxicated and the DMC was operated at a speed under 40 mph (66 kph), the root causes of the accident and injuries were clear. The accident was caused by violating labeling instructions stating that the vehicle should not be driven in icy conditions, and the occupants of the DMC incurred head injuries because they did not wear helmets as directed. The bumper buckled in the crash with the bus (although less severely than expected for the original bumper). Furthermore, carrying a Christmas tree on the roof was a violation of the intended use of the vehicle and a safety hazard to occupants of other vehicles in the event of a collision. Press releases were sent out about how the NHSA has uncovered major new hazards of DMCs and their use to transport Christmas trees.

After months of review, a communication plan is developed and approved by the legal staff to be directed to any DMCs equipped with the problematic bumper. A warning letter is sent, stating that in the event of a head-on collision with a bus at a 35-degree angle, the bumper may fail.

This incident is promoted as clear evidence of the value of the DMC review program and the collection of accident data. This program costs the agency hundreds of millions of dollars per year and requires a great deal of additional staff, but most of the cost is covered by user fees. Importantly, this program is predicted to save billions of dollars due to all of the accidents it will prevent. Adding a barrier to any car customization will also help prevent many drivers from modifying their cars, and that should make cars safer. Any change or anything new inherently poses a risk. This program will also encourage car manufacturers to offer more customization options because there will be less after-market competition.

OberVan was sued for the accident. NHSA reports clearly point out the hazards of DMCs. If only OberVan had avoided using DMCs or followed labeling instructions, this accident and the resultant injuries could have been avoided. There was a very large settlement.

The review of DMCs poses many challenges and weighty issues to the NHSA team that were not anticipated in the original regulations. Suppose drivers want to transport dogs. That does not seem like an intended use. Service dogs are probably OK. After all, there must be accommodations for disabilities. Then, there is the issue of cats. Are there service cats?

3. Discussion

In the foregoing example of the transportation problems of a car service, the premarket review conducted by the NHSA team did not contribute to preventing any of the problems OberVan noted during the month of December, although it added major costs and delay. It is possible that the NHSA review of DMCs could have identified some defect that affected safety, but the car service already performed a thorough mechanical and safety check of each vehicle. Based on the thoroughness of this review, major defects in any of the vehicles should be detected, while the NHSA review only considers vehicles as a group when the vehicles are tested under controlled conditions. Many mechanical issues will not become apparent until after a product has undergone extensive use. The vehicle modifications in this case probably did not represent substantial safety hazards but were classified as hazardous by an inflexible regulatory regime. The regulatory program actually hindered vehicle safety by discouraging measures such as adding stronger bumpers, snow tires, or tire chains that could improve safety. Also, the performance of the vehicles was decreased by a restrictive speed limit. In this case, labeling instructions might have prevented head injuries if helmets had been worn and decreased the risk of an accident if the minivans had not been operated during bad weather. However, these recommendations might apply equally to all vehicles, and there is no particular reason to apply these recommendations specifically to DMCs.

Pemarket reviews by FDA or other agencies of laboratory tests similarly contribute little to error prevention and safety. First of all, the premarket reviews are conducted on the testing device, not the entire testing process. Most errors result from preanalytical factors and other errors in the complete testing process and not in defects of the test device. Second, laboratories are already required to validate the performance of each analyzer before use, with some additional requirements for the validation of LDTs [2,3]. This should identify most systematic problems with test methods and even individual analyzers. Sometimes, even with new analyzers, there is a “lemon” in need of replacement, or, for older analyzers, performance could change with age and wear. Premarket reviews by regulatory agencies provide little quality improvement because they represent a second review and only evaluate performance on a few new instruments, which are optimally maintained, with fresh reagents, skilled operators, and selected specimens, often with care taken for optimal collection and the avoidance of problematic specimens. Some problems with laboratory tests only show up after extended use, wear and tear of analyzers, the use of different reagent lots, suboptimal specimen collection and processing, or testing of specimens from patients on specific medications, diets, or medical conditions.

Laboratory testing is a complex process. Most evaluations of laboratory errors focus on errors that result in qualitatively or quantitatively inaccurate test results, and these evaluations do not even address what often are the most error-prone steps in the process: the ordering of the appropriate tests, efficient test utilization, achieving timely results of tests, and the appropriate interpretation and follow-up of test results [23,24,25]. Failures to obtain the right test and appropriately interpret results probably are the major source of harm related to laboratory testing. The failure to make a timely and correct diagnosis, which includes many components—clinician decision making, radiological studies, clinical findings, and laboratory testing—is increasingly being recognized as a major cause of patient morbidity and mortality [26,27,28]. Improving the complete testing process from test ordering to the interpretation of results offers the greatest potential for improving patient care [29,30,31]. The implementation of diagnostic management teams is one recommended means of achieving this [27,31]. For the present discussion, the focus is on the narrower issue of analytical errors, that is, providing inaccurate results or the inability to obtain results, although the broader overall scope of potential errors should be kept in mind.

All lab tests are imperfect and may generate erroneous results [4,5,6,7,8,9,10]. Despite one’s best efforts toward method evaluation before use, laboratory errors occur, and constant review of the complete testing process is needed. Many tests have error rates in the 0.1–1% range [4,5,6,7,8,9,10], and some tests, such as blood or urine cultures [17,18,19], have substantially higher rates related to specific problems such as specimen contamination. Interferences, specimen contamination, and the effects of preanalytical variables can be complex issues [32,33,34,35,36,37,38,39,40]. The effects of medications on laboratory tests can pose a particular challenge for laboratories [33,34], considering that laboratory results are rarely correlated with medication records except in the case of therapeutic drug monitoring or testing for pain medications or illicit drugs. Many medications can affect laboratory tests, as compiled in a large compendium [34], which has continued to be updated as an online database as the volume of entries has become difficult to manage as a print version. The effects of blood collection devices on laboratory tests can be another challenging and sometimes unrecognized issue for laboratories that is related to the effects of anticoagulants, surfactants, separator gel materials, or other materials [36]. Lipemia, hemolysis, and icterus are common potential interferents with many clinical chemistry tests and must be addressed by laboratories [37,38,39,40]. These interferents can be detected visually or via the automated analysis of serum and plasma specimens but may remain unrecognized in whole-blood specimens.

The diverse range of preanalytical factors that can cause analyzers to produce incorrect values requires a busy laboratory producing thousands of test results per day to address many errors every day to prevent the reporting of inaccurate results. One of the major tasks of staff in clinical laboratories, therefore, is to apply processes and expertise to identify and cancel or correct the multiple errors that occur daily before they are reported to providers and patients. Laboratory staff need to recognize preanalytical issues such as the hemolysis of blood specimens, the contamination of urine or blood cultures, incorrect specimen types, clotted specimens, and interferences by medications, lipemia, icterus, or specimen collection devices and to intervene appropriately. Routine quality control processes do not detect many of these problems. To address these problems, clinical laboratories sometimes repeat testing and use operator expertise and many tools to avoid or fix problems that are identified. Some examples of the measures that laboratories use to detect potentially erroneous results are listed in

Table 1. Methods and tools used to detect, prevent, or address laboratory errors.

Delta Checking (Comparison with Previous Results)
Developing LDTs to address interference or other test limitations
Evaluating running means or positivity rates
Evaluating correlations between different test results, such as those for urea and creatinine, sometimes as ratios
Calculating corrected values, e.g., calcium values
Establishing criteria for repeating tests
Conducting point-of-care testing to address sample stability and turnaround time issues
Checking serum indices or visual inspection of specimens
Conducting ultracentrifugation to remove interferences from lipemia
Warming specimens with cold agglutinins or cryoglobulins
Performing slide reviews to check for cell clumping or unusual morphology
Checking specimens for clots
Conducting a second review of manual data entry
Diluting specimens to check for high-dose hook effects or substrate depletion
Using reflex testing and diagnostic algorithms to rule out possible false positives, e.g., for syphilis, viral hepatitis, or human immunodeficiency virus testing
Conducting critical-value reporting
Developing of lab-specific and age-specific reference ranges
Carrying out consultations and chart reviews for unexpected results
Cancelling test results for suspected interferences or other preanalytical problems
Adding comments and interpretative reports to results
Recollecting specimens with suspected preanalytical problems
Determining correlations with medications
Using patient management teams to guide test ordering and interpretation

.

Many of the measures in Table 1 are not included in protocols or labeling of the test devices or reagent labeling for tests that are cleared or approved by regulatory agencies such as the US FDA. Some measures might even raise questions about whether they are compliant with labeling or the intended use of US FDA-cleared test devices or represent a change considered to be an LDT. Most test devices, for example, have not been evaluated for the testing of ultracentrifuged specimens or warmed specimens to counter the effects of cryoglobulins or cold agglutinins. Does that mean that laboratories should not perform the ultracentrifugation of highly lipemic specimens or warm specimens before analysis in order to obtain results without interference from lipemia, cryoglobulins, or cold agglutinins? If testing is performed on dilution to exclude a high-dose hook effect (antigen excess) in an immunoassay procedure or to exclude substrate depletion in an enzyme assay, would that be compliant with laboratory regulations if the procedures are not spelled out in the manufacturer’s directions? In some cases, regulations lower test quality by serving as barriers to improvements or measures that could help prevent errors. Increased laboratory regulation can result in lower test quality, as recently outlined in the harms of over-regulation [41]. Regulations often serve as a systematic barrier to any improvement of laboratory devices and innovation, such as in the analogy of adding snow tires, tire chains new bumpers, or new engines. Consequently, restrictive regulations often lead to lower quality rather than the stated goal of quality improvement while also increasing costs and imposing restrictions that may lower efficiency, including restrictions such as unnecessarily low speed limits and increased consumption of resources for additional testing and paperwork, adding little to quality.

Point-of-care testing has been expanding in scope and represents an interesting case in which there are opportunities both for quality improvement and increased errors [42,43,44,45,46,47,48]. Immediate testing at the site of specimen collection improves the testing of specimens with limited stability for tests such as activated clotting time and may allow better correlation with a patient’s condition or medications. Also, recollection and repeat testing can be performed immediately to confirm unexpected results. On the negative side, there may be lower laboratory expertise of the testing personnel, and there is less control of reagent and instrument inventory and storage. Small portable devices may be subject to damage during transport, and the use of many testing devices may multiply opportunities for error and complicate the control and oversight of testing. There can be failure to capture test results and feed them into central data systems from manual or widely distributed devices. Point-of-care testing does not fit neatly into the analogy of laboratory testing as it is similar to the operation of a car service. Point-of-care testing is more analogous to providing transportation by bicycles or motorbikes. Transportation can be achieved immediately without waiting for a car to arrive, but the mechanisms of transport are simpler, and the driving of the bikes is performed by nonexpert drivers who may have widely varying degrees of experience and training unlike full-time drivers for a car service.

Efforts toward improving the quality of laboratory testing should consider every opportunity for innovation and using additional tools analogous to snow tires, tire chains, and tow trucks. Regulation of laboratory test devices should consider how to avoid impeding these efforts. Laboratory testing is a complex process that requires attention to the entire process, and usually the analytical performance of the test device is not the primary source of analytical errors in the laboratory. Preanalytical factors have been identified as the most common sources of these errors for most laboratory tests, and it is a critical part of the daily work of a laboratory to detect the preanalytical problems and take appropriate remedial actions. Reducing laboratory errors requires coordination and attention to the entire testing process, skilled staff, and the use of every possible tool for error prevention, detection, and mitigation.