*4.5. Study Strengths and Limitations*

A major strength of this study is the detailed information obtained from various data sources and the triangulation of qualitative and quantitative data. Mixed methods research is valuable because it captures the information from various perspectives and can support qualitative and quantitative findings [42]. This process evaluation is rich in detail and provides a comprehensive picture of the program. Further, the long follow-up period permits a reasonable understanding of the dynamics over time.

On the other hand, evaluation of the PIP's impact was limited by the fact that the sites were not selected at random. The three PIP sites were volunteered by their administrators in response to a recruitment effort by the investigators through a key regional staff member. The three NPHP sites were also selected by the same regional representative, in response to the research team's specified criteria, and then approached for management agreement to permit data collection. As a consequence, there were some anticipated baseline differences between the two groups, both in prior HP activity and possibly in administrator interest in and initiative toward workforce well-being. A further issue is that the study results are not expected to be generalizable, except to other nursing homes with similar management interest and willingness. Nevertheless, the results do demonstrate the feasibility of conducting a participatory change process in this sector, despite (in the U.S.A., at least) tight staffing and scheduling, coupled with low union representation to protect job security for those who voice their opinions about root causes of health and safety problems.

In theory, the gold standard for an intervention study is the randomized controlled trial. However, the benefits of randomization in reducing confounding are not realized except with a large sample. In this case, the intervention was carried out at the level of the entire facility (PIP). In practice, it was not logistically or economically feasible for us to enroll a large number of facilities for such an intervention. Even with some alternative designs, such as the stepped-wedge [43], there is still concern about potential confounding and often a randomization element, thus the number of intervention units remains important. One alternative is to compare the treatment groups on baseline characteristics that might influence the outcome, which we have done here. In fact, perceived working conditions were quite similar except for decision latitude; since that decreased later rather than increasing in the PIP group, the change in time was unlikely to be an artifact of a difference at baseline.

The other advantage of a randomized controlled trial is that with double-blinding, the possibility of information bias can be greatly reduced. However, blinding of participants and researchers is also infeasible for organizational-level interventions. In turn, randomization and blinding may have disadvantages for organizational-level interventions, such as limited capacity to assess multi-dimensional interventions or evaluate process, quality, or performance, and incompatibility with community trust, choice, and participation often needed for successful program design and evaluation [44–46].

Another limitation of this study is the difficulty to show the actual impact of the participatory program due to constant changes in the organization. For example, turnover in leadership in the study sites appeared to affect program success with administrator changes in the PIP sites during the study period. Similarly, employee turnover limited our ability to examine within-person changes over time, through reduced statistical power.

Participatory programs can be challenging to implement and evaluate in a research context because key elements cannot be controlled by the investigators; for example, interventions are selected and designed by workers after the program is already underway, and interventions addressing higher-level organizational obstacles may provoke institutional resistance that might not have been visible or even present at the beginning of the project. These issues were known in advance and cannot be prevented even when they are anticipated. It will always be the case that many organizations and workplaces will refuse voluntary worker health improvement efforts, even when resources are offered to facilitate the program. The criteria for selection of participatory intervention sites have been revised on the basis of this experience, in an attempt to better inform decision-makers in advance about the expected process and its potential benefits and costs.
