Quality of care is a broad and abstract concept and the attempts of measuring quality places constrains on the interpretability of the outcomes. The aim of this research was to study the value of performance indicators in comparing the quality of care between hospitals and their usefulness for the improvement of the quality of care within hospitals.

We found considerable influence of random variation when we compared hospitals using the outcome indicators of the Dutch Health Care Inspectorate. Using the concept of rankability we found that in the ranking of hospitals, both the between hospital uncertainty and the uncertainty of the within hospitals estimates led to unreliable positioning. Therefore, none of the tested indicators could be used for the ranking of hospitals. Low numbers in sample size or event rates lead to an indicator score in which differences between hospitals do not overcome random variation. Both sample size and event rates need to be addressed in the development of indicators. Furthermore, we found that a forest plot gave appropriate insight in the number of Dutch hospitals that actually significantly deviated from the average. The funnel plot provided a visual representation of differences between hospitals therewith allowing simple interpretation of the uncertainty of these differences. We also used rank plots and showed that the substantial uncertainty makes current rankings with these outcome indicators unreliable. Of all three graphical displays used, the funnel plots provided most valuable insight in the magnitude of random variation and is therefore best used for the interpretation of differences between hospitals. Although more research is needed to clarify the desired graphical display in specific conditions, not addressing random variation in graphical displays potentially misleads hospital assessments.
For surgical site infections, we found that the apparent differences between Dutch hospitals in this specific outcome indicator were predominantly attributable to random variation and case-mix. This case study provided a clear illustration that both random variation and case-mix must be addressed systematically in performance measurement before conclusions can be drawn on the quality of hospital care. It is indisputable that correction for patient factors should be part of the assessment of hospital performance.

Exploring the process-outcome relation, we found that the outcome indicator ‘pressure ulcer occurrence’ reflected differences in the quality of the bundle of preventive care processes provided by nurses. This significant relation between outcome and process in pressure ulcer care, supports the usefulness of this indicator in assessing the quality of nursing care. We confirmed that the pressure ulcer prevalence was also determined by several patient factors that cannot be influenced. Addressing the process-outcome relation in performance measurement from another angle, we explored the feasibility of measuring the effect of surveillance by the Dutch Health Care Inspectorate using retrospective data on health outcomes in three health problems: pressure ulcers, suicide and medication errors. We found that in case of clearly defined health problems, such as pressure ulcers and suicide, the frequency of these outcomes could be measured using Inspectorate data and trends could be analysed using an interrupted time series design. However, support of a causal relationship between supervision and observed trends could only be derived with data on external factors that influenced this trend. Although the relationship between process and outcome may seem straight forward and applicable, this is not always confirmed in research. We recommend, that the process-outcome relation should be addressed and explored in existing indicators as well as in the development of new quality indicators.

In a process measure of acute stroke care, we found a significant improvement in “door-to-needle time” (DNT) over recent years. We could not attribute this trend to one or more specific interventions. We hypothesised that the combined effect of various interventions together and the constant focus of care-givers on quality improvement explained the significant improvement of the indicator DNT. In a quality project that aimed to improve pressure ulcer prevalence, we did not find a statistically significant decrease of nosocomial pressure ulcer occurrence using an interrupted time series design. However, we did see a significant improvement in the process measure risk assessment. Outcome indicators should be paired with process indicators to gain insight for the improvement of quality of care.
CONCLUSION
The measure of quality of care is a multidimensional and complex process. We must be aware that a performance indicator offers only a certain signal on quality and is by no means an absolute measure. Like a one hand clock, we roughly know what time it is.