Establish Measures

Observation and measurement are an essential part of making a change. It is important to collect data about what is happening before and after a change is made, and analyze the data to learn if the change made a difference and determine what to do next.3 Timely and ongoing collection and analysis of data is needed to determine if a change has resulted in improvement and to monitor how well the change strategies are working.25

Table 1. Tips for effective measurement 3, 25

Measures should relate directly to the aim of the change initiative. Measures can reflect:

  • Outcomes of care for patients – For example, achievement of therapeutic goals, mortality, and morbidity including adverse outcomes related to care.
  • Steps in the processes of care – Develop a process or flow map to help understand all the steps, activities, tasks, and decisions that are needed to achieve a desired outcome.

Define the measures and how data will be collected.

  • Everyone needs to document and interpret the data in a consistent way.

Use different kinds of measures to get a complete picture of the impact of the change.

  • Quantitative measures – something that can be observed and counted or measured using some kind of tool.
  • Qualitative measures – perceptions and feelings of those affected by an issue or change, usually gathered from interviews, surveys, or focus groups.

Collect data at numerous points over time and look for trends and patterns.

  • A run chart is a graph that tracks the data points over time.89
  • Enough data and time points are required to distinguish between expected fluctuations over time and variations that signal change.

Collect just enough data to know whether a change is an improvement.

  • Avoid collecting information that is ‘nice to know’ but not needed.
  • Avoid collecting personally identifying information about individuals (patients or staff).

Use sampling to make efficient use of resources during data collection.

  • Collect data from a representative subset or sample of the total data available.
  • The sample size or number of measurements taken at each time point needs to be adequate to detect a pattern that signals change. The table below shows suggested numbers of measurements to take for different improvement situations.
  • Integrate measurement into the daily routine using a simple data collection form.
Table 2. Suggested sample sizes for tests of change 3
Number of measurements (sample size) Improvement Situation
Fewer than 10 Expensive tests of change, long periods between available data points, large effects expected
15 to 50 Usually adequate to detect moderate to large changes
50 to 100 Effect of change is expected to be relatively small compared to typical variation
More than 100 Change is intended to affect a rare event

Adapted from Langley and Nolan 3

U for Units: Selecting Measures

The project team decided to collect baseline data from medication orders received by the pharmacy from the target units on five randomly selected days over a two week period:

  • Name of medications being ordered by the dose designation U or units. It was anticipated that most would be for heparin or insulin which are both high-alert medications.
  • Dose designation used – either U or unit.
  • Prescriber initials. It was necessary to identify the prescriber during data collection in order to provide feedback to prescribers during the project. A separate list of prescriber names corresponding to the initials was created.

A simple data collection form was developed with columns for medication name (check off heparin, insulin or other), dose designation used (check off U or units), and prescriber initials.

To analyze the data, the total number of orders for each medication category was tallied for each prescriber, and the proportion of each for which units was used was calculated. The team decided it was important to use positive reinforcement by illustrating how often the desired practice, units, was used. A bar graph was prepared that illustrated by prescriber the proportion of orders for heparin, insulin and other medications for which units was used as the dose designation. Prescriber identity was protected by not including initials on the graph.

See Full Case

U for Units: Selecting Measures

The project team decided to gather additional information to serve as a baseline.
1. The pharmacy collected information about all insulin prescriptions over a 4 week period:

  • Dose designation used – either U or unit
  • Prescriber initials. A separate list of prescriber names corresponding to the initials was created. It was necessary to identify the prescriber during data collection in order to provide feedback to prescribers during the project.
  • Orders transcribed from a verbal prescription were designated with TO.

A simple data collection form was developed with columns for date, dose designation used (check off U or units), prescriber initials, and whether it was a transcribed order.

To analyze the data, the total number of orders for insulin was tallied for each prescriber, and the proportion of each for which units was used was calculated. The team decided it was important to use positive reinforcement by illustrating how often the desired practice, units, was used. A bar graph was prepared that illustrated by prescriber the proportion of written insulin orders for which units was used as the dose designation. Prescriber identity was protected by not including initials on the graph. A separate graph was prepared for verbal prescriptions showing the proportion of transcriptions for which units was used as the dose designation.

2. Over the same 4 week period, the consultant pharmacist reviewed all new and changed insulin orders transcribed at both the facility and the pharmacy. The identity of the transcriber was not collected. A bar graph was prepared that showed the proportion of transcribed verbal orders for insulin at the facility and pharmacy for which units was used as the dose designation.

See Full Case