“In God We Trust. All Others Must Bring Data”
– W. Edwards Deming
Measurement is at the core of improvement and has an essential role in effectively developing, testing, and implementing changes. Measurement helps us answer the second question in the Model for Improvement:
“How will we know a change is an improvement?”
Without feedback, we have no way of knowing whether the changes we are making are leading to improvement. Data usually provide the most useful and objective form of feedback.
By the end of Step 5, you should be able to:
- establish the “family of measures” for your project, including process, outcome, and balancing measures
- operationalize each measure
- develop a measurement plan
Importance of Measurement for Improvement
Measurement in quality improvement allows us to understand whether we have reached our improvement aims. To do so, we must define and operationalize what ‘better’ looks like and measure to know if the changes we make result in the improvements we seek.
Measurement for improvement allows us to answer questions such as:
- What is the current state?
- What does “better” look like?
- How will we recognize better when we see it?
- What are factors, such as processes and activities, have an impact the outcomes?
- Are the processes stable and reliable?
- How do we know if a change is an improvement?
The graphic below depicts some of the different ways that measurement and data support each step throughout our improvement journey (click to enlarge):
Quality improvement requires a comprehensive understanding of the system or how the various processes work together to achieve outcomes. Establishing a “family of measures,” provides a view of the system from the outcomes, to the processes, to the unintended impacts.
The approach below, coined by Robert Lloyd as the Measurement Journey Model, helps us establish valid and actionable measurement to support our improvement projects.
Source: Lloyd, R. Quality Health Care: a guide to developing and using indicators. Jones & Bartlett Publishers 2004
Taking the time to develop a formal “Measurement Plan” can help measure with a purpose and prevent spending excessive time and resources on unnecessary measurement efforts. Measurement plans outline the answers to questions of-what, how, where, when and by whom in the data collection process. Engaging all team members in the creation, communication, and upkeep of the measurement plan creates greater understanding of the outcomes you are trying to achieve and facilitates all other aspects of the improvement journey.
What types of measures are there?
When we create our family of measures for an improvement project, we always want to try to include a balanced set of measures that assess processes and outcomes as well as identifying balancing measures.
- Outcome Measures: How does the system impact the values of students, their health and wellbeing? What are impacts on other stakeholders such as payers, employees, or the community?
- Process Measures: Are the parts/steps in the system performing as planned? Are we on track in our efforts to improve the system?
- Balancing Measures: Are changes designed to improve one part of the system causing new problems in other parts of the system?
We typically establish 5-8 measures, as part of our family of measures, when carrying out an improvement project.
|Wait time for walk-in visit||time from check-in to start of provider assessment|
|Length of time to check ||Staff satisfaction|
|Depression care||% of students on campus who report academic impact from depression||% of students screened for depression in medical services.|
% of students engaged in treatment within 2 weeks of positive screen
|Appointment availability in counseling services|
|Binge drinking||% of students on campus who binge drink|
% of students on campus who experience a negative consequence from alcohol
|% of entering students who complete online alcohol educational module|
% of students screened for high risk drinking
of those students who screen positive, % of students with a documented that brief motivational interview intervention was conducted.
% of students on campus who are trained on bystander intervention
|% students reporting that they received information from their school on other (non-alcohol) related topics|
Differences between Measurement for Judgement versus Research versus Improvement
Measurement for improvement is distinctly different from measurement for research or judgement. These differences are outlined in the table below.
Adapted from: “The Three Faces of Performance Management: Improvement, Accountability and Research.” Solberg, Leif I., Mosser, Gordon and McDonald, Susan. Journal of Quality Improvement. March 1997, Vol23, No. 3.
- If a change is not working, adjust it. The ability to make adjustments along the journey to improvement is not a weakness of the study design; it is an essential element of it.
- Accept that various types of bias can exist in the data. Rather than attempt to eliminate bias by controlling for it, try to keep the context and bias consistent over time.
- Improvement research does not take place in isolated, controlled conditions; typically, it happens in real world settings. Testing changes in the context they will be implemented (“real world settings”) is an essential source of learning about how to make the change work reliably: for every student, every time, in every context.
- Whenever possible, develop measures from data that is already being collected. That’ll make your project a lot simpler without compromising on quality. When it’s necessary to develop new measures, try to make the data relatively easy to obtain.
Improvement Journey Exercise #5
- Establish the “family of measures” for your project, including process, outcome, and balancing measures
- Operationalize each measure & document your measurement plan using the Measurement Planning Form
Once you have operationalized your measurement plan, move on to Step 6: Develop Ideas for Change.