Step 7: Test Changes


“You can’t learn to ride a bike by reading a book on physics.”

 

Once you have developed ideas for change, your next series of steps in the improvement journey are to test on a small scale the impact of each proposed change, including identifying any negative consequences from the change.

At the end of Step 7, your team should be prepared to:

  1. Plan, test, document, and refine your first small scale, rapid cycle test of change.
  2. Develop the Data Collection Plan for your first small scale, rapid cycle test of change.

 

The Plan-Do-Study-Act (PDSA) cycle is a powerful vehicle for testing, learning, implementing, and spreading.  PDSA cycles can help you plan for, carry out, and learn from small tests of change.  By starting small and using iterative, linked cycles, the PDSA framework helps minimize and mitigate common risks associated with problematic or failed implementation of changes. Try a test quickly; at your next team meeting, collectively ask and answer: “What change can we test by next Tuesday?”

  • Taking a change out of the theoretical setting of a meeting room and seeing whether it works in the actual environment where it will be used.
  • Seeing which of multiple proposed changes will lead to the desired improvement
  • Increasing a team’s belief that the change will result in an actual improvement.
  • Learning from the ideas that work, as well as from those that do not.
  •  Minimizing:
    • disruption to the student experience
    • risks and expenditures of time and money
    • resistance to change among staff.  People are far more willing to test a change when they know that changes can and will be modified as needed. Linking small tests of change helps overcome people’s natural resistance to change and ensure buy-in.

PDSA Cycle

PDSA Cycle

tipFidelity to each step in the Model for Improvement and PDSA framework significantly increases the likelihood of developing, refining, implementing, and spreading changes that will lead to achievement of your aim (developed in Step 3).

 

Steps in the PDSA Cycle (from IHI.org)

Step 1: Plan

  • Define the objective(s) of the test.
  • State the scope of the PDSA cycle.
  • Make predictions about what will happen and why.
  • How will you know if the tested change is a success?
  • What data will you collect? Who will collect it? How will it be collected?
  • Develop a plan to test the change. (Who? What? When? Where? How long will the PDSA continue?  Does everyone understand their role?)

Step 2: Do

  • Carry out the test.
  • Document problems and unexpected observations.
  • Begin analysis of the data.
  • Encourage continual feedback within the team and between the senior leader and team.  Consider a brief team huddle (5-10 minutes) after a test of change to accelerate learnings and testing.

Step 3: Study

  • Complete the analysis of the data.
  • Compare the data to your predictions.
  • Summarize and reflect on what was learned.
  • What did it feel like? Did staff and students notice an improvement?
  • Was the process shorter or longer?
  • Did you achieve your objectives? If not, why not?
  • What went well?
  • What could be improved?

Step 4: Act

  • Determine what modifications should be made.
  • Prepare a plan for the next test.
  • When you have achieved your objectives, move to Step 8:  Implementing Changes.  You may need to run a number of PDSA cycles before you see a positive improvement which is ready to be tested with larger numbers, implemented, and ultimately spread.

tipDocumenting each PDSA cycle is an important component of the process.  While it may seem like busy work, documenting each and every PDSA cycle in real time as your team works through each step significantly increases the learnings.

Linking Your Tests of Change (PDSA cycles)

Testing changes is an iterative process: the completion of each PDSA cycle leads directly into the start of the next cycle.  A team learns from the test — What worked and what didn’t work? What should be kept, changed, or abandoned? — and uses the new knowledge to plan the next test. The team continues linking tests in this way, refining the change until it is ready for broader implementation.

Linking PDSA cycles

 

Tips when using PDSA cycles:

PDSA cycles are intended to be learning experiences to improve change with each cycle.  Determine what your team can do by next Tuesday.  Here’s why:

 

 

Start with small tests and scale them up based on what you learn.  That may mean initiating a change with one staff member, for one hour, and changing only a single part of a process.  This approach will help build knowledge sequentially with each PDSA cycle. After studying the results of the PDSA and only after the results suggest improvement, scale up the number of staff involved and students impacted in the next test of change.

to ensure the change has the desired impact under multiple “real-world” situations.  If your first test occurs on Monday morning, try another test on a Tuesday afternoon.

who is willing to try the change, collecting necessary quantitative and qualitative data as necessary, and report back to the team about the experience.  Do not ask your most resistant staff member to test a change in the early stages. (see Step 8 for managing resistant staff in the implementation step)

If there is disagreement about which changes will be most impactful with the least disruption, allow the team to test the conflicting ideas on a small scale.  This will allow for a decision based on experience and evidence rather than emotion.

Instead, replicate changes made elsewhere and use the testing phase to modify / adapt to your unique setting.

Look for the concepts that seem most feasible and will have the greatest impact.

Don’t wait for new equipment or functionality; try testing or recording with existing resources, such as paper and pencil instead.

After making a change, a team should ask: What did we expect to happen? What did happen? Were there unintended consequences? What was the best thing about this change? The worst? What might we do next? Too often, people avoid reflecting on failure. Remember that teams often learn very important lessons from failed tests of change.

If the test shows that a change is not leading to improvement, the test should be stopped. Note: “Failed” tests of change are a natural part of the improvement process. If a team experiences very few failed tests of change, it is probably not pushing the boundaries of innovation very far.

What will you count or measure to be able to demonstrate improvement?  In Step 5, you should have developed a family of measures; however PDSA Measures are those that are collected with each test of change (PDSA) that is carried out. These measures provide information about the effect of each change attempt and may be different than your overall family of measures.  Remember you are collecting data for improvement not research or judgement and thus you do not need a large sample.

 

Improvement Journey Exercise #7

  1. Use the Plan-Do-Study-Act Worksheet  to plan, test, document, and refine your first small scale, rapid cycle test of change.
  2. Develop the Data Collection Plan for your first small scale, rapid cycle test of change.