Ideas and tools to work with The Dynamic Loop Method
Is your training and development initiative flying blind? Jim Kirkpatrick asked me this intriguing question when I met him in Dubai for my Kirkpatrick Silver Certification. And it came back to my mind as I started reflecting on what really matters in this 4th phase of The Dynamic Loop Method.
Find in the following
- What does it mean to fly blind?
- From scrap to hyrid evaluation
- Powerful evaluation questions
- 2 questions for IMMEDIATE USE after the training
- 10 questions for DELAYED use after the training
- Take-Away: Ready to sit in that cockpit and fly full power?
What does it mean to fly blind?
Participants have just finished their training session and we have received some great initial feedback on the value and relevance of the training provided. Tempting and from my experience very common to think “we are all done”. Learners have acquired the intended knowledge and skills and even received a carefully thought-out transfer package to make sure the desired performance or behavior change is achieved. And yes, it is great and encouraging to see more and more L&D professionals thinking beyond just designing a training event. And actually putting together beautiful learning journeys that provide learners with the required support and reinforcement to make learning stick and drive performance. But we are far from being done here.
So what is still needed? Let’s go back to the question Jim asked me. Is your training and development initiative flying blind?
Learning from US Air Force Pilots
Many years ago Jim Kirkpatrick was working with the U.S. Air Force in Utah, where they service and fly F-16s. He gave them a hypothetical: “You have the best-trained pilots in the world, you have the best mechanics, the aircraft is finely tuned. When the pilot takes off – let’s just say he/she turns off the radar and shuts down the instrument panel and the radio.” They responded, “That would be ridiculous. Without radar, they wouldn’t be able to detect any threats; without the instruments, there would be no way to know how the plane is functioning; and without the radio, there would be no communication about issues, problems, flight plans, etc.” Jim followed with, “Why is that a problem? Couldn’t they just look out the window to see where they’re going?” They said: “No, that wouldn’t work; they’d probably fly into a mountain – the pilot would be flying blind.”
Now let’s take this metaphor back to the world of training. Such a situation would be like eliminating funding for ongoing support and evaluation once our learners are back on the job. They do something, for sure. Follow transfer tasks. But are they really doing what they need in order to achieve desired goals? We can’t know. The training is “flying blind”. And we cannot afford that.
Training evaluation cannot only be done at the end. We need to fly our initiatives with the full power of our dashboards, evaluation radar and two-way radio = continuous evaluation throughout the learning experience (training and on the job application). So that we can identify barriers and success factors early and have a chance to adjust, correct or continue doing what we are doing.
Turning on the radar – From scrap to hybrid evaluation
When I shared this idea with one of my clients their initial reaction – and maybe you can relate – was “Okay, makes sense but aren’t we going to drown in evaluation data if we make evaluation an ongoing process? Already now when just doing evaluation at the end of the program we have difficulties analyzing all the data.”
Well, have you ever heard of the term “scrap training?” Frank Anderson, past president of the Defense Acquisition University, defined it as any training that is not meaningful and does not contribute to Level 3 and Level 4 of training.
Tipp: Check out the four levels of training evaluation by Kirkpatrick Partners. Read more here!
Jim Kirkpatrick coined a new term in 2012 which is “scrap evaluation” – this occurs when we are evaluating data that has not been gathered intelligently or that doesn’t provide useful information.
Message for us in HRD:
- We need to be very deliberate and purposeful about the evaluation data we are collecting and not just follow tradition (doing what has always been done).
- We need to make sure that the data we are collecting is useful to us. Examples would be that we can use the data to enhance the program design, remove a barrier or promote and demonstrate success.
My tip: Using a so-called hybrid approach that maximizes useful data with minimal resources
One of the most debilitating misconceptions of the four levels (see again: Kirpatrick Method) is that the proper way to deal with them is one bucket at a time. Nothing could be further from the truth or less effective. The power is in connecting the levels, not keeping them separate. Here is a chart that sets the stage for a powerful yet relatively simple solution to the problem of scrap evaluation:
Source: Kirkpatrick Partners – All Rights Reserved
Powerful evaluation questions
What we call hybrid tools are a good way to maximize evaluation resources and participant time. These are tools that measure multiple levels at the same time, such as a survey with questions related to Levels 1 and 2 immediately following training (possibly also including some predictive Level 4 questions), and a delayed survey with questions on all four levels. Just like The Kirkpatricks I would generally advocate that all evaluation tools be hybrids except in unusual circumstances. This prevents “surveying people out” because you can obtain quite a bit of data with one tool.
To make this more practical and tangible, let me share with you some of my favorite questions that can be included in a hybrid survey.