Ideas and tools to work with The Dynamic Loop Method
Is your training and development initiative flying blind? Jim Kirkpatrick asked me this intriguing question when I met him in Dubai for my Kirkpatrick Silver Certification. And it came back to my mind as I started reflecting on what really matters in this 4th phase of The Dynamic Loop Method.
Find in the following
What does it mean to fly blind?
Participants have just finished their training session and we have received some great initial feedback on the value and relevance of the training provided. Tempting and from my experience very common to think “we are all done”. Learners have acquired the intended knowledge and skills and even received a carefully thought-out transfer package to make sure the desired performance or behavior change is achieved. And yes, it is great and encouraging to see more and more L&D professionals thinking beyond just designing a training event. And actually putting together beautiful learning journeys that provide learners with the required support and reinforcement to make learning stick and drive performance. But we are far from being done here.
So what is still needed? Let’s go back to the question Jim asked me. Is your training and development initiative flying blind?
Learning from US Air Force Pilots
Many years ago Jim Kirkpatrick was working with the U.S. Air Force in Utah, where they service and fly F-16s. He gave them a hypothetical: “You have the best-trained pilots in the world, you have the best mechanics, the aircraft is finely tuned. When the pilot takes off – let’s just say he/she turns off the radar and shuts down the instrument panel and the radio.” They responded, “That would be ridiculous. Without radar, they wouldn’t be able to detect any threats; without the instruments, there would be no way to know how the plane is functioning; and without the radio, there would be no communication about issues, problems, flight plans, etc.” Jim followed with, “Why is that a problem? Couldn’t they just look out the window to see where they’re going?” They said: “No, that wouldn’t work; they’d probably fly into a mountain – the pilot would be flying blind.”
Now let’s take this metaphor back to the world of training. Such a situation would be like eliminating funding for ongoing support and evaluation once our learners are back on the job. They do something, for sure. Follow transfer tasks. But are they really doing what they need in order to achieve desired goals? We can’t know. The training is “flying blind”. And we cannot afford that.
Training evaluation cannot only be done at the end. We need to fly our initiatives with the full power of our dashboards, evaluation radar and two-way radio = continuous evaluation throughout the learning experience (training and on the job application). So that we can identify barriers and success factors early and have a chance to adjust, correct or continue doing what we are doing.
Turning on the radar – From scrap to hybrid evaluation
When I shared this idea with one of my clients their initial reaction – and maybe you can relate – was “Okay, makes sense but aren’t we going to drown in evaluation data if we make evaluation an ongoing process? Already now when just doing evaluation at the end of the program we have difficulties analyzing all the data.”
Well, have you ever heard of the term “scrap training?” Frank Anderson, past president of the Defense Acquisition University, defined it as any training that is not meaningful and does not contribute to Level 3 and Level 4 of training.
Tipp: Check out the four levels of training evaluation by Kirkpatrick Partners. Read more here!
Jim Kirkpatrick coined a new term in 2012 which is “scrap evaluation” – this occurs when we are evaluating data that has not been gathered intelligently or that doesn’t provide useful information.
Message for us in HRD:
- We need to be very deliberate and purposeful about the evaluation data we are collecting and not just follow tradition (doing what has always been done).
- We need to make sure that the data we are collecting is useful to us. Examples would be that we can use the data to enhance the program design, remove a barrier or promote and demonstrate success.
My tip: Using a so-called hybrid approach that maximizes useful data with minimal resources
One of the most debilitating misconceptions of the four levels (see again: Kirpatrick Method) is that the proper way to deal with them is one bucket at a time. Nothing could be further from the truth or less effective. The power is in connecting the levels, not keeping them separate. Here is a chart that sets the stage for a powerful yet relatively simple solution to the problem of scrap evaluation:
Source: Kirkpatrick Partners – All Rights Reserved
Powerful evaluation questions
What we call hybrid tools are a good way to maximize evaluation resources and participant time. These are tools that measure multiple levels at the same time, such as a survey with questions related to Levels 1 and 2 immediately following training (possibly also including some predictive Level 4 questions), and a delayed survey with questions on all four levels. Just like The Kirkpatricks I would generally advocate that all evaluation tools be hybrids except in unusual circumstances. This prevents “surveying people out” because you can obtain quite a bit of data with one tool.
To make this more practical and tangible, let me share with you some of my favorite questions that can be included in a hybrid survey.
2 questions to use IMMEDIATELY AFTER the training
Confidence question for Level 2 – Learning success
Rating scale question: I feel confident about applying what I learned back on the job.
(Optional add-on) If you circled 6 or below, check all that apply.
My confidence is not high because:
- I do not have the necessary knowledge and skills.
- I do not have a clear picture of what is expected of me.
- I have other, higher priorities.
- I do not have the necessary resources to apply what I learned.
- I do not have the human support to apply what I learned. The training didn’t give me the confidence to apply what I learned.
- There is not an adequate system of accountability to ensure the application of what I learned.
- Other (please explain) :____
What I like about this question: It serves as an early warning system regarding possible barriers with regard to application on the job.
Predictive question for Level 4 – Business success
Rating scale question: I believe I will see an impact in the following areas as I consistently apply what I learned (check all that apply):
- Increased productivity
- Improved quality
- Increased personal confidence
- Increased customer satisfaction
- Stronger relationships with my colleagues
- More respect from my peers
- Better organization in my work
- Other (please explain): ___
What I like about this question: It tells me what to keep my eyes on as I am monitoring the impact of the program. The possible impact areas mentioned vary of course based on the training context (expected results).
10 questions to use DELAYED after the training
When running a delayed survey I am a big fan of including questions that not only tell me whether people are applying what they have learned on the job but dig deeper into finding out possible application barriers as well as success factors.
- My supervisor and I set expectations for this training before the class. (rating scale question)
- My supervisor and I determined how I would apply what I learned after training. (scale)
- I have received performance support in order to apply what I learned successfully. (scale)
- I receive support and encouragement for applying my learning to my job. (scale)
- I have the necessary resources to apply what I learned successfully. (scale)
- A system of accountability helps me to apply what I learned. (scale)
- Incentives encourage me to apply what I learned. (scale)
- When I apply what I learned, I am rewarded appropriately. (scale)
- What additional training or support do you need to increase your effectiveness? (open end question)
- What kind of support have you received that has helped you to implement what you learned? (open end)
Take-Away: Ready to sit in that cockpit and fly full power?
It’s a paradox: While we as HRD professionals get better and better at designing learning initiatives that are highly beneficial for participants (say transfer tools, relevance, flexibility…), at the same we often do not know HOW EXACTLY they help and what they do. For both individuals and learners. And this equals having a super-duper high-tech plane but flying around with radar off. No one would do that. And so shouldn’t we. So, let’s take our cool programs and use the power of evaluation – especially hybrid evaluation – to stay on crystal clear tracks, directly leading to the results we want.
Meet the author
Melanie combines her entrepreneurial spirit with her experience in the learning field to help build memorable and results-based learning experiences. Melanie has 15+ years of experience in Learning & Development and has facilitated workshops on a variety of topics such as cross-cultural management, leadership, negotiation, executive presence diversity, and inclusion across the globe. Two years ago, Melanie decided to combine her passion for sports and learning and co-founded The Learning Gym. TLG’s focus is on upskilling learning professionals and helping them stretch, tone and build their learning muscles in the field of needs analysis, training design, delivery, and evaluation.
Along with being an Accelerated Learning Master Facilitator, Melanie is also India’s first Kirkpatrick certified facilitator. The amalgam of these helps her provide customized learning solutions for both the F2F and virtual classroom that significantly improve the return on expectations.
