Satisfaction and Application Survey

Are you trying to measure satisfaction of your learning/training events? Here is a tool that can help. It is rather general, but achieves some success in connecting the dots in your learning environments and operations. Click on the title of this post to see the PDF of this survey.

Communication is essential for all audiences, internal and external…

Open dialogue during the process of creating a storyline is a good way to get others in your organization involved. Share the vision and initial plot with your partners in Training and Operations as well as your Executive. You will develop a much richer story that more people want to follow and bring to life. Your key message points will drive a timeline. Your timeline will contain points at which you share your external plan with your internal partners.

Integration of messages is challenging at best if you do not engage your partners. Key messages will enable executives to understand what measurements are necessary. Communication plans shared with operators encourage ownership rather than detachment between the company and their services.

Customer loyalty is the best measure of the success of a product or service…

Satisfaction only measures an immediate and isolated reaction to a product or service. Multiple measures that tie to an internal listening post build to understanding customer loyalty. These measures are developed from your storyline and key message points.

Understanding the customer experience follows satisfaction in measuring the success of a product or service. Multiple measurements over an extended period of time create your experience measures. Combine these measures with your satisfaction reports and an experience impact is realized.

Building relationships with customers through many points in time, a diverse product line, and a rich storyline is the next step of success. At this point, you will have reinforced your key messages efficiently. You will be within the last phase of your communication plan. Knowing not only what people liked, but what continues to encourage them back is important to maintaining market share and enable predictability in returns on your reinforcement investments.

Loyalty is the final measure you can establish. Once your story is shared, your internal partners have embraced your vision, and your communication efforts have been exhausted, you are ready to measure customer loyalty. By combining satisfaction results with that of experiences and relationships, you will obtain a loyalty factor and understand the breadth of your brand essence.

Branding a part of your organization is a piece to the long-term puzzle, not the answer to short-term finances…

Branding is the successful transition between what a product is and what you wish it to become. It is the one piece that if it sticks well, can mean a lifetime of positive press. If the branding becomes a sign of negativity, much work is needed to reinvent the product. A new look is needed. A new campaign must be established in order to distance yourself from the old brand. A new story will need to be shared, and shared, and shared, in order to create new key message points.

Integrating messages becomes more important when establishing a brand than if you are just selling a product and looking for short-term financial success. Many businesses today design posters and flyers or radio and tv ads, but do not have any other intentions other than the hope that people see the ad and experience the product. Research is conducted by independent firms to see if customers remember the company and product associated with an ad. What is being observed is that even though we can recall an ad because of an emotional connection, we may not be able to remember the product.

Reinforcement is the only way to change this behavior. So, you can either produce and distribute more ads, or you can choose multiple venues and share the key message points created from a distinct storyline and phased timeline. With the later, your brand gains momentum and is more likely to drive continued and predictable ROI rather than immediate and unstable.

Internal buy-in and understanding are signs of a good marketing strategy…

Our employees are our customers whether we like it or not. Whether we sell things employees need or sell only to specific agencies, if the employees buy into the packaging of the product, they can ensure our customers of its quality. If they do not, the product will not sell itself for an extended period of time, if at all. We can create the best device, software package, or process. Without the support of all of our partners, we just have a product.

Projections, for the most part are short-term, short-term in the respect to the life of the company. Storylines and key messages can be reinforced at a later time while new material is created to encourage buy-in and increase positive impact. Phased timelines allow for measures to be taken at key moments of the products lifecycle. These measures are based on the key messages and branding efforts that have been developed. Measurements should not be based on arbitrary satisfaction points as this would only provide satisfaction with the company not necessarily the product or service itself.


Strategy without values sway slightly as the breeze of change shifts with the expectations of solutions…

Trainers are expert trainers….

Trainers know the content of their training enough to apply it effectively. They can tell you or any member of your staff why you conduct one process before another. Trainers can demonstrate the ideal patterns of usage for the content they train. They can also help reinforce the messages of the organization with you.

However, trainers are experts in training and learning. If you pull them to “help” the customers, who is helping your staff? Who is helping you? Customers always come first. However, by focusing on the numbers rather than the quality, you will negatively impact your customer satisfaction in the long run. Busy periods are the best times for the trainers and leaders to spend time with the staff to listen and learn about the flow of the process. Using your trainers to reinforce the content of training and how it connects to the business objectives and marketing plan can help you operate more efficiently. You build the skillset of your staff so that they can handle more customers, more effectively.

If you use trainers to act as expert front-line staff, you will watch the skills of the actual front-line staff decline and your customer satisfaction level out, and even begin to dip over time. You will also see your training quality decline over an even longer time
as your trainers become burned out. They will not consider themselves qualified trainers. They will, however, feel like glorified front-line staff. For those who enjoy the learning process, this will not be enough. Eventually, you will lose your best trainers because you didn’t respect them and the skills they bring. Remember, if you are not respecting your staff and your partners, your customers will eventually feel the same way.


Expertise is a funny creature…
it often disappears when another like beast enters the lair…

Assessments measure knowledge of learners…

Knowledge and skill assessments should be based on the learning objectives for the class. The questions should be distributed proportionally across the learning objectives based on the percentage of content for each objective. The learning objectives themselves are best aligned with business objectives to determine the importance and priority of focus. The business objectives will need to be identified and developed prior to creating learning objectives.
Knowledge and skill assessments are a good source to measure what knowledge the participants have when they leave the learning environment. However, unless you conduct a pre-assessment, it can be difficult to determine if they learned it from the training or enter the class with the knowledge. Participants may not communicate their knowledge effectively in a pre-assessment even if they know the content, especially if the questions still need to be tested for construct (can the knowledge be used effectively) and criteria-related (is the knowledge being used by top performers) validity.

Application evaluations become more difficult when you try to educate a group who might be familiar with the content. How do you know when behaviors stem from a class or learning event you devised? How do you know that your content didn’t trigger an approach participants learned at an earlier stage in their careers? At times, we may not know if someone’s past experience aided him/her in understanding material better. All this should be taken into consideration before a level of competency, application, and ROI are calculated and set.

Optimal learning comes in different formats…

Needs assessments are almost a necessity these days in any training environment. Designers, analysts, and leaders can devise a plan to hold dialogue sessions, focus groups, or focused dialogues with specific questions around the topic at hand. This same group may also align the gaps identified within these sessions to the goals of the organization. They will even develop a class, a tool, or an event to address these gaps.

However, in order for optimal learning to occur, different formats must be created. Using the expertise of an instructional designer and possibly an analyst, you can determine which format is right for the initial training and what may be a better fit for follow-up sessions. You may determine that there are too many questions to be answered with an online resource and start with a classroom setting. However, as your population begins applying the learning, you can introduce a web-based class for reinforcement with various checks for understanding built into the tool.

Integrating a gap analysis approach into your delivery methods will help increase your ROI through connections made within the learning itself. This shows the importance of the events and the organization’s investment in the people rather than the product, service, or system. Integrated learning indicates an investment itself rather than a point in time.

Sometimes, when alignment is out of sorts, it creates all sorts of alternate sometimes…

Satisfaction results are summary points…

After a training class is complete, the first thing we need to do is examine the satisfaction surveys…or, at least that is what so many training professionals do. You may skip to the end and read the comments. Or, look at one particular question or set of questions to determine if your training was successful. This would be a daunting task for even the most mathematical minded among us.

Unless you can calculate correlations and build scattergrams in your head, understanding the success of your training based on quick glances at pieces of paper will probably not occur. What does happen, however, is that we create an imaginary summary and decide to change our approach and sometimes, even modify the content for the next class. This can be hazardous to your training and driving your positive ROI.

Remember to recognize the impact of your approach, delivery, and design. If training is changed constantly and the content shifts, you will lose the largest group of reinforcement you may have…the learners themselves. You may also create more work for yourself, as the job-aids will need to be different from class 1 to class 20. It is a good idea to conduct pilot classes and change content from there. Remember though to bring backthose people when the final class structure is set. Every time you change, you have to determine what influenced the change. Basic statistics can help with this. However, you will need to decide when to use it. Is it in the revision of the training for a new hire class? Or, is it within your reinforcement plan? The goal is to minimize impact and maximize application in this situation.

People ensure learning will be successful…

By aligning your training objectives to business objectives, you help fill gaps in content from a learning perspective. You can conduct multiple needs assessments and analyze the data until the cows come home…however, since most live in a city environment now, you may be analyzing an awful long time. You can even make sure the content aligns proportionally to the greatest and least need of your organization. This still will not ensure success. It only helps it along.

Learning is a challenging task and it takes people to make it work well. Sure, you can have online, web-based, PDA learning events, but someone, somewhere, has to put the event together. An even greater number of people are needed to hold things in place after participants engage in the learning. As adults, we have to link some importance to content before it replaces something else in our short-term or long-term memory. People, whether they are leaders, trainers, or executives, need to connect the dots for us after we attend a learning event by talking about how it relates to our everyday work.

We can look toward measurements to help integrate the learning experiences. Through satisfaction measurements, we are able to understand what marketing of the training needs to take place for the next group. Through knowledge assessments, we are able to revise or reinvent how content is delivered to a group after they attend the initial training so that we can build off of it. With Level 3 surveys, we can recognize the impact of our efforts and minimize the effect of work-arounds or lack of focus on our ROI.

Measuring what is not applied creates more work for those who are measuring…

Creating A Skill-Based Organization - Culture

Culture:
Culture will be impacted by a change itself. As the criteria and compensation change, so will the culture.

Progressing through Change:
The progression of a culture is determined by the ability of the organization to manage change effectively. In order to do so, the following items are recommended prior to implementation and even some points of development:

• Class for leadership on change management
• Discussions with leaders around impacts of change to the individuals, teams, and partnerships
• Class for leaders on communication
• Opportunities for the leaders to demonstrate the ability to adapt to change and lead their teams through change while managing resources

Initiating Continuous Learning:
While criteria for pay and employment may change, one piece should remain constant, the employees ability to learn. As the criteria expands to include skills outside the current expectations, new learning opportunities should be developed or researched in order to accommodate a continuous learning process. The following is a recommended path to facilitate this process (we are already headed here):

• Conduct a needs analysis of organization to determine:
  * Learning styles (Complete)
  * Commitment to organization (Complete)
  * Commitment to learning (In progress)
  * Desire of individuals to advance within organization (In progress)
• Provide a learning plan for each individual that covers the grouped competencies within each role
  Measure success of initial implementation through focus groups with specialists, leaders, leader-of-leaders, and executives.
• Share impact of set learning plan with organization
• Provide training and facilitated discussions to leaders helping them create individual learning plans for themselves. (This is a partnership between our organization and OD…OD started down this path during the last Leadership Summit.)
  Request leaders share plans with their teams and have individuals create plans using the leader plan as an example.
• All training/learning events and assessments should be tracked through a common system (i.e., OnTrack). This would include all new hire, reinforcement, and leader training events.
  * Naming conventions for classes are needed for each Competency and location
  * Responsibility for class creation, tracking, and reporting should be assigned to one member of the local teams and a global resource
  * Rosters for each class should be generated prior to class and closed within 48 hours of the event.

Compensation:
Compensation has a bigger impact on culture than it does on performance. Professional compensation analysts and Human Resource professionals, skilled in budgeting, research, and the current job market should be used to determine what will:

• Encourage applicants to apply
• Provide candidates the opportunity to advance
• Allow for the most effective labor budgeting to occur
• Increase internal transfers
• Decrease external attrition

Creating a skill-based organization

When embarking on a culture reliant on skill-based pay and structure, the following should be kept in mind during development and deployment so as to increase the positive impact on criteria, culture, and compensation.

Criteria:
Criteria for skills and assessments can be established through focus groups of “experts” within the organization and external to it.

Establishing criteria:

• Executives and/or strategy group will create a list of grouped competencies for each level of organization
• Executives and/or strategy group will determine the percentage of time spent at each competency grouping
• Executives and/or strategy group create a base list of skills, knowledge, and experiences that they believe increase the performance of individuals based on the identified competencies.
• Review list with groups of people from the following teams:
  * Compensation
  * Human Resources
  * Other senior leaders
  * Front-line leaders
  * Group effected by skills approach

Initial testing development:
In order testing to be acceptable and defendable in court for employment purposes, it must possess the following forms of validity:

• Face Validity – Do the questions read well? Can they be understood?
• Content Validity – Can this be learned somewhere? Is the content of the test within the training?
• Construct Validity – Can the content of the questions be used in the working environment?
• Criterion-related validity – Do those who are successful in the organization do well in the areas being tested?
  

If organizations use an outside agency to create the assessments, they will mostly cover the first three levels of validity and can provide documentation to that effect. This documentation should be requested. For the last piece, organizations can conduct it on their own, usually over a one-year period or contract an outside organization to do so.

If organizations use an outside agency to create the assessments, they should also look to that organization to establish a recommended proficiency level for each assessment.

Future testing implementation:

Allowing individuals to test for a skill after joining the organization may cause the following questions to arise within the effected group:

• Will I have an opportunity to learn this skill here?
• Will I need to learn this skill on my own?
• Will I be compensated for my labor while learning this skill?
• What level of competency will I need to demonstrate in order to prove proficiency within this skill?
• Who determines whether I pass or fail?
• How often can I take the assessments? (Recommendation would be every 6 months but no more than three times within a 2 year period.)
  

If an organization chooses to construct the assessments internally, the following criteria must be in place in order to complete the levels of validity mentioned above:

• General and specific learning objectives
• Sample and population sizes large enough to conduct a statistically valid analysis from the data.
• Time in which to conduct pilots and revise questions based on the four levels of validity. This could take between 3 to 12 months.
• Team of 2-3 analysts who create assessments and analyze training evaluation data based on the following:
  * Satisfaction
  * Knowledge
  *Application

More later on culture and a skill-based organization...

Reinforcement provides a primary impact to application…

The number one reason why training fails is that it is not supported by the leaders after it is delivered. The number one reason learning fails is that it is not reinforced. Both are similar, yet different in their distinct ways.

Leaders who choose not to reiterate the importance of a training event push the trainees further away from the intent of the training class. The mindset of the participant shifts to, “well, if my manager doesn’t know about the class, why should I?” Negativity fosters negativity. If you choose, or your organization chooses, to have training events, then your leaders should attend the training as well. The success of the initial program is always on the training team. However, the long-term impact is felt more in the operations. By recognizing the impact of training, you will need to set a transition point so that your partners in the operations and support areas can pick up where you left off. They will be significantly challenged to do this if they do not have an idea about what was learned within the training. The higher up you go in the organization, the broader your view.

Leaders who choose not to make the content a part of their every day interactions with their staff will assist in the decay of the learning process. Negativity plays a small role here, but lack of usage, lack of interest, and higher levels of priority push out the content of a learning event. The success of a continuous learning process is always on training and development. As training professionals, we need to organize delivery effectively and provide job-aids or reminders at various times after the training. The reinforcement tools provide leaders with opportunities to talk about and direct their staff toward growing from a learning experience rather than just attending a class.

Strategic Planning

It can be simple, or it can be complex…for the most part, it is both. When planning a project, developing a plan that centers around solid, business objectives is a essential to the overall success of that project. However, defining those business objectives may be a challenge. In our results-orientated world, we need to act, and act quickly, otherwise…well, otherwise we will wait to act…and, that may not be a bad thing.

About five years ago, I experienced some pains on the left side of my chest one morning and didn’t know why. In addition to the pains was this tingling feeling in my left arm. I went to work and hoped that the pain would go away as most simple pains do…By the afternoon, the pains started to feel as if they were going through my chest and coming out my back, making sitting through long and boring meetings, well, long and boring…so, I took some pain killers (acetaminophen) and the pain partially subsided within the hour. By dinner time, however, the pain returned. After expressing some concern to my then fiancé, Teri (now my wife), it was decided that I should call the nurse line that we had access to through our place of employment…I really didn’t decide much, it was strongly encouraged.

So, I did what I should have done in the afternoon and called an “expert.” After navigating through the incredibly intricate phone tree that any technician would be proud of, a very nice nurse answered that phone and asked how she could help. I described the symptoms I experienced at that moment. I heard her voice become more concerned as I answered her question about the tingling feeling in my arm, saying, “yes, I had some of that this morning, but it went away.” She paused and asked if someone could drive me to the emergency room. I was about to say, “sure, I can just call…,” when I realized she thought I was having a heart attack…

OK, OK, I have had these problems before and they all were not heart attacks…I know this because the EKG indicated that the first two times. What was I missing? What was she missing? It was at that point that I realized I was missing the most important part to our conversation, the objective for the call. The objective was not to diagnose my pains. The objective was to help me deal with them. I wanted to understand what muscles may be impacted so that I can choose the best form of treatment. I wanted to understand the types of treatment for sore muscles. I had to back up quite a bit in the conversation including adding that two days prior, I was moving large furniture around in the house as we painted each room. Oh, did I forget that in the story here as well…?

The cause of my pain…the objective of my concern…the plan for treatment, not to mention the timing and cost of recovery would have been negatively impacted if I didn’t share the full, bigger picture. Strategic planning involves sharing the larger vision and creating it if needed. Take it from the top, because the show will go on with or without us.

We would use a simple diagram with branching to serve as a stepping-stone platform from which we can develop our strategic goals, communication plan, and implementation strategy without getting lost so far into the details that our customers think we are having daily “heart-attacks.”

Defining Organizational Impact:

• We begin at the beginning. We know what our Vision and Mission are, we even understand the core strategies and initiatives. However, when developing the organizational objectives, alignment to these items is often missed, causing concern, miscommunication, poor training, and eventually a low or negative return on our original investment.
• As we create the objectives at this level, think about the general items that will change for everyone. These become our organizational objectives or main message points.

Defining Cultural Impact:

• In order to understand the cultural impact, understanding the culture itself is important. What main groups make up the culture? What are the common topics for each group?
• As we create the objectives at this level, think about the organization objectives we have defined. These items are a little more specific for each culture, but should be broad enough to touch multiple areas.

Defining Team Impacts:

• Everything a company does effects it employees first and its customers second. In order to positively impact both, we should ask what both groups need for a product to be successful.
• Objectives for this level are more refined and deal with specifics. We will have many more of these than the other two since there are multiple teams.

Diversity enables peak performance…

Change is always a personal thing. We can say that we accept the change within an organization, but the extreme to which we accept it always varies. Just look at any change cycle model if you question this. Each will show you that there are steps one goes through from intrusion/loss to integration/acceptance. We all start at different places along the path to acceptance. However, it is the reinforcement of the message that will lead us to a more positive environment.
If everyone went through change the same way, we would have very little diversity. Without diversity in actions and thoughts, we would be a rather boring company. We would not be able to market our products well since we would only understand our way, which may be different from what our customers prefer. We would not be able to troubleshoot problems because everyone would come up with the same ideas, the same ones that do not work now. If we all accepted the changes in the same way, we would not be able to grow as a company and as individuals. We would stay where we are personally and professionally.

While supporting an idea or direction is good business sense, supplying alternatives to the approach and offering suggestions for improvements should be welcomed from all levels of the staff. It would make it extremely difficult for marketing to sell a product or service if they don’t know what the customers are saying about our current offerings. It would make it very challenging to lead a team without a focus, and you can not develop a focus unless you listen and communicate.

Leading people encourages peak performance…

If you are a senior leader, you lead the organization and offer assistance to others on how to lead their teams. If you involve yourself in the management of things too oftern, results will be negative. Someone has to keep the focus on a higher strategy. Someone has to ground the organization on what success looks like.

By offering assistance to your organization at various points, and demonstrating how to lead, especially in times of change, you commit to the success of the project and your organization. You also demonstrate that you want continuous ROI, not just one big enough to build your own career.

Driving for results is good if you have a purpose. Negative reinforcement of training by penalizing those that “fail” assessments instead of understanding where gaps lie, will negatively impact results, especially short-term results. Offer opportunities to demonstrate knowledge and application. Offer positive reinforcement through recognition for high levels of application. By leading through change, you can manage your return more thoroughly.

Listening to and trusting employees express understanding and appreciation…

How many times have you heard, “they” don’t know what we do…or, “they” couldn’t do our job…or even, “they” need to do our job, then “they’ll” understand. Guess who the “they” is…If you said, you, the executive, you are correct. It is any decision-maker, but executives get the most of this responsibility.

Many executives have either been there, done that, or feel it is not their place to do the work of their staff. That attitude, like it or not, is negative and will negatively impact the organization. The results can be positive though. Listen to your employees and trust that they know their job. In fact, they possess more knowledge about their roles than you. Communicate your thoughts about the direction of the company, but listen to the experts. Remember, increasing knowledge enhances application. This leads to a more positive return on your investment.

As an executive, you should stay focused on the vision. The more you become involved in the day-to-day activities of your organization, the further from your vision will your organization move. The reason for this is quite simple. If you engage in the daily operations, there is no one to remind the teams why they are working so hard…why the content of the training is important…and why it is significant to transition knowledge into application.


Remembering is only painful if we choose to forget first…

Driving ROI - Customer Satisfaction

Customer satisfaction is key to increased profits…

You can serve customers through marketing, sales, and returns. Just because you tell people about your company, doesn’t mean that they will purchase your items or services. Just because your company sells products or services, doesn’t mean your clients will not return the item itself, or ask for their money to be returned. Ideally, the more customers that purchase your products, the higher your profits should climb. Higher sales, however, do not always equal customer satisfaction.

If you focus on delivering quality to your customers, they will help increase your profits. Word-of-mouth marketing is worth more than a 30 second spot in any major sporting event. In order to understand what your customers need, you should listen to your teams and reinforce their listening skills. When you listen to them, it demonstrates a positive behavior and encourages your team to do the same for the customers. These comments should be shared with your partners in training and marketing.